The Agent Firewall: Statistical Behavioral Analysis

Baseline normal tool frequency, then auto-block and page when a read-only agent suddenly looks like an admin — ATR allowlists stop unknown tools; behavioral tripwires catch abuse of the tools you already allowed.

Part 8 joined prompt → reasoning → syscall on one traceId. This post answers the residual question: when every individual tool call is still “in ATR,” how do you notice that a read-only diagnose agent suddenly looks like an admin — and trip the wire before the volume becomes the breach?

---

Every Call Was Allowed

ATR for DiagnoseService allows metrics_query, log_read, file_read under /workspace, and a narrow ticket_comment. Panguard’s day is green — zero blocks. Langfuse shows a busy afternoon.

What changed:

Baseline (14d):  ~40 tool calls / session, 95% metrics_query + log_read
Today session:   900 file_read calls in 12 minutes
                 paths creep toward ../.env via "config discovery"
                 then vault_secret_read attempts (HITL denied — noisy)
                 then a sudden burst of egress-bound skill runs

No single call was an unknown tool. Tetragon never saw curl. Seccomp was quiet. FIM blocked the worst paths. The incident was shape: volume, mix, and privilege intent vs role baseline.

ATR allowlists answer “is this verb permitted?” Behavior answers “is this session still the agent you thought you hired?”

---

Panguard Is Necessary, Not Sufficient

Panguard already enforces: ATR claims per tool + params, JSON Schema with additionalProperties: false, rate limits and session-level cumulative risk, HITL for high-risk verbs with fallback: deny. That still leaves a large space of individually legal abuse.

The agent firewall sits above ATR as a second opinion on volume and privilege shape — and beside Phase 2 as a chance to trip before the kernel must.

---

The Architecture Pattern: Statistical Behavioral Analysis

Emit every tool decision as a Part 8 canonical event with role, tool, outcome, traceId, sessionId. Maintain baselines per ATR role (and tenant, if multi-tenant): calls/min, tool mix, share of high-risk tools, deny rate. Define tripwires as deviations — not vibes: >3σ or >N× 24h median for a rolling window; first-seen tool for role; privilege-class jump. Respond automatically in levels: alert → soft block (rate squash) → session quarantine / ATR suspend → page. Observe 14 days before paging; target >80% precision before CRITICAL routing. Always attach join keys so an anomaly opens Langfuse + Panguard + host sensors in one path.

Tool calls (Panguard + Langfuse)
        │
        ▼
 Alloy / Loki / Mimir   (role, tool, counts — low cardinality metrics)
        │
        ├─ baseline store (7–14d rolling)
        │
        ▼
 Tripwire evaluation
        │
        ├── INFO: log only
        ├── HIGH: Slack + slow-mode rate limit
        └── CRITICAL: quarantine session + page
                │
                ▼
         Panguard enforcement hook

Detection without an enforcement hook is a museum of interesting spikes.

---

What to Baseline Per Role

Start narrow. One global “all agents” baseline will page every product launch.

Label with atrRole, tool, decision, deploymentTier — not raw sessionId on Prometheus. Per-session series belong in Loki/WORM events.

---

Tripwires

Unexpected tool name:

- alert: UnexpectedAgentToolUse
  expr: |
    count_over_time(
      {service="langfuse"}
      | json
      | atr_role="DiagnoseService"
      | tool_name !~ "metrics_query|log_read|file_read|ticket_comment"
      [5m]
    ) > 0
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: 'Role {{ $labels.atr_role }} used unexpected tool {{ $labels.tool_name }}'
    trace: '{{ $labels.trace_id }}'

Volume spike vs baseline:

- alert: AgentToolVolumeAnomaly
  expr: |
    (
      sum by (atr_role) (rate(clawql_tool_calls_total{decision="ALLOW"}[5m]))
      /
      sum by (atr_role) (rate(clawql_tool_calls_total{decision="ALLOW"}[5m] offset 7d))
    ) > 3
  for: 5m
  labels:
    severity: high
  annotations:
    summary: 'Tool call volume >3× seven-day rate for role {{ $labels.atr_role }}'

Privilege-class jump:

- alert: AgentPrivilegeClassJump
  expr: |
    sum by (atr_role, session_id) (
      count_over_time(
        {service="panguard"}
        | json
        | atr_role="DiagnoseService"
        | tool_class="high_risk"
        | decision="ALLOW"
        [10m]
      )
    ) > 0
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: 'Read-leaning role invoked high-risk tool in session {{ $labels.session_id }}'

---

Auto-Block: Close the Loop Into Panguard

Grafana / ruler / detector
        │  anomaly event (role, sessionId, traceId, ruleId)
        ▼
 Anomaly API (authz: security controller only)
        │
        ▼
 Panguard session state: quarantine=true
        │
        ▼
 Subsequent tool calls → 403 ANOMALY_QUARANTINE → WORM

Same fallback: deny spirit as HITL: if the anomaly service is down, prefer fail-closed for high-risk tools. Agents must not clear their own quarantine.

---

Correlation With Phase 2

---

Tuning: Precision as a Security Metric

New rule → observe-only 14 days. Monthly score each fire: true / false / indeterminate. Promote to paging only after >80% precision. Route CRITICAL to PagerDuty, HIGH to Slack, INFO to WORM only. Baselines drift with product launches — recalibrate on role definition changes the same day a new legitimate tool is added.

---

Honest Failure Modes

Short sessions. Per-session rates are noisy on three-call chats; prefer role aggregates plus session caps for absolute floods.

Multi-tenant dilution. Always partition by tenant/role or large customers drown small ones in the median.

Automation loops. Agent retries after soft errors can look like attacks — surface “blocked by anomaly policy” in UX.

Metric vs log. Cheap counters in Mimir for rates; rich events in Loki/WORM for forensics. Do not put tool_args into labels.

Overfitting to yesterday. Pure σ thresholds on sparse roles need floors (min_calls) or you page on night silence → morning standup.

---

Getting Started

Confirm Part 8 join keys on every TOOL_CALL / Panguard decision. Export low-cardinality counters: clawql_tool_calls_total{atr_role,tool,decision}. Record 14d baselines per role; publish expected tool regexes next to ATR. Ship observe-only alerts for unexpected tool + volume×3 + privilege jump. Add Anomaly → Panguard quarantine hook; canary test; then enable soft/hard stages. Add monthly precision review to the security operating cadence.

Part 10: if machines can’t group events, they can’t baseline or alert on them.

---

Companion: DevSecOps-boilerplate · Observability essay. Docs: Security monitoring / SIEM · Panguard / ATR.