Incident Response: Surviving an Agent Compromise

Wipe-and-redeploy loses the lesson. Snapshot memory, revoke ephemeral JWTs, isolate the node — then sanitize. Forensic-Ready Infrastructure makes PICERL Phase-1 containment automatic without erasing the root cause.

This is the close of Phase 5 and of the series. Parts 1–14 prevent, contain, and observe. This post answers the residual operational question: when something still breaks through, do you wipe the agent and lose the lesson — or do you quarantine fast, preserve evidence, revoke trust, then recover with a root cause you can prove?

---

The Helpful Wipe

PagerDuty: Part 9 quarantine for a diagnose agent. Someone with admin rights “contains” by deleting the pod, wiping the memory volume, and rotating only the deploy key. The page clears. Monday’s postmortem is empty: no memory snapshot, no Langfuse join, no Vault lease list, no sandbox Job leftovers. The same skill ships again Wednesday because nobody preserved the manifestHash that drifted.

The alternative failure: a thorough human investigates for forty minutes while the agent completes three hundred more tool calls. Detection-to-response was the vulnerability.

Speed without forensics teaches nothing. Forensics without automated Phase-1 containment loses the race. The correct order is: preserve → revoke/isolate → eradicate → recover → learn.

---

The Stack Is the Runbook Inventory

Every prior part becomes IR ammunition:

IR quality is capped by what Parts 8 and 13 made durable before the incident.

---

The Architecture Pattern: Forensic-Ready Infrastructure

High-confidence detectors fire automated containment in seconds — quarantine is reversible; snapshots are not optional. Quarantine events hit WORM before containment mutations when possible. Preserve memory, NATS logs, Vault lease audit, pod/FS, Langfuse/Tempo spans for sessionId/traceId. Then revoke JWTs, freeze lease renewal, deny egress, suspend sessions. Humans run PICERL Identify → Contain → Eradicate → Recover → Learn; Prepare is continuous. Auto-delete of memory or audit mid-investigation is never permitted. Circuit breakers reset only with explicit human action.

Prepare ····························· (always on)
   │
Identify ── alert + severity ──► IC assigned
   │
   ├─► Phase-1 AUTO (seconds)
   │     write WORM quarantine event
   │     suspend sessions / Panguard quarantine
   │     deny-all egress NP
   │     freeze Vault lease renewals
   │
Contain ── verify auto quarantine
   │        expand NS isolation if needed
   │        forensic snapshot COMPLETE ─────┐
   │                                        │
Eradicate ◄── preserve before revoke ───────┘
   │        revoke JWTs / certs / NATS
   │        remove bad skill/image; patch ATR
   │
Recover ── clean deploy; Merkle continuity; smoke
   │
Learn ── 48h review → new rule / runbook → WORM

---

Phase-1 Automation: Contain Without Destroying

Talon-class triggers:

• Panguard block threshold exceeded in one session
• Falco / Tetragon critical (unexpected exec, bind 0.0.0.0, …)
• Memory integrity failure (Part 13)
• Part 9 hard anomaly or Part 14 injection storm with host corroboration

Actions — seconds, reversible:

clawql ir quarantine --agent "$AGENT_ID" --session "$SESSION_ID" --trace "$TRACE_ID"
# → Panguard quarantine=true
# → NetworkPolicy egress deny-all on agent pod
# → Vault: freeze lease renewal (do not yet revoke-all)
# → WORM: POLICY/QUARANTINE event first

Circuit breakers:

• Session: >X Panguard blocks in Y seconds → terminate session.
• Tool: >Z errors in W minutes → disable tool fleet-wide until human reset.

Auto-untrip is not permitted. A flapping breaker hides attacker adaptation.

---

Forensic Preservation Sequence

Evidence before revocation — the order is the control.

1. Confirm sessionId / traceId / agent id from the alert (Part 8).
2. Snapshot to a forensics-only bucket (IR ACL, not the agent role):
   • Memory store / local vault copy (ciphertext + metadata; respect Part 13 key policy)
   • Pod filesystem / ephemeral sandbox volumes still alive
   • NATS / control message window for the session
   • Vault lease and auth audit around the window
   • Export Tempo/Langfuse spans + Loki {clawql_trace_id=…}
3. Verify snapshot checksums and object locks landed.
4. Then revoke: session JWTs, refresh tokens, SPIFFE short-TTL wait-out, NATS ACLs, object-store STS.
5. Expand isolation if needed (namespace NetworkPolicy, pipeline halt).

WRONG: kubectl delete pod; rm -rf memory; "we rotated later"
RIGHT: snapshot → quarantine verified → revoke → eradicate → recover

---

PICERL

A learn package that only narrates is a report. A learn package that merges a detector is IR.

---

WORM as Investigation Surface

Reconstruct session timeline by sessionId. Merkle-verify audit and memory roots first — if integrity fails, treat telemetry as suspect. Export a time-bounded, signed evidence pack for legal without dumping other tenants. Correlate: Panguard allow → Tetragon kill → path deny → injection scrub miss (Parts 4–6, 14).

---

IR vs DR

Ransomware: run both. Untested RTO/RPO remain aspirations. IR still needs the forensics bucket even when DR is promoting another region.

---

Honest Failure Modes

Over-automating eradicate. Auto-delete of images/skills without snapshot loses Part 12 evidence.

Under-automating contain. Human-only quarantine is how agents outrun on-call.

Key escrow for memory snapshots. Forensics may need unwrap rights separate from the agent — document dual-control (Part 13 GDPR crypto-shred vs investigation).

Noise. Talon on weak signals causes outages; bind to high-confidence classes and Part 9 precision targets.

Edge laptops. Same script: freeze agent process, image disk/vault, revoke Tailscale/session, then wipe — Seatbelt will not snapshot for you.

---

Getting Started

Write the IR runbook with preserve → revoke order; name IC rotation. Implement Talon/quarantine hooks on Part 9/Tetragon/integrity/injection signals. Stage forensics bucket and snapshot tooling; quarterly dry-run with a canary agent. Add circuit breakers with human-only reset. Practice one Explore path: alert → Langfuse → WORM → host events for a single traceId. Require learn → rule PR for every CRITICAL; store the signed review in WORM.

---

Companion: DevSecOps-boilerplate. Docs: PICERL / Talon · DR / BCP · SIEM.