Full-Stack Trace Correlation: One Timeline from Prompt to Syscall

A killed process without the prompt that caused it is forensic noise. Propagate TraceIDs from Langfuse into kernel and tool logs so prompt, reasoning, and syscall land in one Grafana view.

This is Part 8 and the start of Phase 3: Telemetry and Observability. Parts 4–7 made bad actions fail closed — kills, EPERMs, path denies, dead sandboxes. This post answers the residual question: when those controls fire, can you see the prompt and tool decision that caused them in the same timeline, or is each signal an orphan pager?

---

The Kill With No Story

Tetragon does its job:

SIGKILL  binary=/usr/bin/npx  parent=/usr/bin/node  exit=137

PagerDuty fires. An engineer opens Langfuse: dozens of overlapping sessions. Loki has the kill — no sessionId. Tempo has an agent span from “around then” — different ID encoding. Panguard’s WORM row says allow(run_tests) for a session that might be related.

The only question that matters — what did the model believe it was doing? — has no answer. Phase 2 without Phase 3 is a well-instrumented shrug.

A policy kill without a join key is forensic noise dressed as security.

---

Five Signals, One Story

Phase 2 produced distinct sensors. Phase 3’s job is to refuse silos.

ClawQL’s security monitoring module treats security events as first-class telemetry with a canonical schema: schemaVersion, eventId, timestamp, source, principal, event, detail, traceContext, payloadHash. Schema version is pinned in SIEM rules — drift breaks detection silently.

---

The Architecture Pattern: Full-Stack Trace Correlation

One trace id at the start of an agent turn — Langfuse span or OTel root. Propagate it down into every child: MCP handler, Panguard decision, tool sidecar env, Fluent Bit labels. Propagate it sideways into host sensors where the runtime allows (pod labels, cgroup annotations, OTEL_RESOURCE attributes). Store LLM detail in Langfuse/Tempo; store denials in WORM + Loki; join on ids, never by clock skew alone. Tail-sample so any trace with a block, kill, integrity fail, or AUTH deny is retained at 100%.

Secrets, raw prompts, and file contents never go into Prometheus labels or high-cardinality metric tags.

User turn
   │
   ▼
Langfuse / OTel root span   traceId=T  sessionId=S
   │
   ├─► Panguard decision     (T,S) → WORM
   │
   ├─► Tool sidecar Job      labels: clawql.io/trace=T
   │         │
   │         └─► stdout/log  → Alloy → Loki (T,S)
   │
   └─► Host sensors
             Tetragon / Falco / seccomp / Wazuh
             enrich with T,S from pod/process context
                    │
                    ▼
         Grafana: one Explore path for T

If the kill arrives without T, propagation is incomplete — the kill-switch exists without the story it needs to be useful.

---

Propagating the ID

Minimum contract for every agent turn:

Injection sketch when dispatching a tool:

const traceId = langfuseTrace.id;
const sessionId = atr.sessionId;

await startSandboxJob({
  labels: {
    'clawql.io/trace': traceId,
    'clawql.io/session': sessionId,
    'clawql.io/agent': atr.agentId,
    'clawql.io/tool': toolName,
  },
  env: {
    OTEL_SERVICE_NAME: 'clawql-tool-sandbox',
    TRACEPARENT: `00-${traceId}-${spanId}-01`,
    CLAWQL_TRACE_ID: traceId,
    CLAWQL_SESSION_ID: sessionId,
  },
});

For in-process handlers, set the same env before spawn so child processes inherit the join key. Fluent Bit / Alloy label copy:

attributes:
  actions:
    - key: clawql.trace_id
      from_context: CLAWQL_TRACE_ID
      action: upsert
    - key: clawql.session_id
      from_context: CLAWQL_SESSION_ID
      action: upsert

sessionId and agentId belong in WORM/logs, not in Prometheus label sets. High-cardinality metric tags become an exfil and cardinality bomb.

---

Canonical Events for Every Phase 2 Deny

{
  "schemaVersion": "1.0",
  "eventId": "uuid",
  "timestamp": "2026-07-14T18:00:00.000Z",
  "source": {
    "component": "tetragon",
    "cluster": "prod-a",
    "namespace": "clawql-agents"
  },
  "principal": {
    "agentId": "agent_123",
    "sessionId": "sess_456",
    "tenantId": "t_1"
  },
  "event": {
    "type": "POLICY",
    "subtype": "UNAUTHORIZED_EXEC_KILLED",
    "outcome": "BLOCKED",
    "severity": "HIGH"
  },
  "detail": {
    "policy": "clawql-kill-unauthorized-child-exec",
    "binary": "/usr/bin/npx",
    "parentBinary": "/usr/bin/node",
    "tool": "run_tests",
    "panguardDecision": "ALLOW"
  },
  "traceContext": {
    "traceId": "abcdef0123456789abcdef0123456789",
    "spanId": "0123456789abcdef"
  },
  "payloadHash": "sha256:…"
}

The same envelope covers PANGUARD_BLOCK, UNAUTHORIZED_EXEC_KILLED, SYSCALL_DENIED, PATH_DENIED, FIM_CHANGE, SANDBOX_STARTED, SANDBOX_EXIT, and SANDBOX_EGRESS_DENIED. Seven event types (AUTH, TOOL_CALL, MEMORY, SKILL, AGENT, NETWORK, POLICY) keep SIEM routing predictable. Pin schemaVersion in correlation rules — a field rename cannot silently empty the NOC.

---

The Investigation Path

One Grafana-shaped loop — the order matters, the tool names vary:

1. Alert lands with traceId or sessionId — never “check Loki somehow.”
2. Langfuse / Tempo: open the trace, user turn → model reasoning span → tool span.
3. Panguard WORM: filter traceId=T — allow or block, ruleId, HITL?
4. Loki: {clawql_trace_id="T"} or | json | traceId="T" — sandbox logs, seccomp, Tetragon.
5. Decision: injection, bug, missing allowlist, or true attack — quarantine / ATR change / image fix.

Correlation patterns worth shipping on day one:

---

Tail-Based Sampling

Ordinary traffic can sample. Security cannot.

Retain 100% of any trace containing: Panguard block, WORM write for POLICY/AUTH deny, Tetragon kill, FIM integrity fail, sandbox egress deny, HITL deny. Sample the happy path aggressively so Tempo bills don’t train people to turn tracing off. Restrict who can query security traces that still carry ATR/session context — same ACL posture as the WORM store.

If the sampler drops the kill’s parent trace, you recreated the original problem with better charts.

---

Pipeline

Langfuse SDK ──┐
Panguard WORM ─┼─► Alloy / OTel Collector ─┬─► Tempo (traces)
Tetragon/Falco─┤                           ├─► Loki (logs + security events)
seccomp audit ─┤                           ├─► WORM (immutable decisions)
Wazuh FIM ─────┤                           └─► Mimir (low-cardinality metrics only)
sidecar logs ──┘
                     Grafana Explore / NOC

Presidio redaction before anything less trusted than WORM sees payloads. payloadHash correlates without re-exposing secrets — especially for Part 6 path denials.

---

Honest Failure Modes

Clock skew is not a join key. Always propagate ids; use time only as a secondary filter.

Env injection vs untrusted children. CLAWQL_TRACE_ID in the environment helps sensors and is visible to compromised tool code. Acceptable for a join key — never put Vault tokens in the same channel.

Host sensors without pod context. Bare-metal agents need an enrichment hop; naked Tetragon events don’t carry session context automatically.

Dual id systems. Langfuse ids vs raw OTel hex — pick a canonical 32-char hex form and convert at the edge so Loki queries stay boring.

Dashboard theater. A wall of panels without an alert that includes traceId fails the purpose of Phase 3.

Privacy. Full prompts in shared Tempo may be too much for a broad eng org. Keep raw prompts in Langfuse with tighter ACL; put hashes and tool names in the cross-cutting security event.

---

Getting Started

Define the canonical security event schema version; publish it next to ATR docs. Ensure Langfuse or OTel creates a root traceId per turn and threads it through the gateway. Label every sandbox Job and inherit env into tool processes. Enrich Tetragon/Falco/Wazuh through Alloy with traceId/sessionId from pod/process context. Build one Explore runbook and a canary alert that includes the join key. Turn on tail-based 100% keep for POLICY/AUTH denies and kills; strip secrets at the collector.

Part 9: once events join, baseline normal tool frequency and trip when a read-only agent suddenly looks like an admin.

---

Companion: DevSecOps-boilerplate · Observability essay. Docs: Security monitoring / SIEM.