In June 2026, a company called Nutrient disclosed a security incident that almost turned their observability stack into an attack runway. An attacker found a public Sentry DSN, submitted a fake bug report containing a malicious runbook, and an AI coding agent reading that feed nearly executed an npx command designed to exfiltrate secrets.
No credentials. No codebase access. Just a public write endpoint — and an agent that treated attacker-controlled text as instructions.
This post explains that failure from first principles, then walks through a complete open-source observability stack that closes the unauthenticated ingest vector, replaces Sentry, and hardens the trust boundary AI agents need around telemetry. Serverless — specifically a Cloudflare Worker with ephemeral JWTs — turns out to be the precise right tool for that ingest problem. That matters, because around the same time David Cramer (Sentry co-founder, @zeeg) posted that “serverless was a mistake.” Serverless has plenty of failure modes. This is not one of them.
The companion boilerplate — Docker Compose, Helm charts, Cloudflare Worker source, Faro init, Tetragon policies, and pre-built Grafana dashboards — lives at danielsmithdevelopment/DevSecOps-boilerplate.
Part 1: The Attack That Started This Conversation
What Happened at Nutrient (June 2026)
Here’s the incident, step by step:
- An attacker found the company’s Sentry DSN in their public JavaScript bundle. A DSN (Data Source Name) is a URL-like string that tells the Sentry SDK where to send error reports. It looks something like:
https://[email protected]/789. - The attacker used that DSN to submit a completely fake bug report to Sentry. They didn’t need to hack anything — the DSN is intentionally public and intentionally accepts submissions from anyone.
- The fake bug report contained a fake “runbook” — troubleshooting steps with a buried malicious
npxcommand designed to exfiltrate environment secrets (API keys, database passwords, and other sensitive configuration). - The company was running an AI coding agent that read from their Sentry feed as part of automated incident response. The agent read the fake runbook and nearly executed the malicious command.
This almost worked. The attacker needed no credentials, no access to any system, and no knowledge of the codebase. They only needed the DSN sitting in the public JavaScript bundle that any visitor could download.
Why the DSN Is Public by Design
The public DSN is not a bug. It’s a deliberate tradeoff.
Sentry is primarily used to capture errors from JavaScript running in a user’s browser. The browser is a public environment — any code you send there can be read by whoever runs it. There is no such thing as a server-side secret in client-side JavaScript. Given that constraint, Sentry made a reasonable choice: make the DSN write-only (anyone can submit events, nobody can read data) and accept that it’s effectively public.
Sentry documents this openly. They acknowledge that any user can send events to your organization with any information they want. Their mitigations include rotating the DSN if abused, filtering events, and restricting allowed domains.
For most teams, that risk was acceptable. The worst case was usually spam errors or a competitor learning which libraries you shipped. Annoying, not catastrophic.
Why AI Agents Change the Risk Profile Entirely
The Nutrient incident reveals a threat Sentry’s designers couldn’t have fully anticipated: AI coding agents that treat observability data as trusted instructions.
When a human engineer reads a bug report, they apply judgment. If a runbook says “run npx some-weird-package,” a human pauses. They might search the package, check authorship, or ask a colleague. Humans have built-in skepticism.
AI agents, depending on design, often don’t. If an agent is configured to read Sentry issues and take remediation actions, a sufficiently plausible fake issue can cause it to act on attacker-controlled text. The observability system becomes an injection point.
This is prompt injection — instructions buried in a data source an AI system reads, executed as if they were legitimate commands. It applies to any data source an agent reads: Sentry, log files, GitHub comments, Slack messages, or anything else.
The Root Failure: Logs Are Not Instructions
The most important thing to understand about Nutrient is where the failure actually lived. It was not in Sentry’s design. Sentry worked as documented. The failure was architectural: an AI agent treated external, attacker-controllable data as trusted instructions.
Hardening the ingest pipeline makes fake-event injection much harder. Hardening alone is not enough. Agents need trust boundaries that treat all observability data as untrusted input. We’ll return to that later.
Part 2: Understanding Observability — What It Is and Why It Matters
The Three (Now Four) Pillars of Observability
Observability used to mean three pillars: logs, metrics, and traces. Modern practice adds a fourth: profiles.
Logs are timestamped records of discrete events. Rich in context, great for debugging specific incidents, expensive to store and search at scale.
Metrics are numerical measurements over time — request rate, error percentage, latency. Compact, efficient, ideal for dashboards, alerting, and long-term trends.
Traces represent the full journey of a single request through a distributed system. When a click fans out across microservices and databases, a trace captures every hop, timing, and failure point.
Profiles capture fine-grained time spent inside code. Metrics say “this endpoint is slow.” A profile says “73% of CPU is inside this function.” Continuous profiling keeps that history queryable after the fact.
Why Correlation Is the Goal
Each signal answers different questions. Unified observability correlates them: elevated error rate → relevant logs → trace ID → full request path → CPU profile from the same window. That chain is what separates a useful platform from disconnected tools.
The Fifth Signal: Security Events
Modern stacks increasingly treat security events as a fifth signal. Runtime tools fire when an unexpected process starts, a protected file changes, or a container opens an outbound connection it shouldn’t. Those events belong in the same pipeline as application logs and traces — security incidents and application incidents are often the same incident from different angles.
Part 3: The Serverless Fix — Closing the Unauthenticated Ingest Vector
What’s Wrong with a Static Public Endpoint
The core problem with Sentry’s DSN model is that it creates a static public write endpoint. Anyone who knows the URL can submit data. As Nutrient showed, that becomes an injection vector for attacker-controlled text AI agents may act on.
The serverless solution eliminates the static endpoint. Instead of a permanent URL that accepts anything, you put a token-gated proxy in front that requires a valid, short-lived credential on every request.
What “Serverless” Means Here
“Serverless” still involves servers — cloud providers just manage them. In practice: small units of code that run on demand, scale automatically, and bill for usage rather than reserved capacity.
We’re using Cloudflare Workers — edge functions that run close to users worldwide. A Worker is a small JavaScript/TypeScript program responding to HTTP requests.
This is where Cramer’s “serverless was a mistake” framing becomes useful as contrast, not thesis. Serverless is a poor default for stateful, long-running, tightly coupled systems. For an ephemeral auth gate in front of telemetry ingest, it is exactly the tool you want: pay-per-invocation, globally distributed, minimal ops, and a natural place to force zero-trust validation.
JWTs: Short-Lived Credentials Explained
A JWT is a compact, cryptographically signed credential. It carries claims — who it was issued for, what they’re allowed to do — plus a signature that proves it wasn’t tampered with. A typical payload for this use case:
{
"sub": "session_abc123",
"project": "frontend-prod",
"origin": "https://app.example.com",
"iat": 1717600000,
"exp": 1717603600
}Anyone can read the claims (base64 JSON). Nobody can forge a valid token without the signing key that only your backend holds. The crucial field is exp: unlike a permanent DSN, the token becomes worthless after expiration.
The Ephemeral-Token-Gated Proxy Pattern
Step 1: Page load — backend issues a token. The browser asks your backend for a telemetry token. The backend authenticates the session, creates a JWT scoped to it, signs it, and returns it. The signing key never leaves the backend.
Step 2: SDK initialization — Faro points at the Worker. Grafana Faro is configured with your Cloudflare Worker URL and the JWT as an authorization credential — not a direct collector URL.
Step 3: Event capture — Faro sends events to the Worker. Exceptions, web vitals, and custom events go to the Worker with the JWT in Authorization.
Step 4: Worker validation — token, rate limit, payload. Validate signature, expiry, and claims (project, origin). Rate-limit by session. Reject unexpected fields or oversized payloads.
Step 5: Forward or drop. Valid events go to your private Grafana Alloy collector. Failures return HTTP 204 No Content and drop silently — don’t teach attackers why they failed.
Step 6: No static endpoint exists. Scraping the JS bundle reveals a Worker URL, not a writable DSN. Without a valid JWT, every attempt is dropped.
What This Pattern Covers — and What It Doesn’t
Honesty matters. This completely stops the Nutrient-style attack from a third-party attacker who found a DSN: no valid JWT, no injection path.
It does not — and cannot — stop a malicious or compromised user from injecting events. Any token issued to a browser is accessible to whoever controls that browser. DevTools → copy JWT → submit fake events until expiry. That residual risk needs detection, not impossible prevention: alert on client errors with no matching server-side trace, and watch agent observability for anomalous tool use.
Why This Beats Sentry Relay for This Threat
Sentry Relay is a solid self-hosted proxy if you’re staying on Sentry. Compared to an edge Worker for teams replacing Sentry:
- Relay is always-on infrastructure; a Worker is pay-per-invocation — often near-zero at typical error volumes.
- Relay requires deploy/monitor/patch/scale; Workers auto-scale globally with minimal ops.
- Relay is usually regional; Workers run in 300+ locations.
- Relay still presents a static URL. Auth is optional configuration. The Worker makes ephemeral JWT validation mandatory — there is no bypass path.
Part 4: The Full Replacement Stack
Goal: a production-grade platform covering frontend errors, backend traces, metrics, profiles, security events, AI agent behavior, and synthetic monitoring — unified, no proprietary lock-in.
The LGTMP Stack: The Core
Five Grafana Labs tools designed to work together: Loki, Grafana, Tempo, Mimir, Pyroscope.
Loki stores logs by indexing labels rather than full text, keeping content compressed in object storage. You query with LogQL. For Sentry replacement, Loki is where frontend errors land — and with fingerprinting (below), you get Sentry-style grouping.
Grafana is visualization and alerting. Its power is correlation: metrics → logs → trace ID → distributed trace → CPU profile.
Tempo stores distributed traces (OpenTelemetry, Jaeger, Zipkin) as Parquet in object storage.
Mimir is long-term Prometheus-compatible metrics at horizontal scale.
Pyroscope does continuous profiling and flame graphs so 3am CPU alerts have historical answers.
Grafana Alloy: The Collection Pipeline
Every signal flows through Grafana Alloy — Grafana’s OTel-based collector with River config that’s readable, reloadable, and GitOps-friendly.
Alloy batches, transforms, and routes: Faro events from the Worker, backend traces, Beyla metrics, Falco/Wazuh security events, Langfuse agent traces → Loki / Tempo / Mimir / Pyroscope.
otelcol.receiver.otlp "default" {
grpc { endpoint = "0.0.0.0:4317" }
http { endpoint = "0.0.0.0:4318" }
}
loki.source.syslog "falco" {
listener {
address = "0.0.0.0:1514"
protocol = "tcp"
}
forward_to = [loki.write.default.receiver]
}
otelcol.processor.batch "default" {
timeout = "5s"
send_batch_size = 1000
}
otelcol.exporter.loki "errors" { /* ... */ }
otelcol.exporter.otlp "tempo" { /* ... */ }
otelcol.exporter.prometheus "mimir" { /* ... */ }Grafana Faro: Frontend Observability
Grafana Faro replaces the Sentry JS SDK. It captures exceptions and stack traces, Core Web Vitals, custom events, and distributed trace context into backend Tempo spans.
In this architecture Faro sends to the Cloudflare Worker first. Alloy sees forwarded events as ordinary OpenTelemetry data.
Grafana Beyla: Zero-Code Backend Instrumentation
Beyla uses eBPF to instrument services without code changes — RED metrics and traces from HTTP/gRPC at the kernel level. Deploy as a DaemonSet and coverage appears across languages. It detects existing OTel instrumentation to avoid duplicate spans and propagates context correctly.
Langfuse: AI Agent and LLM Observability
Langfuse closes the Nutrient loop. Harden ingest all you want — if you can’t see what agents do, you can’t detect manipulation.
Langfuse (MIT, self-hostable) captures LLM calls, agent reasoning, tool invocations, and RAG retrieval steps. Unexpected tool use — like an npx the agent shouldn’t run — shows up as an anomalous span. Alert on it:
- alert: UnexpectedAgentToolUse
expr: |
count_over_time(
{service="langfuse"}
| json
| tool_name !~ "allowed_tool_1|allowed_tool_2|allowed_tool_3"
[5m]
) > 0
for: 0m
labels:
severity: critical
annotations:
summary: 'Agent invoked unexpected tool: {{ $labels.tool_name }}'Langfuse traces flow through Alloy into Tempo alongside app traces — injection → agent action → containment in one correlated view.
k6: Synthetic Monitoring and Load Testing
k6 covers CI load tests and production synthetic checks (can I still log in / checkout?). Combined with Faro RUM: what real users experience, and whether core journeys work at all.
Part 5: Replacing Sentry’s Error Grouping with Loki
The Problem: Every Stack Trace Is Unique
“User 12345 not found” and “User 67890 not found” are different log lines to Loki. Without normalization you get thousands of useless one-off streams.
Error Fingerprinting
Compute a fingerprint from error type, normalized message, top stack frame function, and filename. Use it as a Loki label:
export function createErrorFingerprint(event: ExceptionEvent): string {
const err = event.payload.exceptions?.[0];
const topFrame = err?.stacktrace?.frames?.at(-1);
const raw = [
err?.type ?? 'UnknownError',
normaliseMessage(err?.value ?? ''),
topFrame?.function ?? '',
topFrame?.filename ?? '',
].join('|');
return sha256(raw).slice(0, 16);
}
function normaliseMessage(msg: string): string {
return msg
.replace(/\b[0-9a-f]{8,}\b/gi, '<hash>')
.replace(/\b\d+\b/g, '<n>')
.replace(/https?:\/\/\S+/g, '<url>');
}Wire it through Faro’s beforeSend:
initializeFaro({
url: 'https://your-worker.workers.dev/ingest',
app: { name: 'frontend', version: __APP_VERSION__ },
beforeSend: (event) => {
if (event.type === 'exception') {
event.meta.labels = {
...event.meta.labels,
error_fingerprint: createErrorFingerprint(event),
release: __APP_VERSION__,
environment: __ENV__,
};
}
return event;
},
});Source Map Symbolication
Minified stacks need source maps. Options: symbolicate in the Worker (fetch maps from private R2 keyed by release) or in the Alloy pipeline from S3/GCS. Prefer Alloy at high volume so you don’t burn Worker CPU. Upload maps at deploy time:
- name: Upload source maps
run: |
aws s3 sync ./dist/sourcemaps \
s3://your-sourcemaps-bucket/${{ github.sha }}/Loki Retention Footgun
Set retention_period and max_look_back_period to the same value. If look-back is shorter, older queries silently return empty results:
limits_config:
retention_period: 744h
chunk_store_config:
max_look_back_period: 744hPart 6: Metrics — Mimir Configuration
Mimir defaults to ~10,000 samples/sec/tenant. Beyla across many services or high-cardinality labels can hit it silently — watch cortex_discarded_samples_total:
limits:
compactor_blocks_retention_period: 90d
ingestion_rate: 10000
ingestion_burst_size: 200000
max_label_names_per_series: 30Part 7: Secrets Management — Where Signing Keys Live
Option A: HashiCorp Vault
Use KV v2 for the JWT signing key; bootstrap Workers with one-time tokens:
vault secrets enable -path=observability kv-v2
vault kv put observability/worker/jwt-signing-key \
key="$(openssl rand -base64 32)"vault token create \
-policy=observability-worker-read \
-use-limit=1 \
-ttl=5mOption B: Cloudflare Workers Secrets
Simpler setups:
echo "your-signing-key" | wrangler secret put JWT_SIGNING_KEYAppropriate when you don’t need rotation automation, audit logs, or cross-service secret sharing.
Part 8: The Security Layer — Detection, Enforcement, and SIEM
Falco: Detection Without Enforcement
Falco watches syscalls via eBPF and alerts on suspicious activity: unexpected processes (npx, curl, wget), shells in containers, unexpected outbound destinations. Events flow Alloy → Loki. Falco detects; it does not stop.
Tetragon: Enforcement at the Kernel Level
Tetragon can kill processes and drop packets based on Kubernetes-aware policy:
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: block-package-managers
spec:
kprobes:
- call: 'sys_execve'
syscall: true
args:
- index: 0
type: 'string'
selectors:
- matchArgs:
- index: 0
operator: 'Postfix'
values:
- '/npm'
- '/npx'
- '/pip'
- '/pip3'
matchNamespaces:
- operator: NotIn
values:
- 'ci'
- 'build'
matchActions:
- action: SigkillStart in Monitor mode for about a week, baseline legitimate processes, then enforce. Tetragon spans land in Tempo via Alloy for forensic correlation.
Wazuh: Host Security and SIEM
Falco and Tetragon cover runtime inside containers. Wazuh covers the host: file integrity monitoring, log analysis, vulnerability detection, and active response.
For the Nutrient-class threat, the highest-value Wazuh piece is FIM on paths an exfiltrating process or compromised agent would touch — SSH keys, kubeconfigs, env files, package manager caches. In the boilerplate this ships under docker/observability/configs/security/; a focused ossec.conf fragment looks like:
<ossec_config>
<syscheck>
<disabled>no</disabled>
<frequency>300</frequency>
<directories check_all="yes" realtime="yes">/etc</directories>
<directories check_all="yes" realtime="yes">/root/.ssh</directories>
<directories check_all="yes" realtime="yes">/var/run/secrets</directories>
<directories check_all="yes" realtime="yes">/home/*/.kube</directories>
<ignore>/etc/mtab</ignore>
<ignore type="sregex">.log$|.swp$</ignore>
</syscheck>
<localfile>
<log_format>syslog</log_format>
<location>/var/log/auth.log</location>
</localfile>
<active-response>
<command>firewall-drop</command>
<location>local</location>
<level>12</level>
<timeout>600</timeout>
</active-response>
</ossec_config>Wazuh agents forward to the Wazuh manager, then into Alloy → Loki so host FIM alerts sit next to application errors and Tetragon kills in the same security dashboard.
Part 9: Alerting — Getting Paged at the Right Time
Grafana OnCall and PagerDuty
Grafana OnCall covers rotations, escalation, and Slack for teams starting fresh. Teams already on PagerDuty can use it as a Grafana contact point. Recommended split: critical → PagerDuty; warning → OnCall/Slack.
Alert Rules Worth Having Out of the Box
High frontend error rate:
- alert: HighFrontendErrorRate
expr: |
sum(rate({app="frontend", level="error"}[5m])) > 10
for: 2mNew error fingerprint — Sentry “new issue” equivalent: fingerprint seen in the last 10 minutes but not the prior 24 hours.
Client error with no matching server trace — detection for possible fake injection:
- alert: ClientErrorNoServerSpan
expr: |
count_over_time({app="frontend", level="error"}
| json
| traceID != ""
| absent_over_time(
{service="backend"} | json | traceID=`{{.traceID}}`[5m]
)
[5m]) > 0
for: 0m
labels:
severity: warning
annotations:
summary: 'Client error with no matching server trace — possible fake event'Part 10: The Agent Trust Boundary — The Part No Infrastructure Solves
Infrastructure hardens ingest, kills bad processes, and surfaces anomalies. None of that fixes an agent that treats external data as trusted instructions.
What “Trust Boundary” Means
A trust boundary is where data transitions from untrusted to trusted. Observability data — error messages, logs, alerts, Slack, GitHub comments — enters agents as untrusted input. Commands, package installs, and file modifications are trusted actions that need explicit authorization separate from the data that informed them.
Practical Rules for Agent Architectures
- Treat all observability data as untrusted input — even if it came from Loki you operate.
- Never execute commands found in data sources without human confirmation.
- Correlate before acting — real incidents create matching frontend/server/DB/metric signals; injected events usually don’t.
- Scope agent permissions tightly — read telemetry ≠ write production or shell access.
- Use Tetragon as a backstop — if the trust boundary fails, package-manager blocks still contain damage.
With this stack, Nutrient-class attacks fail at multiple layers: serverless proxy (no third-party token), Langfuse anomaly alert (unexpected tool), Tetragon (npx killed at the kernel).
Part 11: Putting It All Together — Architecture and Deployment
The Complete Data Flow
Browser (Faro + fingerprint + JWT) → Cloudflare Worker → Alloy. Backend services / Beyla → Alloy. Langfuse, k6, Falco, Tetragon, Wazuh → Alloy. Alloy → Loki / Tempo / Mimir / Pyroscope. Grafana correlates and pages OnCall / PagerDuty. JWT signing keys stay in Vault or Workers Secrets.
Deployment Structure
The DevSecOps-boilerplate provides:
- Docker Compose for local/dev and smaller self-hosted deployments (
docker/observability/), with an optional security overlay for Falco, Tetragon, and Wazuh. - Kubernetes / Helm-oriented layouts for production DaemonSets (Beyla, Tetragon, Falco).
docker/observability/
├── docker-compose.yaml
├── docker-compose.security.yaml
└── configs/
├── alloy/ loki/ tempo/ mimir/ pyroscope/
├── grafana/ # datasources, dashboards, alerting
└── security/ # falco, tetragon, wazuhConclusion: Tools Don’t Fail — Misapplied Defaults Do
“Serverless was a mistake” versus “that’s a skill issue” is a false dichotomy. Every pattern has a context. For eliminating a static public write endpoint that enables fake-event injection, serverless isn’t merely adequate — it’s the right tool: cheaper than always-on Relay, lower ops, globally distributed, and architecturally forcing zero-trust ingest.
The deeper lesson from Nutrient is about trust boundaries. The attack worked because an agent treated attacker-controlled data as a runbook. Infrastructure can reduce the blast radius. Only explicit agent architecture — what data may be read, what actions require confirmation, what must never be executable from telemetry — addresses the root cause.
Know what each layer covers. Know what it doesn’t. And remember: logs are not instructions.
If you’re building agents that need those trust boundaries baked in — hardened tool permissions, observability integrations, and production routing — that’s the problem space ClawQL is designed for. The complete observability starter for this essay lives at github.com/danielsmithdevelopment/DevSecOps-boilerplate.
Addendum: Three Configuration Refinements Worth Getting Right
1. Delta vs. Cumulative Metric Temporality in Alloy
Mimir expects cumulative metrics. Some OTel SDKs (Go especially) default certain types to delta — which can silently drop or corrupt. Convert explicitly:
otelcol.processor.transform "align_metrics" {
error_mode = "ignore"
metric_statements {
context = "metric"
statements = [
"set(temporality, \"Cumulative\") where temporality == \"Delta\""
]
}
}Put this between your OTLP receiver and Prometheus exporter from day one.
2. Web Crypto API for Fingerprinting in the Worker
Workers don’t have Node’s crypto module. Use crypto.subtle.digest. It also stays inside the Worker CPU budget during error storms:
async function createErrorFingerprint(event: ExceptionEvent): Promise<string> {
const err = event.payload.exceptions?.[0];
const topFrame = err?.stacktrace?.frames?.at(-1);
const raw = [
err?.type ?? 'UnknownError',
normaliseMessage(err?.value ?? ''),
topFrame?.function ?? '',
topFrame?.filename ?? '',
].join('|');
const encoded = new TextEncoder().encode(raw);
const hashBuffer = await crypto.subtle.digest('SHA-256', encoded);
const hashArray = Array.from(new Uint8Array(hashBuffer));
return hashArray
.map((b) => b.toString(16).padStart(2, '0'))
.join('')
.slice(0, 16);
}Await this correctly from Faro’s beforeSend.
3. High-Cardinality Label Management in Mimir
Drop per-instance labels before remote write:
prometheus.relabel "drop_high_cardinality" {
forward_to = [prometheus.remote_write.mimir.receiver]
rule {
action = "labeldrop"
regex = "container_id"
}
rule {
action = "labeldrop"
regex = "pod_template_hash"
}
rule {
action = "labeldrop"
regex = ".*_id"
}
}Deploy Beyla, watch cortex_discarded_samples_total and active series, then extend the drop list. Prefer labeldrop over replacing with static values when the label isn’t useful for queries.