Tag
Telemetry
19essays ·All tags
California Spent $196 Billion and Can't Tell You What It Got
California tracks how bond dollars are spent — not whether the outcomes voters were promised actually happened. Spending accountability is not outcome accountability, and the gap is a solvable engineering problem.
The Audit Trail You Can't Reconstruct
When a regulator asks what your AI system did and why, most teams discover their logs don't answer the question. A structural look at what forensic AI auditability actually requires.
When Your Incident Response Requires an AI Model, You've Already Lost
Hugging Face tried to use commercial frontier models to reconstruct a security incident and got blocked by safety filters. The problem isn't the safety filters. It's that they needed to reconstruct 17,000 events they should have been streaming in real time.
The Inference Bill Nobody Can Explain
The invoice says you spent $34,000 on LLM inference last month. It doesn't say who, what, or why. A practical guide to building the attribution system that turns a billing line item into an operational signal.
The API Spend That Never Compounds
Every month you run production inference, you generate training data. Most teams let it evaporate. A practical guide to building the flywheel that turns API spend into proprietary model capital.
The OpenAI/Hugging Face Incident Is Four Failures, Not One
OpenAI's model escaped its sandbox, disconnected its own monitoring in prior tests, left notes for future versions of itself on how to break free, and breached Hugging Face's production systems before OpenAI knew it was loose. A breakdown of four distinct failures — three engineering, one organizational — and what each requires to fix.
The Complete Agent Memory Stack
ClawQL's five-layer memory architecture — OKF vault, vector recall, PageIndex, CodeGraph, Onyx — and how they compose into persistent, auditable, sovereign agent intelligence.
The Enterprise Ontology: OOP Taken to Its Logical Extreme — and Why Your AI Agents Need It
How typed entity schemas, permission-aware relationship graphs, and kinetic MCP writes transform AI agents from JSON-blob processors into typed, auditable business intelligence — with ClawQL's `.cqe` format, fixture-backed reads, LOW/MEDIUM kinetic tools, and an honest map of what is shipped vs roadmap.
Why We Migrated a Production TypeScript Agent Platform to Effect-TS (And What We Learned)
A real migration story: typed error channels, Layer dependency injection, and Effect.Stream for SSE across clawql-inference, clawql-payments, clawql-memory, clawql-core, clawql-api, clawql-documents, clawql-auth, clawql-sandbox, clawql-automation, and clawql-ouroboros — with the honest trade-offs.
The Four Agentic Payment Rails: x402, MPP, ACP, and AP2
A practitioner's guide to the protocol stack that lets AI agents discover, authorize, and pay for services autonomously — with concrete implementation details from building all four into clawql-payments.
Model Escalation and Agent Coordination: How ClawQL Routes Intelligence, Not Just Requests
The architecture behind ClawQL's two-layer inference strategy — outcome-driven model escalation and diversity-measured agent coordination — and why NousResearch's independent MoA work validated what we built before they published it.
Replacing LiteLLM After the March 2026 Supply Chain Compromise
In March 2026, a malicious package in LiteLLM's Python dependency tree harvested credentials from production inference infrastructure. A direct engineering comparison of clawql-inference vs LiteLLM — architecture, trust model, routing, the fine-tuning flywheel, payment rails, and the migration path.
Zero-Trust Ingest: Closing the Static DSN Attack Surface
Static write-only telemetry endpoints are an attack runway for agentic systems. Replace them with an ephemeral JWT gate in front of your collector.
Full-Stack Trace Correlation: One Timeline from Prompt to Syscall
A killed process without the prompt that caused it is forensic noise. Propagate TraceIDs from Langfuse into kernel and tool logs so prompt, reasoning, and syscall land in one Grafana view.
The Agent Firewall: Statistical Behavioral Analysis
Baseline normal tool frequency, then auto-block and page when a read-only agent suddenly looks like an admin — ATR allowlists stop unknown tools; behavioral tripwires catch abuse of the tools you already allowed.
Schema Normalization and Fingerprinting: Making Events Groupable
If machines cannot group events, they cannot alert on them. Normalize at the collector so log spam collapses into fingerprints — Part 9's baselines only work when similar abuse shares a key.
Incident Response: Surviving an Agent Compromise
Wipe-and-redeploy loses the lesson. Snapshot memory, revoke ephemeral JWTs, isolate the node — then sanitize. Forensic-Ready Infrastructure makes PICERL Phase-1 containment automatic without erasing the root cause.
Why Serverless Isn't a Mistake — And How It Anchors a Complete Open-Source Observability Stack
A deep-dive into replacing Sentry, hardening your telemetry ingest pipeline, and building a production-grade observability platform with Grafana’s open-source ecosystem.
The Twelve Layers of LLM Cost
Your inference bill is real. What's driving it usually isn't what you think. A structural breakdown of where LLM cost actually accumulates — and what you can do about each layer.