Most teams trying to reduce LLM costs are having the wrong conversation. They swap models. They negotiate bulk pricing. They spend hours on prompt engineering. Then they re-run the benchmarks and find they’ve saved maybe 15%.
Those interventions plateau because they attack the wrong variable. In most agentic architectures the real driver isn’t which model you’re calling. It’s how much of the context window you waste before the model sees a single token of actual work.
That waste is not only a cost problem. Context quantity is the enemy of context quality. Overloading the prompt actively degrades retrieval, reasoning, and verification — often worse than giving the model less information. The twelve-layer stack that follows is therefore a performance architecture first; the bill drop is the side effect you can measure. Full product diagrams live at docs.clawql.com/architecture/token-efficiency.
The wrong conversation about LLM cost
Consider what happens when you give an agent access to three common enterprise APIs by loading full tool schemas into context:
| Provider | Operations | Full Spec Tokens |
|---|---|---|
| Google Cloud | 4,141 ops | ~84,000 tokens |
| Cloudflare | 2,697 ops | ~2,206,000 tokens |
| Jira | 336 ops | ~266,000 tokens |
| Combined | 7,174 ops | ~2,556,000 tokens |
Over 2.5 million tokens just to describe the tools — before the agent has done anything. That exceeds the context window of every production model in common use. The agent literally cannot load the tools and start reasoning in the same request.
Cloudflare’s own engineering team measured the same problem independently and arrived at a similar conclusion: roughly 1.17 million tokens for their API surface using an internal estimate. The two numbers measure slightly different artifacts (a full downloaded OpenAPI spec vs. an internal API surface estimate) but point to the same conclusion: the standard approach to tool-calling does not scale to real enterprise API surfaces.
This is the context bloat problem. Token pricing is only the bill you can see. Accuracy, hallucination rate, and time-to-first-token are the bills you pay quietly.
Lesson: if your “hello world” already exceeds the window, you do not have a model problem. You have an architecture problem — and it is already damaging intelligence, not just margin.
Why context bloat is a performance failure
The naive mental model treats tokens as linear cost: more input → proportionally higher spend. Engineering reality is worse on two axes at once. Bloated context costs more and makes the model less reliable on the task you hired it for. The evidence for the second claim is no longer anecdotal.
Lost in the middle
Liu et al. (Lost in the Middle: How Language Models Use Long Contexts, TACL 2024) documented a U-shaped performance curve on multi-document QA and key-value retrieval: models retrieve and reason far more reliably when the relevant evidence sits at the start (primacy) or end (recency) of a long prompt. Performance collapses when the same evidence is buried in the middle — even for models marketed as long-context.
The most uncomfortable finding for “just paste the docs” workflows: when relevant information is placed in the middle of a long context, multi-document QA performance can fall below the model’s closed-book baseline. In other words, stuffing the prompt can make the system worse than providing no documents at all. That is not a pricing quirk. That is a reasoning failure under load.
Tool schemas for thousands of operations you will never call in this turn are textbook middle-context noise. They push the task description, the current goal, and the useful few operations into positions where attention is least trustworthy.
Length alone hurts — even with perfect retrieval
A common rebuttal is: “If retrieval is perfect, long context is fine.” EMNLP 2025 findings disagree. Context Length Alone Hurts LLM Performance Despite Perfect Retrieval tested open- and closed-source models on math, QA, and coding. Even when models could perfectly retrieve every relevant token, accuracy still dropped 13.9%–85% as input length grew — while remaining well inside the advertised context limit.
The degradation held when distractors were replaced with whitespace, and even when irrelevant tokens were masked so the model was forced to attend only to the relevant slice. Sheer sequence length is enough. Trimming is not “tidying the prompt for finance.” It is reducing load on the attention mechanism so reasoning can complete.
Signal-to-noise dilution (context rot)
Large windows invite stuffing: dump the repo, dump the OpenAPI catalog, dump every prior turn. Transformers distribute attention across the full sequence. Every irrelevant schema line and unused JSON field competes with the actual instruction.
That dilution does more than lower retrieval scores. Reasoning Shift: How Context Silently Shortens LLM Reasoning (2026) finds that the same problem under lengthy irrelevant context can shrink reasoning traces by up to ~50%, with fewer self-verification and uncertainty-management behaviors (double-checks, “wait / alternatively / maybe” style recovery). Hallucinations and missed instructions rise because signal is fighting for the same attention budget as noise — and the model quietly skips the verification steps you’d rely on for hard agent work. Irrelevant tokens do not sit there passively. They interfere.
What engineers should take from this
Three practical frames land with builders:
Advertised window ≠ usable window. Every model has a marketed limit and a much smaller maximum effective context window — the span where quality remains acceptable. Operating near the sticker length is, by definition, operating in a high-hallucination zone. Plan for MECW, not brochure tokens.
Attention cost scales badly. In many architectures attention grows superlinearly with sequence length. Doubling context can more than double compute. Bloated prompts raise time-to-first-token and compound latency across multi-step agent loops — a performance tax before you count dollars.
The stuffing trap. When results look weak, the instinct is to add more documentation, more examples, more schema. That moves the useful bits toward the middle, lengthens the sequence, and worsens Lost in the Middle — a feedback loop that guarantees worse outcomes while the invoice climbs.
Lesson: curated, ranked, highly relevant context is a force multiplier for model intelligence. Maximize usable signal, not window occupancy. Cost savings follow.
By the time an agent is sifting ~2.5 million tokens of API schemas, it has buried the primary goal, suppressed verification, and invited Lost-in-the-Middle failure — if the request fits in the window at all. The twelve layers below exist to stop that before the first token of work.
The twelve layers
ClawQL addresses token waste at twelve distinct points in the request/response lifecycle. Because each layer targets a different kind of bloat, their gains compound rather than overlap — less context burned, sharper reasoning on what remains, lower spend as the side effect. The nearest MCP-gateway competitor typically implements one of these. Implementing all twelve produces a different order of magnitude.
- Code Mode — Two-tool pattern; full API specs stay on the server. Always on.
- Response trimming — GraphQL-style projection; responses pruned to consumed fields. Always on.
- Terse output processor — Strips hedging/filler from LLM responses in
clawql-inference. On. - Anthropic cache control — Stabilizes the system prefix for provider-side cache reuse (~10% of normal input cost). On.
- Semantic cache — Embeds requests; returns cached results for similar tasks and skips the model call. On when embeddings are configured.
- History distillation — Distills long transcripts into compact structured summaries. Opt-in.
- Prompt dedupe / truncation — Removes low-value tokens from the assembled prompt before send. Opt-in.
- Model escalation — Routes to the cheapest capable model (Frugal → Standard → Frontier). Opt-in.
- Structured output hints — Forces concise machine-readable formats. On.
- Token budget signaling — Passes
max_tokensas a soft brevity signal. On. - Assistant prefill opener — Pre-populates the response start to bypass hedging preamble. Opt-in.
- Fine-tuning flywheel — Production traffic → scrubbed export → custom model → new Frugal tier. Documented.
This pairs with the Hardened Agentic Stack work on secrets and ingest: efficiency without trust boundaries just makes insecure agents cheaper to run. Efficiency with trust boundaries is what makes sovereignty viable.
Layer 1: Code Mode — the foundation
This layer is always on. It’s the architectural basis for everything else.
Instead of thousands of JSON tool schemas, the agent gets two tools: search() and execute(). It searches for operations it needs, then writes TypeScript against a generated SDK to call them. The full API specs stay on the server.
The reasoning: large language models have seen enormous amounts of real TypeScript during training. They have seen comparatively little of the deeply nested bespoke JSON schemas that tool-calling typically uses. Code Mode plays to a strength the model already has.
Result: base tool-definition footprint of ~1,800 tokens regardless of API surface size.
| Provider | Full Spec | Code Mode | Reduction |
|---|---|---|---|
| Google Cloud | ~84,400 tokens | ~2,200 tokens | ~97% |
| Jira | ~266,600 tokens | ~900 tokens | ~99.7% |
| Cloudflare | ~2,206,000 tokens | ~2,400 tokens | ~99.9% |
| Average | ~852,000 tokens | ~1,800 tokens | ~99.8% |
A typical task uses around 60 of 7,000+ available operations — under 1% of the total surface. The other 99% never enters context.
Honest caveat: this asks the model to write working code, not fill a JSON template. Frontier models handle this reliably. Smaller models may produce syntax errors or type mistakes. Test on your actual workload, and keep traditional tool-calling available as a fallback.
Lesson: don’t ship the catalog into the prompt. Ship a capability to find and invoke.
Layer 2: Response trimming
Code Mode reduces what goes into the model. Response Trimming reduces what comes back — which matters because output tokens cost more than input on every major provider, and tool responses accumulate in history and get reprocessed on every subsequent turn.
When an agent calls an API, the raw response is typically a large nested JSON object full of fields it will never use. ClawQL analyzes the code the agent wrote to determine which fields it actually depends on, then trims the response to exactly those fields.
Concrete example — listing GKE clusters on Google Cloud:
- Raw response: ~421 tokens. Full cluster object including self-link, location, endpoint, version info, status, subnet, complete node pool configuration.
- Trimmed response: ~76 tokens. Name, status, endpoint, self-link only.
- Reduction: 82%.
Across representative workloads the average reduction is around 80%. Jira’s response format is particularly verbose — savings there are even higher.
Lesson: every unused field you keep today is a tax you pay again on every future turn.
Layer 3: Terse output processor
Layers 1 and 2 address structured data. Layer 3 addresses the natural-language filler that wraps everything.
Models prompted conversationally default to hedging: “I’d be happy to help with that! Based on my analysis, it looks like the issue might possibly be related to…”
None of that adds information. The actual signal — what’s wrong and what to do — is often a fraction of the response. The terse output processor strips filler while leaving code blocks, file paths, identifiers, and configuration untouched.
Before: “I would be absolutely happy to assist with that configuration issue! Based on my structural analysis of your active deployment codebase, it appears that the authentication middleware may be incorrectly handling the token object during handshakes. You should consider modifying the configuration block shown below…”
After: “Auth middleware mishandling token. Update config:”
Prose reduction varies with baseline verbosity — heavily hedged responses shrink by 80%+, already-terse responses less so. On average across typical developer-facing responses, roughly half to two-thirds of prose volume disappears.
Layer 4: Anthropic cache control
Most providers offer prompt caching: if the beginning of a prompt is identical to a previous request, the provider reuses cached computation. On Anthropic’s API, reading from a warm cache costs roughly 10% of normal input token price.
The catch: this only works if the prefix stays exactly the same between requests. In an unmanaged conversation that rarely holds — variable-size tool outputs, wandering prose, and growing history all shift where the stable part ends, breaking the cache.
Layers 1–3 are what make Layer 4 actually work:
- Layer 1 keeps tool definitions at a fixed ~1,800 tokens — they never change size.
- Layer 2 means tool outputs are small and consistently shaped, not multi-kilobyte blobs.
- Layer 3 keeps prose terse and consistent rather than varying by verbosity.
Combined, these stabilize early context enough for caching to take hold. Once it does, an increasing fraction of each subsequent call’s cost comes from cheap cache reads rather than full-price input processing. The longer a session runs, the bigger that fraction gets.
Lesson: cache hit rates are an architecture outcome, not a provider checkbox.
Layer 5: Semantic cache
Layer 4 makes repeated calls cheaper. Layer 5 skips some calls entirely.
In any extended agent session, certain sub-tasks recur constantly: checking a deployment’s status before a change, looking up a ticket before updating it, listing records before modifying one. Surrounding conversation differs, so exact-match caching won’t catch this. Intent is often functionally identical.
Approach: incoming requests are embedded and compared against previously cached requests by cosine similarity. If similar enough, return the cached result without calling the model.
Incoming Request → Embed → Check Cache (similarity ≥ threshold?)
↓ ↓
Cache Hit Cache Miss
Return Result Call Model → Cache ResultHonest note: savings depend entirely on workload repetitiveness. A pipeline checking the same handful of statuses repeatedly will see many hits. A workload where every request is genuinely novel will see few, and embedding every request may not be worth it. Measure on your actual traffic.
Critical safety rule: only read operations are cached. Writes, updates, and deletes always execute live. Any write invalidates cached reads for the affected resource — an agent should never act on stale state after a mutation. That rule is agent safety, not just cost control; see the trust-boundary series.
Layer 6: History distillation
Even with Layers 1–5 working, a multi-hour session accumulates a long transcript. Eventually the transcript itself becomes the dominant cost.
Fix: periodically distill message history into a compact structured summary — key facts, decisions, current state — and discard the raw transcript from the hot path, retaining a full copy in cold storage.
In a backend pipeline ClawQL fully controls, this happens automatically when history crosses a size threshold. Inside third-party clients (IDE integrations), ClawQL doesn’t control window management. There the approach is preventive: agents offload working state externally rather than letting it accumulate in the visible conversation.
Opt-in (CLAWQL_INFERENCE_HISTORY_COMPRESS=1).
Layer 7: Prompt dedupe and truncation
After history distillation, this layer inspects the fully assembled prompt — system instructions, tool definitions, memory snapshot, current request — right before send, and removes lower-value tokens while preserving meaning.
Important framing: general-purpose prompt compression tools report 3–8× compression against raw, unoptimized prompts. A prompt that’s already been through Layers 1–6 is already far leaner. Layer 7 on top produces an additional 20–40% reduction — real, but not the headline number from a compression benchmark. That’s the honest comparison.
Opt-in (CLAWQL_INFERENCE_PROMPT_COMPRESS=1). Only works in environments ClawQL fully controls.
Layer 8: Model escalation
Not every step in a multi-step task needs the most capable model. Checking a status, validating a schema, filtering a list, writing a well-scoped snippet — these don’t need the same model as complex multi-step planning.
Model escalation sends tasks to the cheapest model capable of handling them, escalating only when warranted:
| Tier | Example models | Role |
|---|---|---|
| Frugal | Phi-4 14B / local Ollama | Utility: metadata, status checks, list filtering, pre-processing. Zero cloud cost when local. |
| Standard | Qwen3.6-27B / Groq / Together | Primary document work, extraction, code generation, classification. |
| Frontier | Claude / GPT-4 class | Complex multi-step reasoning, cross-document synthesis, ambiguous edge cases. |
Escalation logic: Frugal failure or low confidence → Standard. Standard failure → Frontier. Decomposed sub-tasks always start at Frugal. Top-level orchestration starts at Standard.
Every routing decision is WORM-logged: which tier was selected, which failure signal triggered escalation, what the cost was. Auditable cost optimization, not a black box — the same forensic posture as the secrets flywheel in Part 2.
Opt-in (CLAWQL_INFERENCE_ROUTING_ENABLED=1).
Don’t confuse this with agent coordination. Model escalation chooses which model tier runs a step. When you need multiple agents on the same problem, ClawQL’s term is agent coordination — not “mixture of agents” / MoA.
Naive multi-agent mixtures fan out work and hope diversity emerges. Agent coordination treats the swarm as a control problem: NSV (Normalized Semantic Variance) is a cheap tripwire for cognitive convergence in embedding space; when NSV crosses the calibrated threshold, SGDOP (Semantic GDOP) recovers the exact blind-spot direction the swarm isn’t exploring, so recruitment targets that axis instead of adding another correlated agent. The result is mathematically guided diversity and faster convergence — not another roll of the dice on ensemble noise.
That layer sits beside this essay’s cost stack (strategic coordination via Ouroboros), not inside Layer 8. Layer 8 still saves money on a single call path. Agent coordination decides when and how the swarm should recruit.
Layer 9: Structured output hints
Operate inside clawql-inference where the model generates its response. Force concise machine-readable formats rather than asking for natural language and cleaning up afterward. Eliminate filler at the source. On by default.
Layer 10: Token budget signaling
Pass max_tokens as a soft brevity signal. Measurably reduces verbosity on current models, especially open-ended explanation. On by default.
Layer 11: Assistant prefill opener
Pre-populate the start of the response to bypass hedging preamble. Small saving per call; compounds at volume. Opt-in.
Layer 12: The fine-tuning flywheel
This is where the architecture stops being a static configuration and becomes a self-improving system.
Every other layer eventually hits a floor. You can compress input, cache repeats, route to cheaper models — but the cheapest capable model is still a general-purpose model being asked to do your task. The Flywheel takes production traffic and turns it into a custom model better at your workload and cheaper to run than anything generic.
Step 1: Verdict-filtered export
Not all production traffic is good training data. A call that succeeded on the third retry after two failures is not a good positive example. Export only where the evaluator confirmed correctness:
clawql inference export \
--verdict passed \
--format openai-jsonl \
--output ./training-data/$(date +%Y-%m).jsonlResult: high-quality examples from your actual production workload — not synthetic benchmarks.
Step 2: PII sanitization
If agents process contracts, invoices, medical records, or financial reports, training data contains sensitive information. Presidio scrubbing runs before anything leaves the system:
clawql inference export --verdict passed --scrub-pii --format openai-jsonlNon-negotiable before any external fine-tuning API. The WORM manifest records which Presidio rule version scrubbed each export — the same evidence posture as immutable releases and secret audit trails.
Step 3: Local training (your data graveyard becomes an asset)
Most teams have years of agent logs in cold storage. That’s not only a compliance cost — it’s an unleveraged training set for a domain-tuned model.
Example local path (Unsloth + QLoRA on an RTX 5090-class box):
- Qwen3.6-27B in 4-bit: ~14–16 GB VRAM, leaving headroom on a 32 GB card
- ClawQL-general adapter on 500–2,000 examples: ~2–4 hours
- Training cost on owned hardware: $0
Vertical adapters — legal, commercial lending, healthcare admin — produce the sharpest gains. A model that has seen thousands of MISMO-format mortgage documents processes the next one differently than a generic model seeing it cold.
After training, merge the adapter and re-quantize for production serving (e.g. NVFP4).
Step 4: Register and route
clawql inference finetune register \
--model ./clawql-qwen-legal-v1.gguf \
--tier frugal-custom \
--domain legalTasks that would have burned Standard or Frontier for that domain now route to your custom Frugal model. Model escalation picks it up automatically. Cost drops. Quality — measured against your task types — rises because training data was your task types.
The virtuous cycle:
Production Traffic
↓
Langfuse Verdict Filtering → Only successes become training data
↓
Presidio PII Scrubbing → Clean, safe export
↓
QLoRA Fine-tune on Local Hardware → Domain-specific adapter
↓
Register as Custom Frugal Tier → Lower cost, higher quality
↓
Better production outputs → Better verdicts → Better training data
↓
RepeatSecurity pre-condition: an attacker who poisons training data with injected instructions doesn’t just affect one conversation — they affect every future model trained on that corpus. Steganography / instruction-hiding detection at document ingest must sit upstream of Layer 12 collection. Efficiency without that gate is how you train a compromised Frugal tier.
Lesson: your historical agent logs are not only a storage line item. They are the cheapest path to a model that beats generic APIs on your work.
The compounding math
A GKE cluster listing that would normally consume 400+ tokens in a naive agent:
- After Layer 1: tool definitions are ~2,400 tokens vs. ~84,400 — and the operation is found cleanly.
- After Layer 2: ~76 tokens in the response vs. ~421 enter history.
- After Layer 3: natural-language wrapper is stripped to essentials.
- After Layer 4: tool definitions cost ~10% of normal on subsequent calls in-session.
- After Layer 5: if this status check ran before, it doesn’t hit the model at all.
- After Layer 8: utility task routes to Frugal, not Frontier.
- After Layer 12: after training cycles, your fine-tuned Frugal model may handle this better than the generic Standard model ever did.
Final cost isn’t “~99.8% cheaper on input schemas” plus a bit — the gap widens with session length and flywheel cycles. Layer 1 is a static win. Layer 12 compounds.
This is why implementing only Layer 1 — which most MCP gateways approximate — leaves the majority of gains on the table. You’ve cut input schemas. You’re still paying for verbose outputs, repeated calls, expensive models on simple tasks, and a general model that never specializes.
Observability: proving the savings
Efficiency claims without measurement are marketing. The LGTM+ stack (Loki, Grafana, Tempo, Mimir, Pyroscope) — the same orientation as the serverless observability essay — provides the proof:
Inference cost dashboard: Per-request token spend by layer. Layer 5 hit rate tells you whether semantic caching pays. Layer 8 escalation rate tells you how often tasks exceed Frugal capacity.
Model escalation traces: Every escalation in Tempo — which tasks escalated, which signal fired, whether justified. High escalation on one task type is a domain fine-tune signal.
Flywheel quality metrics: Before/after each cycle, Langfuse verdict rates. Canary promotion fails if the tuned model regresses on production traffic.
Security tie-in: Every fine-tuning export through Presidio generates a WORM entry with document hashes, redaction rule versions, and timestamps. If you process sensitive documents, you can prove what entered training and that it was scrubbed.
Known trade-offs
Code Mode requires a capable model. Savings are real; the model writes code rather than filling JSON. Test before depending on it in production.
Semantic caching isn’t universally beneficial. Embedding latency on every request vs. sometimes skipping model calls. Measure.
Layers 6 and 7 have limited effect inside third-party clients. In IDE AI integrations you don’t own the window manager; you can slow accumulation but not compress history you don’t control.
Layer 7 numbers depend on baseline. 20–40% additional on an already-compressed prompt is the honest post-Layers-1–6 figure — not the 3–8× raw-prompt marketing number.
The Flywheel needs production data. Day one you’re on the base fleet. The efficiency gap vs. a simpler system widens as verdict history accumulates.
Fine-tuning can degrade out-of-domain performance. A legal adapter improves the next legal document; it may worsen cooking Q&A. Keep the base model available for out-of-domain routing.
Efficiency is the path to sovereignty
There’s a dependency that doesn’t get stated often enough: keeping your data in your own infrastructure requires that your own infrastructure is economically viable. If self-hosted inference costs ten times OpenAI and underperforms because the context is bloated, economics eventually win. Data moves not because of malice but because of cost and quality.
The twelve-layer stack answers both problems. By cutting what enters context — and making what remains high-signal — you restore reasoning quality while driving per-inference cost to a fraction of the naive approach. The Flywheel then makes that cost fall further over time.
After several Flywheel cycles on your domain, a Frugal-tier model can outperform a generic Frontier API call on your specific workload — at a fraction of the cost, inside your infrastructure. That’s the endpoint: a sovereign inference system that gets more accurate and cheaper as it runs, on data that never leaves your control.
Pair that with immutable releases for what you ship, and hardened ingest/secrets for what your agents trust, and you have a stack where accurate, cheap, local, and safe stop being trade-offs.
Getting started
Code Mode and Response Trimming live in the ClawQL agent runtime. The inference gateway exposes Layers 3–11 as a drop-in OpenAI-compatible proxy — fastest path to reading savings on your workload:
# Install the inference gateway
npx clawql-inference
# Point any OpenAI SDK at it
export OPENAI_BASE_URL=http://localhost:8080/v1
# Inspect which layers the policy enables
clawql inference policy show
# After a session, inspect spend by layer
clawql inference spend --group-by layerCombine the agent runtime (Layers 1–2) with the inference gateway (Layers 3–11) and you get the full stack end to end. Architecture, IDP pipeline, and steganography detection upstream of the flywheel: Token efficiency architecture.