Your inference bill is real. What’s driving it usually isn’t what you think. A structural breakdown of where LLM cost actually accumulates — and what you can do about each layer.
A breakdown of where LLM cost accumulates structurally, with concrete numbers for each layer. Reference implementation at docs.clawql.com/architecture/token-efficiency. This pairs with Both Sides (search → execute on input and output), the inference gateway patterns in Hardened Agentic Stack, and with model escalation and the agent memory stack for Layers 7–8 and 12.
The Bill That Surprised Everyone
In March 2026, a mid-size engineering team at a fintech company sent their VP of Engineering a screenshot of their LLM spend for the month. $34,000. Roughly 2,000 daily active users. Standard chat completions. No multimodal, no image generation.
The VP asked: “What are we actually paying for?”
Nobody had a precise answer. The dashboard said “tokens.” The breakdown said “input” and “output.” They knew the model, the provider, the month. They didn’t know which feature drove which cost, which team was responsible, or what changed between February ($9,000) and March ($34,000).
Most teams discover LLM cost as a surprise rather than a managed variable. The billing model appears simple: pay per input token, pay per output token. What it obscures is that the number of tokens you send and receive is downstream of at least twelve distinct decisions in your system’s architecture. Most of those decisions were never made explicitly. They’re defaults, inherited patterns, or features that seemed cheap when tested in isolation.
This post breaks down each of the twelve layers. For each one: what it is, how to measure it, what it costs at scale, what to do about it.
Lesson: the billing is simple. The cost drivers aren’t.
Why Routing Alone Doesn’t Fix It
The market’s current answer to LLM cost is routing. Send expensive queries to expensive models, cheap queries to cheap models, save 20–60% on average. Ramp Router (July 23, 2026) and Cursor Router (July 22, 2026) both make this claim with real numbers behind it. It’s a real lever.
It’s Layer 8 of twelve.
Fixing Layer 8 without fixing Layers 1–7 means routing bloated, redundant, uncached prompts to cheaper models. The cheaper model still processes all that context. A 60% routing saving on a prompt that’s 3× larger than it needs to be is a 60% saving on 3× the cost. You’re still paying 20% more than a team that fixed both.
The layers interact. Fixing them in the wrong order leaves money on the table.
Lesson: routing is a multiplier on your current token count, not a substitute for reducing it.
Layer 1: Context Bloat
The most common and most expensive mistake: sending things to the model that the model doesn’t need to see.
A GitHub API spec is roughly 2.5 MB of JSON. Sending it into context so the model can answer “create an issue” costs approximately 625,000 input tokens. At $3 per million input tokens, that’s $1.88 per call. At 1,000 calls per day, $56,400 per month. The model needed maybe 200 tokens: the operationId, the required parameters, the endpoint path. The other 624,800 tokens were noise.
The same pattern appears with document ingestion (sending full PDFs instead of extracted relevant sections), RAG (returning 20 context chunks when 3 would answer the question), conversation history (sending the full transcript instead of a decision-relevant summary), and tool definitions (registering 50 tools when a given query needs 2).
Measuring context bloat:
# Log average input token count per call type
clawql inference spend --group-by operation --field input_tokens
# From raw call store
cat calls.jsonl | jq 'select(.operation == "github_issue") | .input_tokens' | sort -n | uniq -cThe fix is the search + execute pattern. search() returns only the relevant operation slice from loaded specs, not the full document. The model sees 200 tokens instead of 625,000. This is a structural fix at the API bridge layer. For RAG, fix the retrieval before touching the model call. A top_k=3 retriever with a good embedding model almost always outperforms top_k=20 with a mediocre one and costs 85% less at the context boundary.
Lesson: measure what you’re sending before tuning what you’re sending it to.
Layer 2: Verbose Responses
Input tokens are usually the larger cost driver. Output tokens are often billed at 3–5× the input rate.
The default behavior of large frontier models is verbose. Ask “what’s the capital of France?” and a well-aligned model explains that the question is about geography, notes that France is a country in western Europe, and arrives at Paris after a sentence of context. That’s not malfunction. It’s optimization for helpfulness and thoroughness. Helpful and thorough is expensive.
Prompt constraints help in testing and are unreliable at scale. “Answer in one sentence. No preamble.” is advice, not enforcement. A model under ambiguous context will occasionally revert regardless of what the system prompt said.
Three mechanisms are more reliable than prompt advice:
Structured output enforcement. When the response format is specified as a JSON schema or XML template, the model produces structured output because it has to match the schema. Output is structurally bounded.
Response streaming with early termination. For use cases where you need the first N tokens, streaming lets you close the connection when you have enough. Eliminates tail verbosity on responses where useful content is early.
GraphQL projection. When the response is structured, a projection layer extracts only the fields you need before content hits the application layer. Doesn’t reduce billing on the current call, but eliminates passing verbose responses to subsequent model calls where they become context bloat in Layer 1.
Measuring output verbosity:
cat calls.jsonl | jq 'select(.operation == "summarize") | {input: .input_tokens, output: .output_tokens, ratio: (.output_tokens / .input_tokens)}' | jq -s 'sort_by(.ratio) | reverse | .[0:10]'High output-to-input ratios on tasks that should be brief are the diagnostic. A classification call with a 3:1 ratio on a 500-token input is a verbose response problem.
Lesson: structure the response format at the infrastructure layer. Prompts are advice. Schema enforcement is a constraint.
Layer 3: Redundant Static Context
Most system prompts don’t change between calls. Tool definitions don’t change. Persona instructions don’t change.
Sending the same 5,000-token system prompt on every call in a session of 100 turns is 500,000 input tokens of static content the model has already seen. Providers that support prompt caching can eliminate most of this cost. The first call pays for the static context. Subsequent calls pay a fraction.
Anthropic cache_control:
messages = [
{
"role": "user",
"content": [
{
"type": "text",
"text": system_prompt,
"cache_control": {"type": "ephemeral"}
},
{
"type": "text",
"text": actual_query
}
]
}
]Cache hits are billed at roughly 10% of the standard input rate. A 5,000-token system prompt that caches on 99 of 100 calls costs about 10× less than one that doesn’t cache.
The structural requirement: content must be stable within the cache window. Dynamic content mixed into the system prompt breaks caching. Keep static content (persona, tool definitions, background documents) in separate blocks from dynamic content (session state, retrieved context, user-specific data).
Measuring cache miss rate:
cat calls.jsonl | jq 'select(.cache_hit != null) | {hit: .cache_hit, tokens: .input_tokens}' | jq -s '{total: length, hits: [.[] | select(.hit == true)] | length, hit_rate: ([.[] | select(.hit == true)] | length) / length}'A team with zero intentional caching is often leaving 40–60% input cost on the table for high-session-volume workloads.
Lesson: static context should be paid for once per session, not once per call.
Layer 4: Unstructured Output
When the model’s response needs to be parsed or used by downstream code, unstructured output costs twice: once in output tokens (verbose natural language is longer than structured data) and once in the follow-up call when the first response wasn’t quite right.
# Expensive: verbose + parse cost
prompt = "Extract the invoice total from this document. Tell me what you found."
# Response: "Based on my analysis of the document, the invoice total appears to be $1,234.56,
# which I can see on line 23 of the document in the 'Total Due' field..."
# Cheap: structured, parseable, correct
prompt = 'Extract the invoice total. Respond only: {"total": "<amount>"}'
# Response: {"total": "$1,234.56"}For multi-field extraction, JSON schema enforcement via the provider’s structured output feature eliminates parse ambiguity entirely:
response = client.chat.completions.create(
model="gpt-4o-mini",
response_format={
"type": "json_schema",
"json_schema": {
"name": "invoice_extraction",
"schema": {
"type": "object",
"properties": {
"total": {"type": "string"},
"vendor": {"type": "string"},
"date": {"type": "string", "format": "date"}
},
"required": ["total", "vendor", "date"],
"additionalProperties": False
}
}
},
messages=[{"role": "user", "content": document_text}]
)The model produces exactly the specified schema or fails with a structured error. No downstream parse ambiguity. No remediation calls.
Lesson: the schema is the constraint that bounds output length and eliminates remediation calls.
Layer 5: Cache Misses on Repeated Semantics
Most production applications ask the same semantic questions repeatedly. Customer service agents answer the same categories of inquiry thousands of times per day. Code review agents check the same patterns across thousands of commits.
Exact-match caching handles only word-for-word identical prompts. It misses most of the real opportunity.
Semantic caching matches on meaning. “What is the capital of France?” and “France’s capital city?” and “Where is the French government based?” are the same question. A semantic cache returns the cached answer for all three without a model call.
import numpy as np
def semantic_cache_lookup(query: str, cache: list, threshold: float = 0.92) -> str | None:
query_embedding = embed(query)
for entry in cache:
similarity = cosine_similarity(query_embedding, entry["embedding"])
if similarity >= threshold:
return entry["response"]
return None
def semantic_cache_store(query: str, response: str, cache: list):
cache.append({
"query": query,
"embedding": embed(query),
"response": response,
"timestamp": now()
})The threshold matters. 0.92 is a reasonable starting point; calibrate against your domain. Cache hit rate depends on query distribution. For customer service workloads, hit rates of 40–70% are common after warmup. For creative or open-ended tasks, hit rates are near zero. Measure per operation type before building cache infrastructure.
The write cost is real: embedding every query before lookup adds latency. For low-latency paths where most queries are unique, the overhead outweighs the savings.
Lesson: semantic caching converts repeated questions into lookups. Effective on high-volume structured workloads. Profile per operation before deploying.
Layer 6: Undistilled History
Agent sessions accumulate history. A coding agent working on a refactor across 50 turns has 50 exchanges in context. By turn 50, the first 40 turns are mostly noise: debugging tangents, abandoned approaches, status updates. The last 10 turns contain the relevant decisions.
Sending all 50 turns on turn 50 costs roughly 5× what sending a distilled summary would cost. Long context windows also degrade model attention on early content. A model drowning in its own history often produces worse output than one given a clean summary.
History distillation replaces the verbose transcript with a decision-relevant summary as the session grows.
def should_distill(history: list, token_threshold: int = 8000) -> bool:
total_tokens = sum(count_tokens(msg["content"]) for msg in history)
return total_tokens > token_threshold
def distill_history(history: list, client) -> str:
return client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{
"role": "system",
"content": "Summarize the following conversation history into a compact decision log. Include: decisions made, code written, key findings, current state. Omit: debugging tangents, abandoned approaches, status updates."
},
{"role": "user", "content": json.dumps(history)}
]
).choices[0].message.contentUse a cheap model for distillation. This is a routine sub-task that doesn’t need frontier capability. The distilled summary costs a fraction of sending raw history to the primary model.
Lesson: history grows monotonically. Distillation cost is paid once. The alternative is paying for increasingly expensive context windows that degrade output quality as they grow.
Layer 7: Duplicate Prompts Across Sessions
Layer 6 is about redundancy within a session. Layer 7 is about redundancy across sessions.
When an agent starts a fresh session on a task it’s worked on before, it typically has no way to know. It rediscovers context from scratch. It re-reasons through decisions already made. Every session starts from zero.
For a team of ten engineers each starting five AI assistant sessions per day, with sessions that rediscover 2,000 tokens of context each time, that’s 100,000 tokens per day in pure rediscovery cost.
The structural fix is cross-session memory: write context at the end of sessions and read it at the start of new ones, replacing rediscovery work with recall. See the institutional knowledge tax for why re-explanation compounds and the five-layer agent memory stack for the OKF vault and recall patterns that make this concrete.
Without cross-session memory:
100 sessions × 2,000 tokens rediscovery = 200,000 input tokens/day
At $3/M: $0.60/day = $219/year
With cross-session memory:
100 sessions × 200 tokens recall = 20,000 input tokens/day
At $3/M: $0.06/day = $22/year
Storage + embedding: ~$5/year
Net saving: ~$192/year per 100 daily sessionsThe numbers look modest at 100 sessions per day. At 10,000 sessions per day across an organization, or when rediscovery context is 20,000 tokens per session, the math changes shape.
Lesson: sessions are the unit of billing. Memory is the mechanism that makes sessions shorter over time rather than longer.
Layer 8: Wrong Model for the Task
Most tasks don’t need frontier capability. Code completion for boilerplate, document classification, structured extraction, summarization, translation — these tasks are solved at Frugal-tier quality by models that cost 10–50× less than frontier.
Routing is often implemented as a binary choice at the request level. A better approach is task decomposition: break requests into sub-tasks, route each sub-task to the cheapest model that can handle it, compose the results.
User request: "Review this PR for security vulnerabilities and write a summary for the author."
Sub-task 1: Classify file types in the PR → Frugal (pattern matching)
Sub-task 2: Extract changed functions → Frugal (structured extraction)
Sub-task 3: Identify security anti-patterns → Standard (reasoning required)
Sub-task 4: Write author-facing summary → Standard (quality matters)
Sub-task 5: Format the final report → Frugal (template filling)
Without decomposition: 5 tasks × Frontier = 5× Frontier cost
With decomposition: 3 Frugal + 2 Standard ≈ 0.4× Frontier costTier-based escalation adds a different dimension: try Frugal first, escalate to Standard on quality-check failure, escalate to Frontier only as a last resort. Adds latency but reduces cost further on workloads where Frugal handles most requests. ClawQL’s name for this outcome-driven ladder is model escalation, inspired by Q00/ouroboros PAL. See the dedicated post.
def route_with_escalation(prompt: str, quality_checker) -> str:
for model in ["phi-4", "gpt-4o-mini", "gpt-4o"]:
response = call_model(model, prompt)
if quality_checker(response):
return response
# log escalation: tier, reason, cost delta
return responseLog every escalation with the failure reason and cost delta. Escalation patterns over time reveal which request types consistently need higher tiers — candidates for sub-task routing rather than escalation.
Lesson: routing is task classification. Decomposition is routing applied at sub-task granularity. Both are worth doing. Neither addresses Layers 1–7.
Layer 9: Unbudgeted Spend
LLM spend has no natural upper bound. A production system under unexpected load, a user who runs an unexpectedly large workload, an agent loop that runs longer than anticipated — any of these produces billing surprises that model optimization can’t prevent.
The fix is pre-inference budget enforcement: a hard limit at the gateway layer, before tokens are generated, that returns a structured error when the limit is reached.
class BudgetEnforcer:
def __init__(self, budget_store):
self.budget_store = budget_store
def check_and_reserve(self, team: str, estimated_tokens: int, model: str) -> bool:
estimated_cost = estimated_tokens * MODEL_RATES[model]
current_spend = self.budget_store.get_spend(team, period="month")
budget = self.budget_store.get_budget(team)
if current_spend + estimated_cost > budget:
raise BudgetExhausted(
team=team,
current_spend=current_spend,
budget=budget,
estimated_cost=estimated_cost
)
self.budget_store.reserve(team, estimated_cost)
return TrueA spend dashboard tells you money was spent after it was spent. A gateway-layer enforcer prevents spending from occurring. Per-team budgets with virtual keys are the pattern: each team gets a virtual key with a monthly USD cap. The gateway checks the budget before forwarding the request.
clawql inference keys create --team engineering --budget 500 --period monthly
clawql inference spend --group-by team --period monthLesson: dashboards are retrospective. Budgets are preventive.
Layer 10: Prompt Inefficiency
This layer is the most granular and usually the last place to optimize. Layers 1–9 each offer larger gains with less effort.
Prompt inefficiency is the gap between what your prompt says and what the model needs to do the task.
Padding and redundancy. “Please carefully analyze the following text and provide me with a detailed and thorough summary that captures the main points…” is roughly 25 tokens to say “Summarize:”. At 10,000 calls per day, that’s 250,000 wasted input tokens daily.
Over-specified constraints. Constraints the model will satisfy anyway (“Be factual.” “Don’t make things up.”) add tokens without changing behavior on well-aligned models.
Unnecessary examples. Few-shot examples are powerful when format or style is hard to specify directly. They’re expensive overhead when a clear instruction works just as well.
Verbose tool definitions. Tool schemas with lengthy descriptions for self-explanatory parameters consume tokens on every call where the tool is registered.
The prompt audit:
def audit_prompt_efficiency(prompt: str) -> dict:
tokens = count_tokens(prompt)
padding_patterns = [
r"please carefully",
r"thorough and detailed",
r"as an AI language model",
r"I'd be happy to help",
r"certainly, here is",
]
found_padding = [p for p in padding_patterns if re.search(p, prompt, re.IGNORECASE)]
return {
"total_tokens": tokens,
"padding_patterns_found": found_padding,
"estimated_waste_tokens": len(found_padding) * 8
}Systematic audits across highest-volume call types tend to find 10–20% token reduction opportunities.
Lesson: prompt optimization is fine-tuning. Fix Layers 1–9 first.
Layer 11: Missing Prefill
Several providers support assistant-turn prefill: you provide the beginning of the model’s response, and the model continues from there rather than generating the opening from scratch.
For structured output patterns this matters more than it first appears:
# Without prefill: model generates opening tokens before reaching JSON
messages = [
{"role": "user", "content": "Extract the key facts from this document as JSON."}
]
# Model might output: "Here is the JSON extraction:\n```json\n{..."
# You have to parse past the preamble
# With prefill: model continues from the JSON opening brace
messages = [
{"role": "user", "content": "Extract the key facts from this document as JSON."},
{"role": "assistant", "content": "{"}
]
# Model continues: '"vendor": "Acme Corp", "total": 1234.56, ...'Beyond format steering, prefill eliminates the warming-up tokens that precede actual content on many response types. The saving per call is small, typically 20–50 tokens. On high-volume structured extraction pipelines with millions of calls per month, 50 tokens per call is meaningful.
Prefill support varies by provider. Anthropic supports it natively. OpenAI’s structured output feature serves a similar purpose without explicit prefill.
Lesson: prefill is a micro-optimization worth implementing on high-volume structured pipelines. It’s not where you start.
Layer 12: The Flywheel Nobody Built
Layers 1–11 reduce the cost of the models you’re using. Layer 12 changes which models you use.
The production traffic flowing through your inference gateway is training data. Every call with a verified positive outcome is an example of the task you want a cheaper model to learn. Most teams don’t capture this. The call happens, the response is consumed, the tokens are billed, and nothing is retained except the invoice.
# Export calls where outcome was verified as correct
clawql inference export \
--verdict passed \
--format openai-jsonl \
--min-date 2026-06-01 \
--output ./training/2026-06.jsonl
# Scrub PII before training data leaves the system
presidio-anonymizer \
--input ./training/2026-06.jsonl \
--output ./training/2026-06-clean.jsonl
# Submit fine-tuning job
clawql inference finetune \
--dataset ./training/2026-06-clean.jsonl \
--base-model gpt-4o-mini \
--provider openai
# Register the fine-tuned model as the new Frugal tier
# tier-map.json: "frugal": "ft:gpt-4o-mini:your-org:task-name:abc123"After two or three Flywheel cycles — typically four to eight weeks of production traffic each — the custom Frugal model handles the task type that needed Standard before. The model escalation tier distribution shifts: more calls resolve at Frugal, fewer escalate. The cost curve bends downward without changing the task or the quality threshold.
Layers 1–11 are efficiency improvements on your current models. The Flywheel converts production spend into a proprietary model asset that accumulates value indefinitely. The model trained on your production traffic is more accurate on your specific task types than the generic base model because it was trained on your data, with your quality labels, in your specific domain.
The switching cost that creates: the custom Frugal model’s training data is your production calls, WORM-logged with the provenance of what entered the dataset. Moving to a different inference gateway means abandoning that model and starting accumulation from scratch. That’s value naturally accruing to wherever your calls land — not designed lock-in.
See the API spend that never compounds for verdict-filtered export, Presidio scrubbing, tier registration via model escalation, and the pipeline worker that closes the loop automatically.
Lesson: Layers 1–11 are costs you reduce. Layer 12 is capital you accumulate.
What the Layers Look Like Together
A realistic before/after for a document processing workload at 10,000 calls per day:
| Layer | Before | After | Daily saving |
|---|---|---|---|
| 1 Context bloat | 8,000 input tokens/call | 800 tokens/call | 72M tokens → $216 |
| 2 Verbose output | 1,200 output tokens/call | 400 tokens/call | 8M tokens → $40 |
| 3 Redundant static | 2,000 tokens uncached | 200 tokens cached | 18M tokens → $54 |
| 4 Unstructured output | 2 calls/request | 1 call/request | 2,000 calls → $12 |
| 5 Semantic cache | 0% hit rate | 45% hit rate | 4,500 calls avoided → $27 |
| 6 Undistilled history | 12,000 tokens avg history | 3,000 tokens avg | 90M tokens → $270 |
| 7 Cross-session rediscovery | 3,000 tokens/session start | 300 tokens/session | 27M tokens → $81 |
| 8 Wrong model | 100% Standard ($0.15/1k) | 70% Frugal ($0.02/1k) | $4,200 → $588/day |
| 9 Unbudgeted | Surprise bills | Hard cap per team | Variable |
| 10 Prompt efficiency | 250 avg overhead tokens | 25 avg tokens | 2.25M tokens → $7 |
| 11 Missing prefill | 40 avg preamble tokens | 0 | 400k tokens → $1.20 |
| 1–11 total | ~$4,700/day | ~$508/day | ~$4,200/day |
| 12 Flywheel (cycle 3) | Standard as baseline | Custom Frugal as baseline | Further 60–80% reduction |
The numbers will differ for your workload. The structure won’t. Layers 1–7 address token count. Layer 8 addresses per-token price. Layers 9–11 are hygiene. Layer 12 changes the game.
Lesson: the bill is a composite. Treating it as a single variable is how teams spend twelve months on model selection while leaving most of the money on the table.
Adoption Path
Nothing here requires rebuilding your system. The layers are addressable incrementally, roughly in order of impact.
Day zero: instrument your call store. Log input tokens, output tokens, model, operation, and outcome per call. Start with a JSONL file if you have nothing else.
Week one: audit Layer 1. For your highest-volume operation type, log the raw input token count per call. If it’s more than 2× what the task semantically requires, fix context bloat first. This alone often halves the bill.
Week two: enable semantic caching (Layer 5) for your highest-volume structured query types. Measure hit rate after one week. Above 20% hit rate means you’ve found a real lever.
Week three: enable prompt caching (Layer 3) for your static system prompt and tool definitions. Usually one configuration change per provider.
Month two: implement routing or model escalation (Layer 8). Start with a simple binary — “this operation type is Frugal, everything else is Standard” — rather than per-request classification. Measure escalation rate and adjust.
Month three: start collecting the Flywheel dataset (Layer 12). You won’t fine-tune yet. Build the training corpus. Run clawql inference export --verdict passed weekly and store the cleaned output.
Month four: first fine-tuning cycle. Submit the accumulated dataset. Benchmark the result against the base Frugal model on your specific task types. Register the winner.
Ongoing: revisit Layers 2, 6, and 7 once the higher-impact layers are fixed. The absolute savings are smaller but the infrastructure is already in place.
The team that fixes all twelve over six months is operating at a qualitatively different cost structure than the team still evaluating models.
Reference implementation for the full twelve-layer stack: docs.clawql.com/architecture/token-efficiency. Source: ClawQL on GitHub. Why input-only compression is half the problem: Both Sides. Measured benchmark against Executor: Both Sides of Context Compression.