The API Spend That Never Compounds

Every month you run production inference, you generate training data. Most teams let it evaporate. A practical guide to building the flywheel that turns API spend into proprietary model capital.

This pairs with Layer 12 of the twelve layers of LLM cost, model escalation, and the audit trail you can’t reconstruct. The flywheel only compounds when verdicts, redaction, provenance, and tier registration are part of the inference path.

---

The Spend That Evaporates

In January 2026, two operations teams started the same structured extraction project. Both processed semi-structured business documents — onboarding forms, vendor packets, renewal notices, and PDF attachments that needed fields extracted into a clean schema. Both used a frontier model because the documents were messy and the cost of extraction errors was high. Both handled about 30,000 extraction calls per month.

Team A treated inference as a utility bill. The model returned JSON, the application consumed it, and the provider invoice arrived at the end of the month. When humans corrected bad extractions, those corrections stayed in the review UI. When downstream validation accepted good extractions, that signal stayed in the workflow database. The inference layer saw tokens and latency, not outcomes.

Team B treated inference as a data acquisition loop. Every extraction call wrote a structured record: prompt hash, response hash, schema, model, tier, cost, document type, validation result, human correction when present, and correlation_id. Passed calls became candidate training examples. Failed calls became evaluation cases. Corrected calls became high-value supervised examples after redaction.

By July 2026, the teams had different businesses.

Team A’s July bill looked like January with slightly more volume — still paying for frontier extraction on routine documents.

Team B had two flywheel cycles behind it. A custom Frugal model handled the high-volume document types. Frontier was reserved for ambiguous edge cases. Their July bill was lower than January despite higher traffic, and their model encoded patterns specific to their documents, schemas, and reviewer preferences.

The difference was that Team B converted production spend into proprietary training capital, then registered that capital back into production through model escalation.

---

What the Flywheel Requires

The loop itself is simple:

production calls
  → outcome-labeled examples
  → scrubbed training corpus
  → fine-tuned model
  → cheaper routine inference
  → more production calls
  → better examples

Most teams still don’t build it because three things are missing before the first training run.

You need outcome signals — some way to know whether the model output was correct. Token logs don’t tell you that. Provider invoices don’t tell you that. A prompt and response without a verdict is a transcript, not training data.

You need safe export. Production prompts and responses contain client names, account numbers, contract terms, health data, source code, and other sensitive material. Export has to scrub PII and preserve provenance before data leaves the operational store.

You need production registration. A fine-tuned model that lives in a notebook or provider dashboard isn’t compounding. It has to become the default Frugal model for the task type, with fallback through model escalation when confidence drops.

Team A lacked all three. Team B built them before the first training run.

---

The Flywheel Log

The core object is a production inference record with an outcome.

{
  "id": "call_2026_03_14_000912",
  "correlation_id": "corr-doc-2026-03-14-87921",
  "team": "operations",
  "operation": "structured_extraction",
  "document_type": "vendor_onboarding_packet",
  "schema": "vendor_onboarding_v3",
  "model": "anthropic/claude-sonnet-4",
  "tier": "frontier",
  "input_tokens": 18420,
  "output_tokens": 1120,
  "estimated_cost_usd": 0.84,
  "prompt_hash": "sha256:0b25...",
  "response_hash": "sha256:6a91...",
  "cache_hit": false,
  "verdict": "passed",
  "verdict_source": "downstream_validation",
  "validator": "schema_and_erp_match",
  "created_at": "2026-03-14T17:22:09Z"
}

The verdict field is the hinge. Without it, the record is spend telemetry. With it, the record can enter the training pipeline.

A failed call is useful too:

{
  "id": "call_2026_03_14_000944",
  "correlation_id": "corr-doc-2026-03-14-87953",
  "operation": "structured_extraction",
  "document_type": "renewal_notice",
  "verdict": "failed",
  "verdict_source": "human_review",
  "failure_reason": "missed_auto_renewal_clause",
  "corrected_output_hash": "sha256:bd77..."
}

Passed calls train the model on correct behavior. Failed calls become holdout cases or corrected examples once a reviewer supplies the target output. The flywheel starts with one extra field — verdict — and everything else depends on having it.

---

Outcome Signal Approaches

Outcome signals don’t have to be perfect on day one. They have to be explicit.

Team B started with downstream validation because the structured extraction output was already checked against schemas and ERP records. They later added human correction capture for fields that validators couldn’t judge, such as non-standard termination clauses.

The export command filtered by signal strength:

# Passed production calls become training candidates
clawql inference export \
  --operation structured_extraction \
  --verdict passed \
  --exclude-cache-hits \
  --min-date 2026-03-01 \
  --max-date 2026-03-31 \
  --format openai-jsonl \
  --output ./training/structured-extraction-2026-03.jsonl

# Failed and corrected calls become a supervised set
clawql inference export \
  --operation structured_extraction \
  --verdict failed \
  --require-correction \
  --format openai-jsonl \
  --output ./training/structured-extraction-corrections-2026-03.jsonl

Cache hits are excluded because they don’t represent fresh model behavior. They may be useful for replay tests, but they shouldn’t teach the next model to imitate stale cached outputs.

Verdict quality controls model quality. Start with the strongest signal available, label its source, and keep weaker signals out of promotion-critical datasets until they’re calibrated.

---

Presidio Export

Production examples are sensitive. Team B’s export path ran PII detection and structured redaction before writing training files.

clawql inference export \
  --operation structured_extraction \
  --verdict passed \
  --exclude-cache-hits \
  --scrub-pii presidio \
  --format openai-jsonl \
  --output ./training/structured-extraction-2026-03-scrubbed.jsonl

The pipeline:

call store
  → verdict filter
  → cache-hit exclusion
  → Presidio detection
  → structured redaction
  → JSONL formatter
  → WORM TrainingLineage manifest

A scrubbed line preserves structure without leaking values:

{
  "messages": [
    {
      "role": "system",
      "content": "Extract fields into vendor_onboarding_v3 JSON."
    },
    {
      "role": "user",
      "content": "Vendor: [REDACTED:ORGANIZATION]. Tax ID: [REDACTED:US_TIN]. Contact: [REDACTED:EMAIL]. Payment terms: Net 45..."
    },
    {
      "role": "assistant",
      "content": "{\"vendor_name\":\"[REDACTED:ORGANIZATION]\",\"tax_id\":\"[REDACTED:US_TIN]\",\"payment_terms\":\"Net 45\"}"
    }
  ],
  "metadata": {
    "correlation_id": "corr-doc-2026-03-14-87921",
    "document_type": "vendor_onboarding_packet",
    "schema": "vendor_onboarding_v3",
    "verdict": "passed"
  }
}

The exact redaction policy depends on the domain. The important property is that export can’t bypass it. Data that isn’t scrubbed and recorded in lineage isn’t eligible for training.

PII scrubbing is a gate in the export command, not a notebook preprocessing cell. Provenance is attached.

---

TrainingLineage WORM

Every dataset export needs a lineage record.

{
  "event_kind": "TRAINING_EXPORT",
  "export_id": "exp_structured_extraction_2026_03",
  "operation": "structured_extraction",
  "source_window": {
    "from": "2026-03-01T00:00:00Z",
    "to": "2026-03-31T23:59:59Z"
  },
  "filters": {
    "verdict": "passed",
    "exclude_cache_hits": true,
    "document_types": ["vendor_onboarding_packet", "renewal_notice"]
  },
  "source_records": 28734,
  "exported_records": 24108,
  "presidio_version": "2.2.354",
  "redaction_policy_hash": "sha256:8801...",
  "input_hash": "sha256:fa44...",
  "output_hash": "sha256:190d...",
  "sample_correlation_ids": ["corr-doc-2026-03-01-00102", "corr-doc-2026-03-14-87921"]
}

That record is written to WORM storage and chained with the same audit mechanism used for runtime decisions. When someone asks “what data trained structured-extraction-v2?” the answer should be a lineage chain:

TRAINING_EXPORT exp_structured_extraction_2026_03
  → PRESIDIO_SCRUB redaction_policy sha256:8801...
  → FINE_TUNE_JOB ft_structured_extraction_v1
  → BENCHMARK_RESULT bench_structured_extraction_v1
  → TIER_REGISTRATION frugal structured_extraction

No lineage, no production registration. The proprietary model asset is only defensible if you can prove what went into it, how it was scrubbed, and why it was promoted.

---

Fine-Tuning Economics

Team A and Team B spent similarly in January and February because both were collecting production volume on frontier models.

The first cycle didn’t eliminate frontier calls. It moved routine structured extraction to a custom Frugal model and escalated uncertain cases.

After two cycles, the custom Frugal model knew the team’s schema conventions: which fields were mandatory, how reviewers represented missing values, how renewal clauses appeared in their documents, which vendor-packet sections were irrelevant. That knowledge came from production traffic with verdicts, not from a synthetic benchmark. Fine-tuning paid because it changed the tier distribution — the win is moving routine volume to a custom Frugal baseline, not making any individual call cheaper in isolation.

---

Tier Registration via Model Escalation

The trained model has to enter the gateway’s tier map. Team B registered the second-cycle model as the Frugal default for structured extraction.

{
  "policyVersion": "2026.07.01",
  "modelEscalation": {
    "enabled": true,
    "tiers": {
      "frugal": {
        "default": "local/structured-extraction-v2",
        "maxCostPer1kTokens": 0.0004
      },
      "standard": {
        "default": "groq/llama-3.3-70b",
        "maxCostPer1kTokens": 0.002
      },
      "frontier": {
        "default": "anthropic/claude-sonnet-4",
        "maxCostPer1kTokens": 0.015
      }
    },
    "domainRoutes": {
      "structured_extraction": {
        "defaultTier": "frugal",
        "frugalModel": "local/structured-extraction-v2",
        "minConfidence": 0.86,
        "escalateOn": ["schema_validation_failed", "low_confidence", "policy_conflict"]
      }
    }
  }
}

clawql inference policy apply ./tier-map.json

clawql inference policy route \
  --operation structured_extraction \
  --document-type vendor_onboarding_packet

Model escalation runs the loop: start on the custom Frugal model, validate output against schema and downstream systems, escalate to Standard when confidence or validation fails, escalate to Frontier for the hardest cases, WORM-log the routing decision and verdict, export passed Frugal calls for the next cycle.

The “start cheap, escalate on failure” idea comes from Q00/ouroboros PAL work. ClawQL’s model escalation applies that principle inside a gateway with WORM-logged transitions, verdict-aware export, and policy-manifest tier registration.

A fine-tuned model compounds only after it becomes the production Frugal route for its task. Model escalation keeps quality from falling while the distribution shifts.

---

The Switching Cost

The custom Frugal model is trained on the organization’s document types, schemas, corrections, validation rules, redaction policy, and WORM lineage. Moving inference traffic to another gateway that doesn’t preserve verdicts, exports, manifests, and tier registration stops the flywheel. The model artifact can move if licensing permits. The accumulated lineage and routing history can’t be recreated unless the new system carries the same records.

That switching cost is capital formation, not a dark pattern. Value naturally accrues where production calls, verdicts, and training lineage accumulate together.

This is why Layer 12 is different from caching or prompt trimming. Layers 1–11 reduce current cost. The flywheel builds an asset that changes what future costs look like.

The defensible asset is the chain of verified production examples, redaction manifests, benchmark results, and tier registrations — not merely the fine-tuned weights.

---

Observability for the Flywheel

You should be able to see whether the flywheel is spinning.

# Is Frugal handling more structured extraction over time?
clawql inference spend \
  --operation structured_extraction \
  --group-by tier \
  --period month

# Are verdict rates stable after registration?
clawql inference export \
  --operation structured_extraction \
  --verdict passed \
  --count-only \
  --period week

# What training lineage produced the current Frugal model?
clawql inference lineage show local/structured-extraction-v2

If Frugal share rises but verdict pass rate falls, the model was promoted too aggressively. If verdict rate is stable but Frontier share doesn’t fall, escalation thresholds may be too conservative. If corpus size grows but benchmark quality stalls, the dataset may be too noisy or too narrow. These three states require different responses — promotion rollback, threshold tuning, and data quality review respectively.

---

Honest Failure Modes

Cold start. A new task type with no verified examples can’t produce a useful custom model. Plan for one or two months of collection before the first serious training run.

Noisy verdicts. Treating “workflow completed” as equivalent to “model was correct” lets the dataset accumulate silent errors. Separate strong signals from weak signals and benchmark before registration.

PII leakage. Exporting production data without structural redaction creates compliance risk. The export command should fail closed when redaction policy is missing.

Domain drift. A model trained on January vendor packets may degrade when July packets use a new template. Model escalation catches some of this by escalating failures, but monitoring has to catch the drift.

Out-of-domain routing. A structured-extraction model performs badly on legal synthesis or support chat. Domain routes need task classification, and misrouted calls should not enter the training corpus.

Training data poisoning. If upstream document ingest admits adversarial or poisoned content, the flywheel amplifies it. Ingest scanning and audit provenance are prerequisites, not optional extras.

The flywheel compounds whatever your production process emits. Clean, verified examples produce value. Noise produces risk.

---

Implementation Path

Start by enabling the inference call store. Capture operation, document type, schema, model, tier, cost, prompt hash, response hash, cache status, and correlation_id. In week one, add verdicts — start with downstream validation for structured extraction: schema checks, ERP match, required-field checks, and human approval when available.

In week two, run a dry export. Filter to verdict=passed, exclude cache hits, scrub with Presidio, and inspect the JSONL manually. Count examples by document type.

By month two, write TrainingLineage WORM records for every export — source window, filters, redaction policy hash, input hash, output hash, and sample correlation IDs.

In month three, fine-tune the highest-volume task type. Benchmark against the current Frugal and Standard models on held-out examples. Only register the model if it improves cost without unacceptable quality loss.

In month four, register the model in tier-map.json as the Frugal default for that operation. Enable model escalation to Standard and Frontier on low confidence or validation failure.

From month five, repeat the export with production calls handled by the custom Frugal model. Train the second cycle. Compare tier distribution before and after registration. Start new flywheels per task type rather than globally — they compound separately and the lineage stays clean.

At every monthly review: corpus growth, pass rate, escalation rate, Frontier share, cost per document, lineage verification. The architecture decides whether production calls become disposable expenses or candidate training examples. Once the loop is closed, each month of inference makes the next month more efficient.

---

Reference implementation: docs.clawql.com/architecture/token-efficiency (Layer 12). Source: ClawQL on GitHub. Related: twelve layers of LLM cost, Both Sides, model escalation, audit trail reconstruction.