Replacing LiteLLM After the March 2026 Supply Chain Compromise

A direct comparison of clawql-inference vs LiteLLM: architecture, trust model, routing philosophy, fine-tuning flywheel, payment rails, and a concrete migration path.

This pairs with Mini Shai-Hulud and supply-chain controls, immutable releases, and the twelve layers of LLM cost (model escalation and the fine-tuning flywheel). Product reference: docs.clawql.com/inference/clawql-inference.

What Happened in March 2026

In March 2026, LiteLLM had a supply chain compromise. A malicious package in LiteLLM’s Python dependency tree was used as an infection vector, exposing teams running LiteLLM in production CI/CD pipelines and inference infrastructure to credential harvesting.

This isn’t a knock on the LiteLLM team. Supply chain attacks against Python packages are a documented, growing, and now open-sourced threat class — the Mini Shai-Hulud campaign that followed in May 2026 proved that the tooling for this attack is freely available and actively being copied. The LiteLLM incident is a consequence of the ecosystem’s current security posture, not a unique failure of LiteLLM specifically.

But it does crystallize a question that many teams are now asking: if you’re running an AI inference gateway in production, what does it mean to actually trust it?

This post answers that question with a direct comparison. Not a marketing comparison — an engineering one. Where LiteLLM is genuinely strong, that’s stated. Where clawql-inference is ahead, that’s demonstrated with specifics. Where clawql-inference has gaps, those are named.

What LiteLLM Actually Is (And Why It Got Popular)

LiteLLM is a Python-based proxy that provides a unified OpenAI-compatible API surface across 100+ LLM providers. The value proposition is simple and real: you point your existing OpenAI SDK at LiteLLM, configure your providers, and your code works with Anthropic, Groq, Together, Bedrock, Vertex, and everything else without changing a line of application code.

It has ~40,000 GitHub stars, is YC-backed, and is used in production at companies including Netflix. The 100+ provider coverage is genuinely impressive and represents years of maintenance work across dozens of provider API formats.

That’s the honest assessment of what LiteLLM does well. Keep it in mind while reading the comparison below — the question isn’t whether LiteLLM is a bad product, it’s whether it’s the right product for teams that need more than routing.

The Architecture Difference

This is the most important difference and it’s structural.

LiteLLM is a proxy. It sits between your application and LLM providers, normalizes the API surface, and routes calls. It has no knowledge of what your agent is trying to accomplish, whether it succeeded, what it cost relative to your plan limits, or what the output looked like. The proxy’s job ends when the provider responds.

clawql-inference is a gateway that closes the production loop. Every call goes through a decorator stack that observes, evaluates, caches, routes adaptively, enforces entitlements, and records to an auditable call store. The output of one call can inform the routing of the next. Production traffic accumulates into training data. Training data produces a fine-tuned model. The model registers back into the routing tier. The loop closes.

The decorator stack, from outermost to innermost:

Traced → Observed → Entitlement → TokenEfficiency → Cache → Fallback → Configured → provider

Each layer is independently toggleable. Every call passes through all active layers. The call store is populated by ObservedInferenceGateway on every successful completion — not as a side effect you might forget to wire up, but as a structural part of the gateway.

LiteLLM logs to Langfuse or your configured callback. That’s a side effect. If the callback fails, the log is lost. If the callback is misconfigured, nothing is recorded. clawql-inference’s call store is the primary record, not a secondary logging destination.

Lesson: a proxy ends at the provider response. A production gateway treats observation, routing feedback, and the call store as part of the happy path — not as optional callbacks.

The Trust Model Difference

This is the March 2026 incident in concrete terms.

LiteLLM:

• Python codebase with a dependency tree of hundreds of packages
• Any package in that tree is a potential supply chain attack vector
• The March 2026 compromise came through a dependency, not through LiteLLM’s own code
• No content-addressed artifact pinning — packages install from upstream PyPI
• No SBOM per release
• No startup integrity verification

clawql-inference:

• TypeScript-native, same monorepo as the full ClawQL platform
• Every release generates a CycloneDX SBOM recording every dependency and its hash
• Container images Cosign-signed with SBOM attached
• Layer 0 release manifest records SHA-256 of every artifact, anchored to Arweave (permanent, content-addressed storage)
• clawql doctor --smoke verifies the installed binary against the manifest hash on startup — if the binary has been tampered with, startup fails
• Integration catalog mirrors all provider adapters to ClawQL-controlled Harbor/R2 before distribution — upstream updates cannot silently reach users

The startup verification is the key property. A compromised LiteLLM dependency that modifies the installed package after the fact continues running silently. A modified clawql-inference binary is detected at next startup before it handles any requests.

Same posture as the Layer 0 / Arweave release story: process reputation is not enough; content hashes and startup verify are.

Lesson: “we install from the official package” is not a trust model after March 2026. Mirror, sign, pin, verify on boot — or accept silent post-install mutation as in scope.

Routing: Static vs Outcome-Driven

LiteLLM routing is load balancing and cost rules. You configure which models to use, optional weights or RPM limits, and LiteLLM distributes calls accordingly. It doesn’t know whether a call succeeded in a meaningful sense — it knows whether the provider returned HTTP 200.

Model escalation in clawql-inference is outcome-driven. The router knows whether the agent’s task actually succeeded, whether confidence was high, whether drift has been detected, and which tier is appropriate given the current task context. (This is Layer 8 in the twelve-layer cost stack — not to be confused with agent coordination, which fans work across multiple agents.)

The three tiers:

The escalation rules:

• Decomposed child tasks (sub-agent work) start at Frugal
• Top-level tasks start at Standard
• Failure at the current tier escalates one notch — never skips tiers
• combined_drift > 0.3 triggers agent coordination (fan-out via Hermes when configured)
• Every escalation decision is WORM-logged with the specific failure signal that triggered it

# Enable routing
export CLAWQL_INFERENCE_ROUTING_ENABLED=1
export CLAWQL_INFERENCE_MODEL_FRUGAL=ollama/phi4
export CLAWQL_INFERENCE_MODEL_STANDARD=groq/llama-3.3-70b
export CLAWQL_INFERENCE_MODEL_FRONTIER=anthropic/claude-sonnet-4

# Pin to a single model (bypasses the escalation ladder)
export CLAWQL_INFERENCE_MODEL_PIN=anthropic/claude-sonnet-4

LiteLLM has no concept of “this task failed, try a more capable model.” It has retry logic for provider errors, but that’s different — retrying the same model on a transient error vs. escalating to a more capable model because the task requires more capability.

The Semantic Cache Difference

LiteLLM caching uses Redis for exact-match or near-match caching. It’s effectively string matching — the same prompt returns the cached response. Useful for repeated identical calls, limited utility for semantically similar but textually different requests.

clawql-inference semantic cache embeds every request using an OpenAI-compatible embedding model and compares against cached embeddings using cosine similarity. Requests that are semantically equivalent — “summarize this document” vs “give me a summary of this doc” — hit the cache even though the text is different.

export CLAWQL_INFERENCE_SEMANTIC_CACHE=1
export CLAWQL_INFERENCE_CACHE_THRESHOLD=0.92    # cosine similarity floor
export CLAWQL_INFERENCE_CACHE_TTL=24h
export CLAWQL_EMBEDDING_MODEL=text-embedding-3-small

The cache is backed by pgvector when a Postgres inference database is configured, falling back to in-memory for single-instance deployments.

Critical design decisions:

Embedding failures fail open. If the embedding service is unavailable and the cache can’t be checked, the request proceeds to live inference. The cache is an optimization — it never blocks inference. This is explicitly handled in the Effect service layer rather than being a try-catch assumption.

Write operations are never cached. Only read-safe operations are eligible for cache hits. This is enforced in the gateway stack, not left to caller discipline.

Cache hits are flagged in the call store with cache_hit: true on the InferenceRecord. The --exclude-cache-hits flag on export removes them from fine-tuning datasets — you don’t want to train on cached responses, only on real model outputs.

The Fine-Tuning Flywheel: The Gap LiteLLM Doesn’t Close

This is the most significant architectural difference and the one that produces compounding advantages over time.

LiteLLM routes inference. It does not close the production loop. The output of your agents today has no connection to the cost or quality of your agents tomorrow. Every inference call is independent.

clawql-inference closes the loop:

Production traffic
  → WORM-logged call store (prompt, response, tier, latency, verdict)
  → Evaluator verdicts filter to high-quality examples
  → PII-scrubbed JSONL export with Merkle-anchored dataset manifest
  → Fine-tuning job (OpenAI or Anthropic fine-tuning API)
  → Custom model registered in tier-map.json as new Frugal tier
  → Model escalation uses custom model for matching task types
  → Better cheap-tier results → better verdicts → better training data
  → Repeat

Each piece of this is a specific CLI command, not aspirational architecture:

# Export verdict-filtered training data
clawql inference export \
  --verdict passed \
  --tier frugal \
  --format openai-jsonl \
  --min-score 0.8 \
  --output ./training-data/$(date +%Y-%m).jsonl

# Submit fine-tuning job
clawql inference finetune \
  --dataset ./training-data/2026-07.jsonl \
  --base-model gpt-4o-mini \
  --provider openai

# Check status
clawql inference finetune status --job-id ftjob_abc123

# Register the resulting model as your new Frugal tier
clawql inference finetune register \
  --job-id ftjob_abc123 \
  --tier frugal \
  --alias ollama/phi4-production-v3

After registration, model escalation automatically uses the custom model for matching task types at Frugal tier. The cost drops. The quality on your specific workload is higher than the generic model it replaced because it was trained on your actual production traces.

The flywheel accelerates: better outputs → more passed verdicts → larger training corpus → better fine-tuned model → lower cost per task. LiteLLM has no equivalent. There is no path from “this call succeeded” to “the model handling similar calls next month is better and cheaper.”

The dataset manifest is what makes this auditable. Every export writes a WORM manifest recording the sample hashes, filter criteria (verdict threshold, tier, date range), Presidio scrub version, and Merkle root. If you’re processing sensitive documents — contracts, medical records, financial data — you can prove exactly what entered your training data and that PII was removed before export. This matters for compliance and it’s not something you can add as an afterthought.

The Payment Rails Difference

LiteLLM has API key management and basic spend tracking. It does not have payment rails.

clawql-inference ships four independent billing modes via clawql-payments:

Plan entitlements — pre-call monthly caps with structured 402 responses when exceeded:

export CLAWQL_PAYMENTS_ENFORCE_INFERENCE=1
# Returns HTTP 402 with structured error when monthly cap hit:
# { "error": { "type": "insufficient_quota", "reset_at": "...", "current": 10000, "limit": 10000 } }

Stripe metered billing — post-call usage reporting to Stripe Billing Meters:

export CLAWQL_PAYMENTS_REPORT_STRIPE_METER=1
export STRIPE_METER_EVENT_NAME=clawql_inference_calls
# Reports each inference call to Stripe for metered billing

x402 per-request USDC micropayments — HTTP-native stablecoin payment gating:

export CLAWQL_X402_ENFORCE=1
# Returns 402 Payment Required with x402 challenge
# Agent pays in USDC, presents payment proof, proceeds
# No account required, no signup, sub-second settlement

MPP session-based payments — pre-authorized spending sessions for high-frequency agents:

export CLAWQL_MPP_ENABLED=1
# Agent pre-authorizes a spending limit once
# Micropayments stream within the session
# No per-call blockchain transaction

These aren’t just for operators billing their own customers — ClawQL’s own managed hosted tiers run on the same package. When you use clawql-inference on the hosted plan, your plan limits are enforced by the same EntitlementEnforcedGateway that your own deployment uses. Dogfooding is structural, not claimed.

The x402 + MPP combination is the agentic payment story that doesn’t exist in LiteLLM or any other inference gateway: agents can discover, pay for, and consume inference endpoints autonomously without human involvement, with full WORM-audited payment history linked back to the specific inference calls that triggered the payments via correlation_id.

Three Usage Systems (Don’t Conflate Them)

One operational complexity worth naming explicitly: clawql-inference has three distinct usage tracking systems that serve different purposes.

Plan entitlements ($CLAWQL_HOME/Payments/usage.json) — monthly inference call counts checked before each completion. This is the gate that enforces plan limits. Resets monthly.

Inference call store (calls.jsonl or clawql_inference_calls Postgres table) — full InferenceRecord per completion including tokens, latency, model, tier, verdict, correlation_id. This is what clawql inference spend queries. This is the source for fine-tuning export. This is the audit record.

Virtual key USD budgets ($CLAWQL_HOME/Inference/virtual-keys.json) — per-team spending caps in USD based on estimated token costs. This is the per-team limit, separate from the plan limit.

All three can be active simultaneously and they track different things. Don’t assume that clawql inference spend output reflects the same number as your plan usage counter — they’re measuring different things (token costs vs. call counts) over different windows (rolling vs. calendar month).

The Observability Difference

LiteLLM observability is callback-based. You configure a Langfuse, Helicone, or custom callback, and LiteLLM calls it after each completion. If the callback fails, the event is lost. If it’s misconfigured, nothing is recorded. Observability is a side effect.

clawql-inference observability is structural. The ObservedInferenceGateway layer writes to the call store as part of the completion pipeline. The call store write happens before the response is returned to the caller. It’s not optional and it’s not a callback.

On top of the call store:

• OTLP spans to Tempo or any collector (CLAWQL_ENABLE_OTEL_TRACING=1)
• Langfuse work-trace emission via OTLP (CLAWQL_ENABLE_LANGFUSE=1, opt-out per ADR 0005)
• clawql inference logs — tail recent completions with model/tier/time filters
• clawql inference trace --correlation-id <id> — full lifecycle of a single request
• clawql inference spend --group-by model|tier|team|provider — cost breakdown across any dimension

Every escalation decision, cache hit, fallback attempt, entitlement check, and payment event carries the same correlation_id. A compliance query for “show me everything that happened during this agent session” is a single index lookup across the WORM trail, not a join across five different log sources.

The Policy Manifest

LiteLLM configuration is environment variables and a YAML config file. These work, but there’s no versioning, no Merkle anchoring, and no way to prove what configuration was active at a specific point in time.

clawql-inference supports a policy manifest YAML at $CLAWQL_HOME/Inference/policy.yaml that merges with environment variables at runtime (env wins on conflicts). The manifest is versioned with a policyVersion field:

policyVersion: '2026.07.01'
inference:
  escalation:
    enabled: true
    tierMap:
      frugal: ollama/phi4
      standard: groq/llama-3.3-70b
      frontier: anthropic/claude-sonnet-4
  cache:
    enabled: true
    threshold: 0.92
    ttl: 24h
  fallback:
    enabled: true
    frugal: [ollama/phi4-backup, openai/gpt-4o-mini]
  pipelineWorker:
    enabled: true
    schedule: '0 2 * * 0'
    minSamples: 500
  observability:
    profile: external

The manifest governs the live gateway stack — not just policy show display but actual routing behavior, cache thresholds, fallback chains, and pipeline worker configuration. resolveInferenceEffectiveEnv() is the same merge function used by clawql inference policy show and clawql inference serve, so what you see in policy show is what the gateway is actually using.

Multi-Instance Deduplication

LiteLLM supports horizontal scaling with standard load balancer patterns. Fine-tuning pipeline scheduling in a multi-instance deployment requires external coordination to avoid duplicate cron jobs.

clawql-inference handles this with Postgres advisory locks when CLAWQL_INFERENCE_DATABASE_URL is configured:

// pg_try_advisory_lock keyed by pipeline schedule + UTC minute
// Only one replica runs each cron tick
// Others skip the tick without blocking

If you’re running multiple inference serve replicas behind a load balancer, only one will run the fine-tuning pipeline worker per scheduled interval. No external coordination required. The lock releases automatically when the tick completes.

Migration Path

The migration is designed to be zero-code. Any client that respects OPENAI_BASE_URL works:

# Install
curl -fsSL https://clawql.com/install | bash
# or: npm install -g clawql-mcp

# Start the gateway
clawql inference serve --port 8080

# Point your existing client at it
export OPENAI_BASE_URL=http://127.0.0.1:8080/v1
export OPENAI_API_KEY=your-existing-key  # passed through to provider

# Your existing code runs unchanged

Provider configuration:

# OpenAI (passthrough)
export OPENAI_API_KEY=sk-...

# Anthropic
export ANTHROPIC_API_KEY=sk-ant-...

# Ollama (local)
export OLLAMA_BASE_URL=http://127.0.0.1:11434

# Groq
export GROQ_API_KEY=gsk_...

Model IDs: use provider/model format (anthropic/claude-sonnet-4, ollama/phi4, groq/llama-3.3-70b) or bare public IDs (gpt-4o, claude-sonnet-4) when a matching provider is configured.

Start with observation only:

# Run the full gateway with just call store recording
# No routing changes, no cache, no entitlements
# Just: every call is observed and recorded
clawql inference serve --port 8080

After a week of production traffic you’ll have a call store to query, a spend breakdown by model and tier, and enough data to know whether semantic caching and model escalation are worth enabling for your workload.

Enable features incrementally:

# Week 1: observation only (default)
clawql inference serve

# Week 2: add semantic cache
export CLAWQL_INFERENCE_SEMANTIC_CACHE=1
clawql inference cache  # verify config before serve

# Week 3: add model escalation
export CLAWQL_INFERENCE_ROUTING_ENABLED=1
clawql inference escalation show  # verify tier map

# Week 4: add fallback chains
export CLAWQL_INFERENCE_FALLBACK_ENABLED=1
export CLAWQL_INFERENCE_FALLBACK_STANDARD=groq/llama-3.3-70b,anthropic/claude-haiku-4

# Month 2: first fine-tuning export when call store has enough data
clawql inference export --verdict passed --format openai-jsonl --output ./export.jsonl

The Honest Gaps

Being direct about where LiteLLM leads:

Provider breadth. LiteLLM supports 100+ providers representing years of maintenance work. clawql-inference ships built-in support for OpenAI, Anthropic, and Ollama with a plugin API for additional providers. If you need Azure OpenAI, AWS Bedrock, Google Vertex, Hugging Face, Together, Fireworks, Replicate, and a dozen others out of the box, LiteLLM’s coverage is broader today. The plugin API closes this over time but the ecosystem is not yet equivalent.

Python ecosystem integration. LiteLLM is Python. If your stack is Python-native — LangChain, LlamaIndex, DSPy, most ML tooling — LiteLLM integrates without friction. clawql-inference is TypeScript. The OpenAI-compatible REST surface means any HTTP client works, but native Python SDK integration is simpler with LiteLLM.

Hosted cloud option. LiteLLM Enterprise offers a managed hosted version. clawql-inference is self-hosted as a standalone package or included in ClawQL’s managed hosted tiers. If you want a managed inference gateway with zero operational overhead, LiteLLM’s hosted offering is an option; ClawQL’s hosted tiers include clawql-inference but are a broader platform commitment.

Community and documentation maturity. LiteLLM has a larger community, more Stack Overflow answers, more blog posts, and more example integrations. clawql-inference documentation is thorough but the community is earlier stage.

Summary

LiteLLM solved the “I need a unified API for 100 providers” problem and solved it well. If that’s the full scope of your requirement, it’s a reasonable choice.

If you need the production loop closed — routing that responds to outcomes, a fine-tuning flywheel that makes your cheapest tier better over time, a supply chain posture that survived March 2026, and payment rails for the agentic economy — clawql-inference is the answer.

---

clawql-inference is available as a standalone npm package (npx clawql-inference) and as part of the ClawQL platform. Full reference: docs.clawql.com/inference/clawql-inference. Migration guide and provider plugin docs: docs.clawql.com.