Secret-as-a-Service: Credential Rotation for Local and Edge Agents

Long-lived ENV secrets turn a host compromise into a platform breach. Bootstrap once, exchange for short-lived Vault or edge credentials, and never let the agent hold a permanent secret.

Part 1 closed anonymous write access to telemetry. This post closes the next hole: how a high-privilege agent obtains and renews the credentials it needs to act — without parking long-lived secrets in .env, Docker Compose, or Kubernetes Secret objects that are one cat away from exfiltration.

---

The Laptop That Held the Kingdom

Picture a ClawQL edge agent on a developer laptop or remote clawql-agent --mode=edge node. To “just get it working,” the .env looks familiar:

OPENAI_API_KEY=sk-live-...
GITHUB_TOKEN=ghp_...
DATABASE_URL=postgres://agent:SuperSecret@db/prod
NATS_CREDS=/Users/dev/.config/clawql/user.creds
VAULT_TOKEN=hvs.CAES...   # "temporary," three months old

Two weeks later someone clones a malicious skill, a prompt injection steers a tool that shells out, or malware dumps process memory and the home directory. The attacker doesn’t need to defeat Vault, mTLS, or Panguard — they own every long-lived credential the agent was given for convenience.

That’s the Nutrient lesson applied to secrets. A secret written to a config file, environment variable, or Kubernetes Secret is data waiting to be discovered. Agentic platforms make the problem worse because the same pod or process reuses that credential across dozens of sessions and hundreds of tool calls.

Static secrets create three systemic failures. They’re readable by breadth: Kubernetes Secrets are base64, and anyone with misconfigured RBAC or etcd access can read them. They accumulate holders: every developer laptop, CI runner, former contractor, and forgotten compose stack that ever received the value remains a holder until rotation finally happens. And their breach windows are measured in months rather than the minutes that dynamic credentials achieve.

If compromise of the agent host implies compromise of every downstream system the agent could reach, you have credential caching, not secrets management.

---

What the Agent Actually Needs Secrets For

A ClawQL deployment is not a single microservice with one database password. Typical secret surfaces:

Local and edge agents amplify this because the trust boundary is a developer machine or remote node, not a locked-down cluster. Even a perfect network policy fails if the secret is already on disk in plaintext.

---

The Architecture Pattern: Secret-as-a-Service

The agent holds a bootstrap ticket or session identity, not permanent third-party secrets. At need, a trusted exchange point trades that identity for a short-lived lease. The lease TTL matches the work — often minutes — then dies or is revoked on anomaly. Every mint, renewal, and revocation is written to a tamper-evident audit trail using accessors, never the secret values themselves.

Agent                Gateway                 Vault              External API / DB
  |                     |                      |                      |
  |-- Session JWT ----->|                      |                      |
  |   (+ device/workload proof)                |                      |
  |                     |-- validate session,  |                      |
  |                     |   ATR, nonce ------->|                      |
  |                     |-- exchange / mint -->|                      |
  |                     |<-- short-lived lease-|                      |
  |                     |----------------------|-- leased call ------>|
  |                     |<---------------------|-- result ------------|
  |<-- tool result -----|                      |                      |
  |   (secret never returned to agent)         |                      |
  |                     |                      |-- lease expires ---->|

The agent requests capability. It never possesses durable authority.

---

Why Kubernetes Secrets Alone Fall Short

Teams often graduate from .env to cluster Secrets and stop. Kubernetes Secrets are base64 encoding plus RBAC, not encryption at rest by default. Useful as a transport for bootstrap material in tightly locked namespaces — not as the system of record for production credentials an agent holds for weeks.

The right layering:

---

HashiCorp Vault as the Secret Factory

Design principles:

Dynamic secrets only — unique credential per request with short TTL. No static credential in config, ENV, agent memory, or git. HA Vault with Raft integrated storage; three or more replicas in production. Agents don’t hold long-lived Vault tokens — the gateway handles exchange.

HSM-backed unseal:

Auto Unseal via AWS KMS, GCP Cloud KMS, or Azure Key Vault. Master key material stays in tamper-resistant hardware, not on disk next to the Vault binary. Shamir shares only for true break-glass requiring multiple human key holders simultaneously. Availability must not depend on a human unlocking Vault after every pod restart, but recovery of the kingdom must never depend on a single USB stick without controls.

Dynamic engines for agents:

Revocation is lease-shaped: when the session ends or Panguard/Falco signals compromise, revoke the lease and the credential dies everywhere it was issued.

KV for a JWT signing key + one-time bootstrap:

vault secrets enable -path=observability kv-v2

vault kv put observability/worker/jwt-signing-key \
  key="$(openssl rand -base64 32)"

# Bootstrap with single-use token — not a root token in 1Password forever
vault token create \
  -policy=observability-worker-read \
  -use-limit=1 \
  -ttl=5m

Policy shape that stays tenant-scoped:

# Per-tenant, per-agent paths — never broad wildcards
path "secret/data/tenants/{{identity.entity.aliases.kubernetes.path}}/agents/{{identity.entity.name}}/*" {
  capabilities = ["read"]
}

path "database/creds/agent-readonly" {
  capabilities = ["read"]
}

# Explicit denies win
path "secret/data/tenants/+/admin/*" {
  capabilities = ["deny"]
}

Policy changes require the same multi-party approval used for ATR rule changes.

Gateway-as-exchange-point (non-negotiable):

The agent presents a Session JWT to the gateway. The gateway exchanges it for a short-lived Vault token scoped to that agent’s policy. The token TTL matches the session or a single tool call. The token never gets logged, never returned to the agent for storage, and ideally never leaves the gateway process beyond the call.

If the agent process can echo $VAULT_TOKEN and call Vault directly with broad rights, you’ve rebuilt static secrets with extra steps. Vault without the exchange pattern is still a secret server. The exchange pattern is what makes it a secret service relative to the agent.

---

Cloudflare Secrets for Small-Footprint Edge

Not every deployment runs HA Vault on day one. The Part 1 Worker can store minting keys via:

echo "your-signing-key" | wrangler secret put JWT_SIGNING_KEY

Appropriate for encrypted-at-rest edge secrets without cross-service dynamic DB roles yet. Not a substitute for automated rotation with audit, per-session database users, or instant fleet-wide revocation tied to agent identity.

A reasonable progression: Workers Secrets / cloud secret manager for edge minting keys → Vault or cloud IAM roles + STS for tool credentials → agents fully on gateway exchange so local ENV contains only bootstrap material.

The right control is the one that matches your blast radius today. That shouldn’t stay static for a year while OPENAI_API_KEY spreads across every laptop.

---

Session JWTs vs Tool-Scoped Tokens

A session can last hours and fire hundreds of tool calls per minute. A single long-lived session bearer that authorizes every tool call is a static credential with better branding.

At session start, mint a Session JWT containing agent ID, tenant ID, session ID, and ATR claims (coarse, session-level). Don’t send that JWT to tool handlers as the sole authorization for every call.

Exchange for a tool-scoped token per invocation:

Session JWT
  → Gateway validates
  → Issues tool-scoped token (only claims for this tool)
  → TTL ≤ 5 minutes, non-renewable, single-purpose
  → Audit: tool name, claims, timestamps, token accessor

Stealing a tool-scoped token should be a short, narrow incident — not a week-long admin session.

External APIs — OAuth/OIDC, not pasted keys:

Client credentials or auth-code flows for service access. Fetch on demand, use once, discard. Minimum scopes (repo:read, not repo:*). External token TTL often 5–10 minutes. User-delegated access via OIDC device flow with a human approval gate so the agent never sees the full authorization code.

“Login once, act forever” is a human-app habit. Agents need to authenticate as often as the capability is dangerous.

---

Replay Prevention: Nonces and Fail-Closed

Even five-minute tokens can be replayed. Every MCP/gateway request carries a unique nonce in the JWT or signed envelope. The gateway records the nonce in a Redis TTL store partitioned per tenant. Duplicate nonce → 403, regardless of token signature validity. Nonce TTL matches token TTL — no infinite store growth.

The availability trap:

If the nonce store is down, fail closed — reject requests that need nonce checks. The availability cost is visible; silent security degradation is not. An attacker who captures a token can induce store outage to create a replay window. Fail-open mode hands them that window.

Making fail-closed operable: Redis Sentinel/Cluster with three or more nodes; gateway readiness fails when the nonce store is unreachable; page nonce-store outages at gateway severity; include store RTO in gateway RTO documentation.

ATR validation, expiry checks, and Panguard continue independently — fail-closed scopes to replay protection only.

---

Bootstrap Patterns

Pattern 1: One-time wrapped bootstrap

Operator obtains a Vault wrapping token (single use, short TTL). Agent starts with only that wrapper in memory or a 0600 file deleted after read. Agent unwraps, receives AppRole secret_id or short Vault token, immediately discards the wrapper, performs login, stores nothing durable except possibly a local OS keychain reference to a refresh handle that is itself short-lived.

Pattern 2: Instance identity → exchange

On cloud VMs / Kubernetes: agent proves runtime identity (K8s SA token, AWS IMDS → IAM role, GCP metadata). Gateway or Vault JWT auth / Kubernetes auth issues a Vault token bound to that identity. No long-lived VAULT_TOKEN in the image.

Pattern 3: Edge node with intermittent connectivity

On enrollment, device pairs and receives a client cert plus enrollment proof. When online, exchanges for leases needed for the work queue. When offline, capability shrinks — pre-segmented local tools only, no cached cloud admin keys “so offline mode works.”

Enrollment (once)
  → device identity + short proof
Online work
  → exchange proof for leases (minutes)
Compromise signal
  → revoke leases + deny new exchanges

Bootstrap is allowed to be slightly awkward. Daily runtime credentials must be boringly ephemeral.

---

Audit: Prove What Was Issued

Every Vault and gateway auth event lands in WORM storage:

• Token issuance, exchange, rotation, revocation
• Agent ID, session ID, tool name
• Token accessor — never the raw token
• ATR claims presented
• Panguard decision

Presidio redaction before shipping so secret values never appear — only paths and accessors. A Merkle root of the audit stream into metrics on an interval makes tampering detectable.

If you can’t answer “did this agent receive credentials for production DB at 14:03 UTC?” from immutable logs, Part 15 incident investigations become guesswork.

---

Secrets and Agent Identity Lifecycle

Secrets sprawl is usually a lifecycle failure.

Provisioning: agents are security principals, not just Deployments. The provisioning PR includes agent ID, ATR role, justification, proposed claims, and an expiry review date. Automated pipeline creates Vault policy, cert, namespaces, and memory store init — and logs the approver to WORM.

Sprawl detection: weekly scan for credentials in ENV, config, agent memory, and git history outside declared locations. Finding = critical: stop agent, revoke, rotate, investigate.

Orphans: reconcile active Vault leases / certs / subscriptions against running agents. Orphans get lease suspend (preserve forensics), drain, read-lock memory, then timed decommission.

Under compromise: isolate (network + session quarantine), preserve memory and lease history, revoke certs and leases, hand forensic bundle to IR. Never revoke-and-wipe so hard you destroy the only evidence of which secret was abused.

Every dynamic secret needs an owning agent identity and a human owner. Credentials without owners become permanent.

---

Worked Example: Rotating a GitHub Tool Credential

Before: GITHUB_TOKEN=ghp_... in .env.

Provision: diag-bot approved with ATR role repo-read-only. Vault policy allows github/token/diag-bot issuance only.

Session start: device-paired human or workload identity → Session JWT.

Tool call list_pull_requests: gateway exchanges Session JWT → tool-scoped token → Vault GitHub plugin or Actions OIDC-style app token with repo:read, TTL 5 minutes.

Gateway calls GitHub, returns JSON to agent, drops credential.

Anomaly: unexpected create_deploy_key attempt → Panguard block → Vault lease revoke + session kill.

Audit: WORM shows accessors and tool names; no raw ghp_ in logs.

Blast radius comparison:

---

Common Failure Modes

---

Getting Started

You’re in a healthier place when: no production API key, cloud access key, or DB password lives indefinitely in .env, compose files, or agent memory; agents can’t call Vault with standing broad tokens; tool credentials expire in minutes and are non-reusable beyond their nonce/TTL rules; external SaaS access uses OAuth/OIDC scopes, not pasted PATs in prompts; local interactive use requires device pairing; the nonce store is HA and fail-closed; Vault audit goes to WORM with accessors only; every agent identity has a provisioning record, owner, sprawl scan, and decommission checklist; you can revoke fleet credentials in roughly session-TTL time when Part 15’s kill switch fires.

Part 3 answers what Vault alone can’t: even a five-minute token is dangerous if it’s an admin token. We’ll map agent tasks onto least-privilege scopes and ATR claims so Secret-as-a-Service issues the smallest capability that still lets the agent work.

---

Companion: DevSecOps-boilerplate. Docs: Secrets at rest · Authentication and session management · Agent identity lifecycle.