Secret Rotation Patterns for Local/Edge Agents

This is Part 2 of Hardened Agentic Stack. Part 1 closed anonymous write access to telemetry. This essay closes the next hole: how a high-privilege agent obtains and renews the credentials it needs to act — without parking long-lived secrets in .env, Docker Compose, or Kubernetes Secret objects that are one cat away from exfiltration.

We consolidate the production curriculum from ClawQL’s secrets and auth modules — especially Secrets at Rest, Authentication & Session Management, and Agent Identity Lifecycle — into one builder-facing lesson: Secret-as-a-Service.

The “Oh No” moment: the laptop that held the kingdom

Picture a ClawQL edge agent on a developer laptop or a remote clawql-agent --mode=edge node. To “just get it working,” the .env looks familiar:

OPENAI_API_KEY=sk-live-...
GITHUB_TOKEN=ghp_...
DATABASE_URL=postgres://agent:SuperSecret@db/prod
NATS_CREDS=/Users/dev/.config/clawql/user.creds
VAULT_TOKEN=hvs.CAES...   # “temporary,” three months old

Two weeks later someone clones a malicious skill, a prompt injection steers a tool that shells out, or malware on the laptop dumps process memory and the home directory. The attacker does not need to defeat Vault, mTLS, or Panguard. They already own every long-lived credential the agent was given “for convenience.”

That is the Nutrient lesson applied to secrets: the static credential is the attack runway. In Module 8 of the ClawQL curriculum this is stated without euphemism — a secret written to a config file, environment variable, or Kubernetes Secret is no longer a secret. It is data waiting to be discovered. Agentic platforms make the problem worse because the same pod or process reuses that credential across dozens of sessions and hundreds of tool calls.

Static secrets create three systemic failures:

Readable by breadth. Kubernetes Secrets are base64, not encryption. Anyone with the wrong RBAC — or etcd access — can read them. Local .env files are worse: world-readable mistakes, IDE indexing, shell history, and backup agents all expand the audience.
Accumulating holders. Every developer laptop, CI runner, former contractor, and forgotten compose stack that ever received the value remains a holder until rotation finally happens — often months later.
Breach windows measured in months. Dynamic credentials with minute-scale TTLs collapse that window. Static ones do not.

Lesson: if compromise of the agent host implies compromise of every downstream system the agent could talk to, you do not have secrets management. You have credential caching with hope.

Learning goal for this post

By the end you should be able to:

Explain why ENV and Kubernetes Secret objects are the wrong durable store for agent credentials.
Describe the bootstrap → exchange → short-lived lease lifecycle.
Stand up (or specify) Vault (or Cloudflare Secrets for smaller setups) so agents never hold permanent cloud/API/database keys.
Separate Session JWTs from tool-scoped tokens, and know why each exists.
Apply replay protection with a fail-closed nonce store.
Connect secret leases to agent identity lifecycle (provision, sprawl detection, decommission, compromise).

Companion code and compose sketches: DevSecOps-boilerplate.

ClawQL context: what the agent actually needs secrets for

A ClawQL deployment is not a single microservice with one database password. Typical secret surfaces include:

Surface	Examples	Why it hurts if static
Inference	Provider API keys, gateway signing keys	Stolen key = unbounded model spend + data exfil channel
Tools	GitHub, cloud APIs, SaaS tokens	Stolen token = production change authority
Control plane	NATS creds, MinIO/S3 keys	Stolen creds = bus/read of other agents’ workspaces
Data stores	DB roles, Redis, memory backends	Stolen DB URL = tenant data
Platform	Vault tokens, mTLS key material	Stolen Vault token = secret factory access

Local and edge agents amplify risk because the trust boundary is the developer machine or remote node, not a locked-down cluster alone. Part 11 covers edge networking. Part 2 assumes a harder truth: even a perfect network policy fails if the secret is already on disk in plaintext.

The architecture pattern: Secret-as-a-Service

Name it so the rest of the series can point at it.

Secret-as-a-Service means:

The agent holds a bootstrap ticket or session identity, not permanent third-party secrets.
At need, a trusted exchange point (usually the gateway) trades that identity for a short-lived lease.
The lease TTL matches the work (often minutes), then dies — or is revoked on anomaly.
Every mint, renew, and revoke is written to a tamper-evident audit trail (accessor only — never the secret value).

Text sequence for the happy path:

Agent                Gateway                 Vault              External API / DB
  |                     |                      |                      |
  |-- Session JWT ----->|                      |                      |
  |   (+ device/workload proof)                |                      |
  |                     |-- validate session,  |                      |
  |                     |   ATR, nonce ------->|                      |
  |                     |-- exchange / mint -->|                      |
  |                     |<-- short-lived lease-|                      |
  |                     |----------------------|-- leased call ------>|
  |                     |<---------------------|-- result ------------|
  |<-- tool result -----|                      |                      |
  |   (secret never returned to agent)         |                      |
  |                     |                      |-- lease expires ---->|
  |                     |                      |   or revoke kills    |
  |                     |                      |   credential         |

Compare that to the anti-pattern:

Agent process
  └─ ENV: CLOUD_KEY=AKIA... (lives for months)
       └─ every tool call reuses AKIA...

Lesson: the agent should be able to request capability, not possess durable authority.

Why “just use Kubernetes Secrets” is not enough

Teams often graduate from .env to sealed-looking cluster Secrets and stop. Module 8 is explicit: Kubernetes Secrets are not encryption at rest by default; they are base64 encoding plus RBAC hope. Useful as a transport for bootstrap material in tightly locked namespaces — disastrous as the system of record for production credentials an agent will hold for weeks.

Prefer this layering:

Layer	Role
Vault (or cloud KMS + short-lived cloud tokens)	System of record; dynamic issuance; audit
Gateway exchange	Agents never speak Vault protocol with broad tokens
Kubernetes Secret / Workers Secret	Thin bootstrap only (AppRole wrapped token, one-time init)
Agent memory / logs	Must never store raw credentials (redaction + policy)

Infrastructure fix A: HashiCorp Vault as the secret factory

Design principles (from the curriculum)

Dynamic secrets only. Unique credential per request with short TTL.
No static credential in config, ENV, agent memory, or git.
HA Vault with Raft integrated storage; three+ replicas in production (quorum (n/2)+1).
Agents do not hold long-lived Vault tokens. Exchange at the gateway.

HSM-backed unseal

Vault’s master key protects the keyring that protects everything else. Curriculum recommendation:

Auto Unseal via AWS KMS, GCP Cloud KMS, or Azure Key Vault (HSM-backed).
Master key material stays in tamper-resistant hardware — not on disk next to the Vault binary.
Keep Shamir shares only for true break-glass, requiring multiple human key holders simultaneously.

Lesson: availability must not depend on a human unlocking Vault after every pod restart, but recovery of the kingdom must never depend on a single USB stick in one engineer’s drawer without controls.

Dynamic engines you will actually use for agents

Need	Vault engine	TTL intuition
Postgres / MySQL roles	Database secrets engine	Session length or shorter
AWS / GCP temporary access	Cloud secrets engines	Minutes; task-scoped
Internal mTLS	PKI engine	Hours/days max; prefer short
Shared app config that must be static-ish	KV v2 only for non-bearer material (e.g. public JWKS), still with tight ACL	Prefer not for API keys

Revocation is lease-shaped: when the session ends or Panguard/Falco signals compromise, revoke the lease and the credential dies everywhere it was issued for.

Example: KV for a JWT signing key + one-time bootstrap

For the Part 1 ingest Worker’s signing key (from the observability essay), KV v2 is acceptable for the minting service, not for every agent laptop:

vault secrets enable -path=observability kv-v2

vault kv put observability/worker/jwt-signing-key \
  key="$(openssl rand -base64 32)"

Bootstrap a Worker or gateway with a single-use token — not a root token in 1Password forever:

vault token create \
  -policy=observability-worker-read \
  -use-limit=1 \
  -ttl=5m

Example: policy shape that does not cross tenants

Policies are one-per-agent-role, path-tight, deny-heavy:

# Simplified illustration — never use broad wildcards across tenants
path "secret/data/tenants/{{identity.entity.aliases.kubernetes.path}}/agents/{{identity.entity.name}}/*" {
  capabilities = ["read"]
}

path "database/creds/agent-readonly" {
  capabilities = ["read"]
}

# Explicit denies win
path "secret/data/tenants/+/admin/*" {
  capabilities = ["deny"]
}

Curriculum rules worth memorizing:

Path patterns scoped per tenant and agent — not secret/*.
Explicit denials for admin/break-glass paths.
Policy changes require the same multi-party approval you use for ATR rule changes.

Gateway-as-exchange-point (non-negotiable)

Module 8’s secure exchange:

Agent presents Session JWT to the gateway.
Gateway exchanges it for a short-lived Vault token (or directly for a dynamic secret) scoped to that agent’s policy.
Vault token TTL matches the session or the single tool call — not “until reboot.”
Token is never logged, never returned to the agent for storage, and ideally never leaves the gateway process beyond the call.

If the agent process can echo $VAULT_TOKEN and call Vault directly with broad rights, you have rebuilt static secrets with extra steps.

Lesson: Vault without the exchange pattern is still a secret server. The exchange pattern is what makes Vault a secret service relative to the agent.

Infrastructure fix B: Cloudflare Secrets and small-footprint edge

Not every deployment runs HA Vault on day one. Part 1’s Worker can store minting keys via:

echo "your-signing-key" | wrangler secret put JWT_SIGNING_KEY

That is appropriate when you need encrypted-at-rest edge secrets without cross-service dynamic DB roles yet. It is not a substitute for:

Automated rotation with audit
Per-session database users
Instant fleet-wide revocation stories tied to agent identity

Progressive path many teams take:

Workers Secrets / cloud secret manager for edge minting keys.
Introduce Vault (or cloud IAM roles + STS) for tool credentials.
Move agents fully onto gateway exchange so local ENV contains only bootstrap material.

Lesson: pick the control that matches your blast radius today, but do not let “we use Wrangler secrets” become an excuse for OPENAI_API_KEY on every laptop for a year.

Session JWTs vs tool-scoped tokens (auth is also secrets)

Module 9 reframes authentication for agents: a session can last hours and fire hundreds of tool calls per minute. A single long-lived session bearer that authorizes every tool call is just a static credential with better branding.

Issue a Session JWT — then refuse to use it for tools

At session start, mint a Session JWT containing at least:

Agent ID
Tenant ID
Session ID
ATR claims (coarse, session-level)

Do not send that JWT to tool handlers as the sole authorization for every call.

Exchange for a tool-scoped token per invocation

Session JWT
  → Gateway validates
  → Issues tool-scoped token (only claims for this tool)
  → TTL ≤ 5 minutes, non-renewable, single-purpose
  → Audit: tool name, claims, timestamps, token accessor

Stealing a tool-scoped token should be a short, narrow incident — not a week-long admin session.

External APIs: OAuth/OIDC, not pasted keys

Never embed static SaaS API keys in agent context or memory.

Client credentials or auth-code flows for service access.
Fetch on demand, use once, discard.
Minimum scopes (repo:read, not repo:*).
External token TTL often 5–10 minutes (harder to revoke remotely than internal Vault leases).
User-delegated access: OIDC device flow + human approval gate so the agent never sees the full authorization code.

Local humans vs CI

Actor	Second factor pattern
Human on local gateway	Device pairing with hardware-backed key (YubiKey / platform authenticator). Stolen JWT alone cannot start a new session from an unpaired device.
CI runner	OIDC workload identity federation (GitHub Actions / GitLab). Exchange workflow OIDC token for a session JWT with CI-minimum ATR claims.

Lesson: “login once, act forever” is a human-app habit. Agents need authenticate as often as the capability is dangerous.

Replay prevention: nonces and fail-closed

Even five-minute tokens can be replayed. Module 9’s pattern:

Every MCP/gateway request carries a unique nonce in the JWT or signed envelope.
Gateway records the nonce in a Redis TTL store (partitioned per tenant in multi-tenant deploys).
Duplicate nonce → 403, regardless of token signature validity.
Nonce TTL matches token TTL — no infinite store growth.

The availability trap (curriculum addendum)

If the nonce store is down:

Mode	Meaning	Verdict
Fail open	Skip nonce checks	Wrong. Attacker who captured a token gets free replay for the rest of the TTL — and can induce store outage to create that window.
Fail closed	Reject requests that need nonce checks	Correct. Availability cost is visible; security degradation is not silent.

Make fail-closed operable:

Redis Sentinel/Cluster, 3+ nodes, not a single instance under the gateway’s SLA.
Gateway readiness fails when the nonce store is unreachable (LB drains the replica).
Page nonce-store outages at gateway severity.
Include store RTO in gateway RTO documentation.

ATR validation, expiry checks, and Panguard continue independently — fail-closed scopes to replay protection, not “turn off all auth.”

Lesson: a security dependency that fails open under attack is a security control that only works on sunny days.

Bootstrap patterns for local and edge agents

This is the Part 2 “how do I actually start the binary” section.

Pattern 1: One-time wrapped bootstrap

Operator or provisioner obtains a Vault wrapping token (single use, short TTL).
Agent starts with only that wrapper in memory or a 0600 file deleted after read.
Agent (or sidecar) unwraps → receives AppRole secret_id or short Vault token.
Immediately discards wrapper; performs login; stores nothing durable except maybe a local OS keychain reference to a refresh handle that is itself short-lived.

Pattern 2: Instance identity → exchange

On cloud VMs / Kubernetes:

Agent proves runtime identity (K8s SA token, AWS IMDS → IAM role, GCP metadata).
Gateway or Vault JWT auth / Kubernetes auth issues a Vault token bound to that identity.
No long-lived VAULT_TOKEN in the image.

Pattern 3: Edge node with intermittent connectivity

On enrollment, device pairs and receives a client cert + enrollment proof.
When online, exchanges for leases needed for the work queue.
When offline, capability shrinks — do not cache cloud admin keys “so offline mode works.” Offline mode should use pre-segmented local tools only.

Enrollment (once)
  → device identity + short proof
Online work
  → exchange proof for leases (minutes)
Compromise signal
  → revoke leases + deny new exchanges

Lesson: bootstrap is allowed to be slightly awkward. Daily runtime credentials must be boringly ephemeral.

Audit: prove what was issued without creating new stealables

Every Vault and gateway auth event should land in WORM storage (S3 Object Lock COMPLIANCE, or equivalent):

Token issuance, exchange, rotation, revocation
Agent ID, session ID, tool name
Token accessor — never the raw token
ATR claims presented
Panguard decision

Presidio (or equivalent) redaction before ship so secret values never appear — only paths and accessors. Module 8 also recommends a Merkle root of the audit stream into metrics on an interval so tampering is detectable.

Lesson: if you cannot answer “did this agent receive credentials for production DB at 14:03 UTC?” from immutable logs, you cannot investigate Part 15 incidents honestly.

Tie secrets to agent identity lifecycle

Secrets sprawl is usually lifecycle failure. From Module 10:

Provisioning

Agents are security principals, not “just Deployments.” Provisioning PR should include agent ID, ATR role, justification, proposed claims, and an expiry review date. Automated pipeline then creates Vault policy, cert, namespaces, memory store init — and logs the approver to WORM.

Sprawl detection

Weekly scan for credentials in ENV, config, agent memory, and git history outside declared locations. Finding = critical: stop agent, revoke, rotate, investigate.

Orphans

Reconcile active Vault leases / certs / subscriptions against running agents. Orphans get lease suspend (preserve forensics), drain, read-lock memory, then timed decommission.

Compromise order (preview of Part 15)

Isolate (network + session quarantine)
Preserve memory, lease history, message logs
Revoke certs and leases
Hand forensic bundle to IR

Never revoke-and-wipe so hard you destroy the only evidence of which secret was abused.

Lesson: every dynamic secret needs an owning agent identity and a human owner. Credentials without owners become permanent.

Worked example: rotating a GitHub tool credential

Educational walkthrough end-to-end.

Before: GITHUB_TOKEN=ghp_... in .env (classic fail).
Provision: agent diag-bot approved with ATR role repo-read-only. Vault policy allows github/token/diag-bot issuance only.
Session start: device-paired human or workload identity → Session JWT.
Tool call list_pull_requests: gateway exchanges Session JWT → tool-scoped token → Vault GitHub plugin or Actions OIDC-style app token with repo:read, TTL 5 minutes.
Gateway calls GitHub; returns JSON to agent; drops credential.
Anomaly: unexpected create_deploy_key attempt → Panguard block → Vault lease revoke + session kill.
Audit: WORM shows accessors and tool names; no raw ghp_ in logs.

Compare blast radius:

Design	Attacker with host access gets
Static `.env`	Durable GitHub token until manual rotation
Secret-as-a-Service	At most a minutes-lived lease, often already expired; minting requires gateway + identity

Common failure modes (study these)

Failure	Why it happens	Fix
“We put Vault tokens in ENV for convenience”	Ops urgency	Bootstrap wrap + exchange only
KV used for everything	Dynamic engines feel hard	Prefer DB/cloud engines for bearer creds
Fail-open nonce store	Fear of downtime	HA Redis + readiness + pages
Agent logs tool args including tokens	Debug culture	Structured redaction; never log Authorization
Scope expansion via hot rewrite	“Just unblock the agent”	PR + trial window + heightened Panguard logging
Offline cache of admin cloud keys	Edge product pressure	Shrink offline capability instead

Safety check

You are “trusted enough” on secrets when:

No production API key, cloud access key, or DB password lives indefinitely in .env, compose files, or agent memory.
Agents cannot call Vault with standing broad tokens; gateway (or equivalent) performs exchange.
Tool credentials expire in minutes and are non-reusable beyond their nonce/TTL rules.
External SaaS access uses OAuth/OIDC scopes, not pasted PATs in prompts.
Local interactive use requires device pairing (or equal); CI uses workload OIDC.
Nonce store (if used) is HA and fail-closed.
Vault (or cloud secret system) audit goes to WORM with accessors only.
Every agent identity has provisioning record, owner, sprawl scan, and decommission checklist — including forensic-first revoke under compromise.
You can revoke fleet credentials in roughly session-TTL time when Part 15’s kill switch fires.

How this connects to the series

Part	Relationship to secrets
1 — Zero-trust ingest	Ephemeral JWTs for telemetry write; signing keys live here in Vault/Workers Secrets
2 — This post	How agents obtain any credential safely
3 — Scoped credentials	Maps tasks → IAM/JWT scopes so leases are not “admin with a short TTL”
4–7	Kernel/sandbox layers assume stolen host ≠ stolen permanent cloud keys
15	Revoke leases + preserve forensic lease history

What’s next

Part 3 answers the question Vault alone cannot: even a five-minute token is dangerous if it is an admin token. We will map agent tasks (DiagnoseService, QueryMetrics, PerformRollback) onto least-privilege scopes and ATR claims so Secret-as-a-Service issues the smallest capability that still lets the agent work.

Draft outline (until published): Scoped Credentials: The Least-Privilege Agent.

Source modules consolidated here

Secrets at Rest: Vault, HSM, audit
Authentication & session management
Agent identity lifecycle
Related deep dive on ingest signing keys: Why Serverless Isn’t a Mistake…