Admin tokens for troubleshooting turn a minor compromise into a takeover. Map agent tasks to tight JWT scopes, Kubernetes identities, and ATR claims enforced at every tool call.
Part 1 closed anonymous telemetry writes. Part 2 stopped parking long-lived secrets on the agent host. This post closes the privilege hole that remains even when secrets are short-lived: the agent still holds too much authority once it can act at all.
Admin “Just for Troubleshooting”
An on-call engineer is debugging a flaky summarizer agent in the agents namespace. Loki queries fail with permission errors. Someone pastes a ClusterRoleBinding “temporarily,” or the session JWT for the agent is minted with exec, Vault admin, and file:write across every tenant prefix “so we can finish the ticket.”
Two days later a prompt-injected document steers the same agent through a tool call that would have been blocked under a narrow ATR profile. The call succeeds. The agent now has Kubernetes API verbs it never needed, a Vault path tree it can read or renew, write access outside its workspace, possibly cloud roles assumed via a shared ServiceAccount.
The compromise wasn’t “the model went rogue.” The compromise was authority available before the bad instruction arrived. The blast radius was baked into the identity.
If “troubleshoot” means “give it admin,” you’re designing incident response that starts after takeover.
Three Layers of “Who Is This Agent?”
A ClawQL agent is not one secret. It’s a stack of identities that must each be least-privilege:
| Layer | What it answers | Typical artifact |
|---|---|---|
| Platform identity | Which workload is this in the cluster? | Dedicated ServiceAccount, Role/RoleBinding, IRSA/WI |
| Session identity | Which agent/tenant/session is acting? | Session JWT with ATR claim set |
| Call identity | Which tool may run right now? | Short-lived tool-scoped token + nonce |
| Runtime enforcement | Does this parameter match the claim? | Panguard + JSON Schema + path/HITL gates |
Part 2 gave you Secret-as-a-Service: tickets instead of permanent keys. Part 3 requires that those tickets be narrow. A five-minute Vault lease with cluster-admin is still a five-minute catastrophe window.
Short TTL without tight scope is a faster way to lose the same farm.
The Architecture Pattern: Role-Based Agent Scoping
Every agent task maps to a named role (DiagnoseService, QueryMetrics, SummarizeWorkspace, PerformRollback). Every role maps to an explicit ATR claim set — no implied “same as prod admin.” Platform RBAC and cloud IAM for that workload match the same story — no shared agents-sa. Session JWTs carry only that claim set; tool tokens carry a subset for one tool. Panguard denies anything the schema or claim forbids before handler code runs. Scope expansion is a controlled security event; scope contraction is always safe and immediate.
If you can’t write the task → claim matrix on one page, the agent isn’t ready to provision.
Task → Permission Matrix
Start with jobs, not tools. Tools are how jobs happen; jobs are what you authorize.
| Task | Tools (examples) | ATR claims (examples) | Must not include |
|---|---|---|---|
| QueryMetrics | metrics read, Loki query | telemetry:read:namespace:X | DB write, exec, Vault admin |
| DiagnoseService | logs read, describe deploy | k8s:get:pods, telemetry:read | create/delete, secrets write |
| SummarizeWorkspace | file_read, file_write | file:read:workspace, file:write:workspace | Paths under .clawql, host FS, exec |
| PerformRollback | deploy rollback | k8s:patch:deployments (named) + HITL | Cluster-wide * verbs |
Concrete ATR rule shape:
- claim: file:write:workspace
tool: file_write
allowedPaths:
- '/workspace/'
deniedPaths:
- '/workspace/.clawql/'
Claims live in the session JWT and are validated on every tool call. Agents cannot self-assign new claims. Rule changes require four-eyes approval; monthly reports should show ATR violations by rule ID and agent role.
Layer 1: Platform Identity
One ServiceAccount per workload — never one SA for the whole agents namespace. automountServiceAccountToken: false unless the pod truly needs the Kubernetes API. Prefer namespace-scoped Role, not ClusterRole, unless cluster scope is documented and reviewed. No wildcards in production (verbs: ["*"], resources: ["*"]).
apiVersion: v1
kind: Pod
metadata:
name: summarizer
spec:
serviceAccountName: summarizer-sa
automountServiceAccountToken: false
containers:
- name: agent
image: harbor.example.com/golden/node@sha256:...
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: summarizer-read-config
namespace: agents
rules:
- apiGroups: ['']
resources: ['configmaps']
resourceNames: ['summarizer-config']
verbs: ['get']
For cloud credentials, bind this ServiceAccount via IRSA / Workload Identity:
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/summarizer-role
Trust policy condition must nail the subject:
{
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/ABC123:sub": "system:serviceaccount:agents:summarizer-sa"
}
}
Cloud AssumeRole scoped to the wrong ServiceAccount is ClusterRole with better branding.
Layer 2: Session JWT vs Tool-Scoped Token
Session JWT (issued at session start) carries agent ID, tenant ID, session ID, and the ATR claim set. It’s not presented raw to tool handlers.
Tool-scoped token (gateway ↔ Vault exchange) carries: audience bound to one tool, subset of ATR claims for that invocation, TTL measured in minutes (default max 5 minutes), unique nonce — no renewal, no reuse on another tool.
Illustrative session payload:
{
"sub": "agent:summarizer",
"agent_id": "summarizer",
"tenant_id": "tenant-123",
"session_id": "sess-abc123",
"atr": ["file:read:workspace", "file:write:workspace"],
"exp": 1760003600
}
Illustrative tool token:
{
"aud": "tool:file_write",
"tool": "file_write",
"atr": [
{
"claim": "file:write:workspace",
"allowedPaths": ["/workspace/"],
"deniedPaths": ["/workspace/.clawql/"]
}
],
"nonce": "01JABCDE9VZ8Q3...",
"exp": 1760000300
}
Replay protection is fail-closed. Every MCP request includes a unique nonce. The gateway records it in a TTL store that expires with the token. Duplicate nonce → 403 even if the JWT still verifies. If the nonce store is down, fail closed — a replay window is worse than a brief tooling outage.
Audit each exchange without logging the secret:
{
"event_type": "tool_token_exchange",
"agent_id": "summarizer",
"session_id": "sess-abc123",
"tool_name": "file_write",
"token_accessor": "vault-accessor-xyz",
"atr_claims": ["file:write:workspace"],
"panguard_decision": "allow"
}
Layer 3: Panguard
The choke point between the MCP protocol handler and the tool dispatcher:
Agent tool request
→ MCP protocol handler
→ Panguard
· ATR claims allow this tool + params?
· JSON Schema valid? (additionalProperties: false)
· Path traversal / deniedPaths?
· Rate / cumulative session risk?
· HITL required?
→ tool handler (or 403)
Schema that earns its keep:
{
"type": "object",
"additionalProperties": false,
"required": ["path", "content"],
"properties": {
"path": {
"type": "string",
"maxLength": 4096,
"pattern": "^/workspace/(?!\\.clawql/).+"
},
"content": { "type": "string", "maxLength": 1048576 }
}
}
Validate outputs too — undeclared response fields are a smuggling channel.
HITL for irreversible verbs: require human review for exec, broad file_write, vault_secret_read, and external mutations. Default: fallback: deny, timeout_seconds: 300. Timeout without approval is denial.
The trust boundary for agents is the structured tool call — not the natural-language preamble.
Layer 4: Provisioning as a Security Principal
Agents are long-lived principals with certificates, Vault policies, memory paths, NATS namespaces, and ATR roles. Treat onboarding like joining a workforce — not like spinning a Deployment.
requested_by: [email protected]
agent_id: summarizer
atr_role: summarizer-basic
justification: 'Summarizes documents in tenant workspaces'
proposed_claims:
- file:read:workspace
- file:write:workspace
expiry_review_date: '2026-10-14'
Hard requirements: ATR role already exists and is documented; exec / admin-class claims need explicit justification; no agent without an expiry review date.
Scope expansion is a security event: PR + written justification + owning team + security sign-off + red-team case that exercises the new scope + 7-day observation window with heightened Panguard logging. Contraction is always safe and applied immediately.
Orphan handling: weekly reconcile Vault leases, NATS subscriptions, cert-manager Certificates, and running pods. Suspend (not destroy) leases for forensics, drain queues, read-lock memory, 7-day review, then automatic decommission.
Under compromise: isolate → preserve → revoke → hand off. Never delete memory or audit trail mid-investigation.
Getting Started
Inventory agents; kill shared ServiceAccounts and auto-mounted tokens you can’t justify. Write a one-page task → ATR matrix for the next agent you ship; refuse silent “admin for debug.” Wire session JWT → tool-scoped exchange with nonce store fail-closed. Put Panguard in front of every MCP tool dispatch; schema additionalProperties: false. Require expiry_review_date in every agent provisioning PR. Run kubectl auth can-i --list and a Vault/NATS/orphan reconcile this week.
Part 4: process containment — when a compromised agent still tries to spawn npx or curl despite your beautiful JWT.
Companion: DevSecOps-boilerplate. Docs: K8s least privilege · Scoped tokens · Agent lifecycle · Panguard ATR.
