A comprehensive guide to ClawQL’s five-layer memory architecture — OKF vault, PageIndex, CodeGraph, Onyx semantic search, and vector recall — and how they compose into persistent, auditable, sovereign agent intelligence.
This pairs with the institutional knowledge tax (why re-explanation compounds and what cross-session recall requires), the Enterprise Ontology (entity instances as vault knowledge), twelve layers of LLM cost, and memory residency in the Hardened Agentic Stack. OKF serialization notes: docs/memory/okf.md. Token-efficiency layers: docs.clawql.com/architecture/token-efficiency.
Format note: the vault ships OKF-compatible Markdown by default (memory_ingest writes .md with required type). .cqk is the ClawQL Knowledge extension for the same frontmatter contract (ADR 0010) — optional promotion, not a hard break from OKF. Examples below use .cqk for clarity; treat .md with the same fields as the Day-1 path.
The Amnesia Problem
Every AI coding agent session starts the same way. You open Claude Code, Cursor, or Codex. You type “continue working on the authentication refactor.” The agent asks: “Could you give me some context about the codebase?”
It has no idea what the authentication refactor is. It doesn’t know why you chose JWT over sessions last week. It doesn’t remember that parseConfig is the function that keeps breaking during the token validation step. It doesn’t know that you tried bcrypt and switched to argon2 for specific performance reasons that took three hours to benchmark.
It starts from zero. Every single session.
This is not a model capability problem. The models are capable of remembering — they demonstrate this within a session. The problem is architectural: there is no persistent memory layer between sessions. When the context window closes, everything the agent learned is gone.
The cost of this amnesia is invisible because it’s distributed across thousands of micro-frictions: the context-setting at the start of every session, the re-explanation of architectural decisions, the redundant file reads to re-discover code structure the agent already understood yesterday, the re-benchmarking of approaches that were already tried and rejected.
Most coding agents waste 60–70% of their tokens re-discovering code structure on every task. When you ask Claude Code to “refactor parseConfig to support YAML,” the agent typically reads 5–15 files, runs grep on a few symbols, traces a couple of imports, then writes the actual change. That exploration is repeated every time — there’s no persistent memory of the codebase shape.
The token cost of codebase re-discovery is measurable. The productivity cost of context re-establishment is harder to quantify but larger. The organizational knowledge cost — the reasoning and decisions that never make it from one session to the next — is the one that matters most over time.
ClawQL’s memory stack addresses this at five distinct layers. Each layer addresses a different kind of memory that agents need. They compose. They are all persistent. They are all auditable. They are all yours.
Lesson: amnesia isn’t a prompt failure. It’s a missing persistence plane between sessions.
Five Kinds of Memory Your Agent Needs
Before describing the implementation, it’s worth being precise about what “memory” means for an AI agent. There are at least five distinct kinds:
Codebase memory — the structure of the code the agent is working on. Which files exist, how they relate, what functions are defined where, how routes connect to handlers, which modules import which. This is structural knowledge that should be pre-indexed, not re-discovered on every tool call.
Decision memory — the reasoning behind choices made in previous sessions. Why JWT instead of sessions. Why argon2 instead of bcrypt. Why the current architecture instead of the three alternatives that were considered and rejected. This is narrative knowledge that should be explicitly captured and recalled.
Entity memory — facts about the business domain. Which customers have which contracts. Which patients have which diagnoses. Which assets have which maintenance histories. This is typed, relationship-aware knowledge that should be queryable against a schema (the Ontology layer).
Cross-session context — what was happening at the end of the last session. What was in progress, what was blocked, what the next step was. This is temporal knowledge that bridges session boundaries.
Organizational knowledge — the accumulated intelligence of the entire team. Architectural decisions made by other engineers, debugging patterns that worked on similar problems, domain knowledge that took years to accumulate. This is collective knowledge that should be shared across the swarm.
Standard AI tooling provides zero of these five in a persistent form. ClawQL’s stack targets all five, with different technologies optimized for each kind.
The Five-Layer Memory Stack
Layer 5: Onyx — Semantic enterprise knowledge search (40+ connectors, hybrid recall)
Layer 4: CodeGraph — Pre-indexed codebase structure (token compression on code tasks)
Layer 3: PageIndex — Embedding-free hierarchical document retrieval
Layer 2: Vector Recall — Semantic similarity search over OKF vault entries
Layer 1: OKF Vault (.cqk / OKF .md) — Durable, auditable, structured knowledge storageEach layer builds on the one below. The OKF Vault is the persistence substrate — everything is stored there. Vector Recall is how you search it semantically. PageIndex is the alternative retrieval path that doesn’t require embeddings. CodeGraph is the specialized index for codebase structure. Onyx is the semantic layer over enterprise knowledge sources beyond the vault.
When memory_recall fires, it queries across active layers and ranks results by relevance, recency, and confidence. The agent doesn’t need to know which layer produced the answer. It receives typed, ranked, cited knowledge entries and uses them.
Layer 1: The OKF Vault — The Persistence Substrate
Every memory entry ClawQL stores is OKF-compatible: Markdown body with YAML frontmatter, file path as concept identity, concepts linked through wikilinks. The ClawQL Knowledge promotion (.cqk) carries the same contract with an explicit extension for tooling.
The OKF (Open Knowledge Format) standard establishes just enough convention — required type field, optional resource, description, tags, index.md catalog, log.md changelog — to make different producers and consumers interoperable without imposing a rigid schema. ClawQL extends OKF with worm_ref, correlation_id, agent_id, verdict, and confidence_score fields that tie every knowledge entry to the WORM audit trail.
The knowledge entry format
---
type: decision # required OKF field
title: 'Authentication: JWT over sessions'
description: 'Decision to use JWT for stateless auth rather than server-side sessions'
resource: null
tags: auth, jwt, security, performance
timestamp: 2026-07-14T14:32:00Z
confidence_score: 0.94
agent_id: agent-daniel-dev-01
correlation_id: sess-8821-tool-047
worm_ref: sha256:a1b2c3d4e5f6... # ClawQL extension — links to WORM entry
verdict: passed # Evaluator verdict — used in Flywheel export
session_id: sess-8821
project: clawql-auth-refactor
---
## Decision
Chose JWT (JSON Web Tokens) over server-side sessions for the authentication layer.
## Reasoning
Server-side sessions require a shared session store (Redis) that becomes a single
point of failure and complicates horizontal scaling. JWT tokens are stateless —
the server validates the token cryptographically without any external lookup.
## Constraints Considered
- Mobile clients cannot use cookies reliably → JWT in Authorization header
- Token invalidation before expiry requires a blocklist → acceptable for logout
(rare event) vs. the complexity of session management
## Alternatives Rejected
- **Server-side sessions**: Requires Redis cluster, adds latency on every request
- **OAuth2 delegation only**: Too heavy for internal API auth
- **API keys only**: No expiry mechanism, rotation complexity
## Related Decisions
[[Authentication: argon2 over bcrypt]]
[[Authentication: 15-minute access token TTL]]
## What To Watch
If horizontal scaling becomes complex, revisit. The JWT blocklist approach
(for logout) adds a Redis dependency anyway — at that point, sessions may
make more sense.The wikilinks at the bottom — [[Authentication: argon2 over bcrypt]] — create the knowledge graph. memory_recall can traverse these links the same way a human would navigate a wiki: start at one decision, follow the links to related decisions, build a coherent picture of the full decision space.
The worm_ref field connects this knowledge entry to its WORM audit record. When memory_ingest wrote this entry, it wrote a corresponding WORM entry with the session ID, the agent ID, the full prompt that generated the entry, and the Evaluator verdict. The knowledge entry is auditable — you can prove when it was created, by which agent, during which session, and whether it was verified correct.
The Vault Directory Structure
~/.ClawQL/
vault/ # Primary OKF knowledge bundle
index.md # OKF catalog — surveyed first on recall
log.md # append-only changelog of all entries
decisions/ # type: decision
auth-jwt-over-sessions.cqk
auth-argon2-over-bcrypt.cqk
arch-effect-ts-migration.cqk
context/ # type: context — cross-session bridges
session-2026-07-14-end.cqk
project-clawql-auth-status.cqk
entity_instances/ # type: entity_instance — populated Ontology
contracts/
contract-abc123.cqk
customers/
customer-acme.cqk
task_results/ # type: task_result — completed work records
weekly-summary-2026-07-14.cqk
security-patch-cve-2026.cqk
runbooks/ # type: runbook — static operational knowledge
deploy-to-prod.cqk
incident-response.cqk
errors/ # type: error — failure patterns + solutions
jwt-validation-timeout.cqk
postgres-connection-pool-exhaustion.cqk
patterns/ # type: pattern — recurring solutions
retry-with-exponential-backoff.cqk
effect-ts-error-channel-pattern.cqk
ontology/ # Schema definitions (Git-tracked, not synced as data)
entities/
Contract.cqe
Patient.cqeThe vault is stored in S3/R2, not Git. Git holds schemas (.cqe, .cqm files). R2 holds instances (knowledge entries). The distinction matters: schemas change through PRs and deploy pipelines. Knowledge accumulates continuously as agents work and should never require a deployment to persist.
clawql sync push writes new entries to R2. clawql sync pull retrieves them on other edge nodes. The virtual gateway holds the authoritative copy. Edge gateways maintain hot-tier local caches of recent entries for sub-millisecond recall.
The OKF Index: The First Stop on Recall
The index.md file is what makes the vault searchable without loading every entry:
# ClawQL Agent Memory Vault
## Decisions (47 entries)
- [[auth-jwt-over-sessions]] — Authentication token strategy
- [[arch-effect-ts-migration]] — TypeScript error handling architecture
- [[infra-pulumi-over-terraform]] — Infrastructure provisioning choice
...
## Context (12 entries)
- [[session-2026-07-14-end]] — Last session state — auth refactor
...
## Errors (23 entries)
- [[jwt-validation-timeout]] — JWT validation latency spike — RESOLVED
...
## Patterns (18 entries)
- [[retry-with-exponential-backoff]] — Standard retry pattern for Effect-TS
...When memory_recall fires, it reads index.md first. The agent surveys the catalog in one small context window before loading any full entries. This is the OKF spec’s key insight applied to agent memory: the index provides a structured catalog that enables targeted retrieval rather than blind embedding search over everything.
Scanning the index costs approximately 500 tokens. Loading the index plus the 5 most relevant full entries costs approximately 3,000 tokens. Loading every entry in a large vault would cost 50,000–200,000 tokens. The index-first pattern is what makes the vault scale beyond a few dozen entries.
The log.md: Session-Continuity Changelog
The log.md is an append-only record of every memory_ingest event:
## 2026-07-14T14:35:22Z — sess-8821
**Ingested:** decision/auth-jwt-over-sessions
**Agent:** agent-daniel-dev-01
**Project:** clawql-auth-refactor
**Summary:** JWT chosen over server-side sessions. Reasoning: stateless, horizontal scaling.
**WORM:** sha256:a1b2c3d4...
## 2026-07-14T14:38:07Z — sess-8821
**Ingested:** decision/auth-argon2-over-bcrypt
**Agent:** agent-daniel-dev-01
**Project:** clawql-auth-refactor
**Summary:** argon2id chosen over bcrypt. Reasoning: memory-hard, GPU resistance.
**WORM:** sha256:b2c3d4e5...When a new session starts, memory_recall reads log.md filtered to the last N days and the current project. The agent immediately sees what was worked on recently without loading any full entries. The log.md is the fastest possible session continuity mechanism: a dated changelog of what was captured, readable in a few hundred tokens.
Layer 2: Vector Recall — Semantic Similarity Search
The OKF vault stores knowledge as structured text. Vector Recall makes it semantically searchable — finding entries that are relevant to a query even when they don’t share keywords.
When memory_ingest creates an entry, it can also generate an embedding vector of the entry’s content (frontmatter + body) and store it alongside the file:
vault/
decisions/
auth-jwt-over-sessions.cqk # The knowledge entry
auth-jwt-over-sessions.vec # The embedding vector (Float32Array, 1536 dims)
indexes/
memory.db # SQLite with metadata + vector index
memory.db-shm
memory.db-walThe memory.db SQLite database maintains a lightweight inverted index and a pgvector-compatible vector store for similarity search. On edge gateway devices, this is a local SQLite file — zero network latency for recall. In the Virtual Gateway, it’s pgvector running in Postgres — shared across the team.
The Recall Query Pipeline
When memory_recall(query, options) fires:
Step 1 — Index survey: Read index.md. Identify candidate categories (decisions? errors? patterns?) based on the query type. Cost: ~200 tokens.
Step 2 — Full-text search: SQLite FTS5 searches entry titles, tags, and description fields for keyword matches. Fast, zero-cost, catches exact terminology matches.
Step 3 — Vector similarity: Embed the query using the same model that embedded the entries. Cosine similarity search in the vector index. Returns top-K entries by semantic similarity.
Step 4 — Wikilink traversal: For each high-scoring entry, follow wikilinks to related entries. Adds relational context — the JWT decision links to the argon2 decision links to the token TTL decision. The agent gets the full decision cluster, not just the nearest match.
Step 5 — Reranking: Cross-encoder reranking over the candidate set (FTS results + vector results + linked entries). Produces a final ranked list.
Step 6 — Confidence-filtered loading: Load full entry content only for entries above the confidence threshold. Return structured MemoryEntry objects to the agent.
The full pipeline typically costs 500–1,500 tokens for the index survey and structured results. Compare to loading every entry in a medium-sized vault (20,000–100,000 tokens). The pipeline is what makes recall practical at scale.
The Semantic Cache Integration
Layer 2 connects directly to the inference gateway’s semantic cache (Layer 5 of the 12-layer token efficiency stack). When a memory_recall query returns a high-confidence hit, the result can be flagged cache_eligible: true. Subsequent similar queries — different phrasing, same underlying intent — return the cached recall result rather than re-running the embedding pipeline.
This is the compounding efficiency: the memory vault gets cheaper to query the more it’s used, because semantically similar queries cache. The fine-tuning flywheel then trains on successful recall events — the model learns which vault entries are relevant for which query types, improving future retrieval without embeddings.
Layer 3: PageIndex — Embedding-Free Hierarchical Retrieval
PageIndex is a tree-based retrieval approach using hierarchical categories — LLM reasoning for search instead of embedding similarity. It is an alternative memory backend for agents that can’t or won’t run embeddings — no API key, no local GGUF model, no vector database required.
The insight: before vector embeddings existed, information retrieval was hierarchical. A library uses a Dewey Decimal system. A file system uses directories. A wiki uses categories. Hierarchical organization is cognitively natural, often more precise than semantic similarity for certain query types, and entirely free of embedding infrastructure.
How PageIndex Works
ClawQL’s PageIndex implementation maintains a category tree over the OKF vault:
knowledge/
technical/
authentication/
jwt/
auth-jwt-over-sessions.cqk
auth-jwt-15-minute-ttl.cqk
passwords/
auth-argon2-over-bcrypt.cqk
architecture/
effect-ts/
arch-effect-ts-migration.cqk
pattern-effect-ts-error-channels.cqk
infrastructure/
infra-pulumi-over-terraform.cqk
operational/
incidents/
...
runbooks/
...
domain/
contracts/
...
customers/
...The category tree is maintained by clawql-memory when entries are ingested — the LLM classifies each entry into the appropriate category path based on the entry’s type, tags, and content. The category assignment is stored in the entry’s frontmatter:
page_index_path: technical/authentication/jwt/
page_index_categories: [technical, authentication, jwt, stateless-auth]PageIndex Recall
When memory_recall runs with backend: page_index or when the embedding service is unavailable:
// PageIndex recall — no embeddings required
const recall = async (query: string): Promise<MemoryEntry[]> => {
// 1. LLM classifies the query into category path
const categories = await classifyQuery(query);
// Returns: ["technical", "authentication", "jwt"]
// 2. Walk the PageIndex tree to the matching path
const candidates = await pageIndex.walk(categories);
// Returns: all entries under technical/authentication/jwt/
// 3. LLM ranks candidates by relevance to query
const ranked = await rankByRelevance(query, candidates);
// 4. Return top-K entries
return ranked.slice(0, options.topK ?? 5);
};The LLM classification step uses a Frugal-tier model — Phi-4 or a local Ollama instance. Classification is a simple routing task that doesn’t need a Frontier-tier model. The total cost of a PageIndex recall is one cheap classification call plus one ranking call over a small candidate set.
When to Use PageIndex vs Vector Recall
PageIndex is better for:
- Precise categorical queries: “what decisions did we make about authentication?” — the category tree gives exact results
- Environments without embedding infrastructure (offline, air-gapped, cost-constrained)
- Large vaults where vector similarity gets noisy — the category hierarchy provides precision
- When the query is inherently categorical: “show me all error patterns for Postgres”
Vector Recall is better for:
- Semantic queries: “what did we try when the token validation was timing out?” — the error may be categorized differently but semantically similar to the query
- Cross-domain queries that span categories: “what performance optimizations have we made?” might span authentication, database, and infrastructure categories
- Novel queries where the right category isn’t obvious
ClawQL uses both simultaneously and merges results. PageIndex provides precision. Vector Recall provides semantic breadth. Reranking over the merged candidate set produces better results than either alone.
Lesson: embeddings are optional infrastructure, not the definition of memory. Hierarchy + FTS still recall when the embedding service is down — fail open on retrieval, not fail closed.
Layer 4: CodeGraph — Pre-Indexed Codebase Structure
CodeGraph hit GitHub #2 trending on May 23, 2026 — thousands of stars in a day — with a local knowledge graph aimed at cutting Claude Code, Codex, Cursor, OpenCode, and Hermes token spend on code exploration. When you ask Claude Code to “refactor parseConfig to support YAML,” the agent typically reads 5–15 files, runs grep on a few symbols, traces a couple of imports, then writes the actual change. That exploration is repeated every time.
Reported compression on large codebases (including the Swift Compiler suite and a broader multi-repo test set) lands in the same order of magnitude: far fewer tool calls, far fewer tokens, faster time-to-edit — because structure is pre-indexed instead of rediscovered.
ClawQL treats CodeGraph as a first-class component of the memory stack. The reasons are direct: codebase structure is the most frequently needed memory for software development agents, and re-discovering it on every session is the largest single source of avoidable token waste in the development workflow.
What CodeGraph Indexes
CodeGraph supports a broad language set (TypeScript, JavaScript, Python, Go, Rust, Java, C#, PHP, Ruby, C/C++, Swift, Kotlin, Dart, Lua, Svelte, and others) and detects route files/declarations in many web frameworks, connecting URL patterns with handler functions (Django, Flask, FastAPI, Express, NestJS, Laravel, Rails, Spring, Gin, Axum, ASP.NET, Vapor, React Router, SvelteKit, and others).
The index captures:
Symbol index — every function, class, interface, type, and variable definition, with their file paths, line numbers, and docstrings.
Import graph — which modules import which, enabling the agent to understand dependency relationships without tracing them manually.
Call graph — which functions call which, enabling the agent to understand execution flow without reading through implementations.
Route map — URL patterns connected to handler functions. When an agent needs to work on a specific API endpoint, CodeGraph tells it exactly which file and function handles that route.
Type hierarchy — class inheritance, interface implementation, and type relationships.
Change index — which files changed recently (from git log), weighted so recent changes surface first in queries.
Auto-Sync on File Changes
CodeGraph uses native filesystem events — FSEvents on macOS, inotify on Linux, ReadDirectoryChangesW on Windows — with debounced auto-sync. The index updates as local code changes, so users do not need to rebuild it manually after every edit.
This is the property that makes CodeGraph genuinely persistent rather than a one-time snapshot. The agent always has an up-to-date picture of the codebase structure. When a function is renamed, the index updates within seconds. When a new module is created, it’s indexed before the agent’s next query.
Nested Repository Support
CodeGraph descends into nested repos that have a real working tree on disk and indexes each as its own embedded repository, recursively, so every layer of a stacked workspace is covered. Active submodules are handled; a nested repo under a dependency directory such as vendor/ or node_modules/ stays excluded.
For ClawQL’s monorepo — multiple packages and submodule integrations — this is the difference between indexing a fraction of the codebase and indexing the packages that matter. Every package in packages/clawql-*/ can be indexed as an embedded repository. The import graph across packages stays correct.
CodeGraph in the MCP Tool Surface
ClawQL exposes CodeGraph through MCP tools that replace the most token-expensive common agent operations:
// Instead of: read_file, then grep, then read_file again...
cg_find_symbol('parseConfig');
// Returns: { file: "src/config/parser.ts", line: 47, signature: "parseConfig(...)", docstring: "..." }
// Instead of: ls, then read multiple files to understand structure...
cg_explain_module('packages/clawql-inference');
// Returns: module summary, exported symbols, key dependencies, recent changes
// Instead of: grep across all files...
cg_find_callers('processDocument');
// Returns: every call site, with file + line + surrounding context
// Instead of: manually trace imports...
cg_dependency_path('clawql-payments', 'clawql-core');
// Returns: the exact import chain between two modules
// Instead of: read multiple route files...
cg_find_route('/v1/chat/completions');
// Returns: { handler: "handleChatCompletion", file: "src/gateway/routes/completions.ts", middleware: [...] }Each of these replaces 3–15 tool calls with one tool call. The token savings are structural, not marginal.
CodeGraph as a Knowledge Entry Generator
CodeGraph and the OKF vault integrate bidirectionally. When an agent makes a significant change to the codebase — refactors a function, changes an API contract, adds a new module — ClawQL can generate a type: code_change entry that captures the change, its reasoning (from the agent’s session context), and its impact (from the CodeGraph diff).
---
type: code_change
title: 'parseConfig: added YAML support'
description: 'Extended parseConfig to handle .yaml and .yml files alongside .json'
timestamp: 2026-07-14T16:22:00Z
affected_symbols: [parseConfig, ConfigOptions, ConfigParser]
affected_files: [src/config/parser.ts, src/config/types.ts]
breaking_change: false
correlation_id: sess-8821-tool-089
worm_ref: sha256:c3d4e5f6...
---The change entry links the code change to the reasoning that drove it. Future agents querying “why does parseConfig support YAML?” find this entry through both the OKF vault and the CodeGraph symbol index. The knowledge is captured at the moment of creation, not reconstructed weeks later from git blame.
Layer 5: Onyx — Enterprise Semantic Knowledge
The first four layers handle agent-specific memory: the OKF vault for structured knowledge entries, Vector Recall for semantic search over those entries, PageIndex for embedding-free hierarchical retrieval, and CodeGraph for codebase structure. Onyx handles something different: the enterprise knowledge that already exists in Confluence, Notion, Slack, Google Drive, GitHub, Jira, and dozens of other sources.
Onyx is an open-source enterprise search platform (the same Onyx ClawQL uses throughout the IDP and inference stack). For the memory layer, it provides:
Pre-built connectors — Slack, Confluence, Notion, Google Drive, GitHub, Jira, Linear, Zendesk, HubSpot, Salesforce, email, and more. The organizational knowledge that already exists in these systems is indexed automatically.
Real-time indexing — streaming indexing so updates land quickly enough that the next agent query can see them.
Hybrid search — keyword + vector search combined. High precision on exact terminology (important for technical docs) with semantic breadth for conceptual queries.
Permission-aware retrieval — Onyx respects the permission model of each source system. A Confluence page marked private doesn’t appear in search results for users who don’t have access. The agent’s ATRClaims govern which Onyx sources it can query.
Citation-backed results — every Onyx result includes the source document, the specific passage, and a confidence score. The agent knows not just what was found but where and why.
Onyx vs the OKF Vault
The distinction is important: the OKF vault is for ClawQL-generated knowledge entries — things agents learned, decisions they made, patterns they discovered. Onyx is for existing organizational knowledge — documentation, discussions, tickets, communications that predate ClawQL’s involvement.
An agent working on the authentication refactor queries both:
- OKF vault: “what authentication decisions did we make in previous sessions?” → typed decision entries
- Onyx: “what does our security documentation say about token handling?” → Confluence pages, Slack discussions, GitHub issues
The two sources complement each other. The OKF vault captures agent-generated knowledge. Onyx surfaces pre-existing organizational knowledge. Together they give the agent access to everything the organization knows about the problem it’s working on.
The knowledge_search Tool
The knowledge_search MCP tool queries across Onyx’s connector catalog:
knowledge_search("JWT token expiry best practices", {
sources: ["confluence", "slack", "github"],
date_from: "2025-01-01",
max_results: 10,
permission_level: "agent_atrclaims" // respects ATRClaims authorization
})
// Returns:
[
{
source: "confluence",
title: "Security Standards: Authentication Token Policy",
excerpt: "Access tokens should expire within 15 minutes for API endpoints...",
url: "https://company.atlassian.net/wiki/...",
confidence: 0.91,
last_updated: "2026-03-15"
},
{
source: "slack",
channel: "#security",
excerpt: "After the JWT timeout issue last month, we decided 15 min TTL...",
timestamp: "2026-02-18T10:23:00Z",
confidence: 0.87
},
...
]The Onyx path can also participate in x402 micropayment gating when organizations expose knowledge search as a monetizable service.
How the Five Layers Compose
The full memory_recall pipeline queries active layers and merges results:
const memoryRecall = async (query: string, context: RecallContext): Promise<MemoryRecallResult> => {
// Parallel queries across layers
const [indexSurvey, vectorResults, pageIndexResults, codeGraphResults, onyxResults] =
await Promise.all([
vault.surveyIndex(query),
vectorRecall.search(query, { topK: 10 }),
pageIndex.search(query, { topK: 10 }),
isCodeQuery(query) ? codeGraph.search(query) : Promise.resolve([]),
onyx.search(query, context.atrclaims),
]);
// Load full content for high-confidence candidates
const vaultEntries = await vault.loadEntries([
...vectorResults.filter((r) => r.score > 0.75),
...pageIndexResults.filter((r) => r.score > 0.7),
]);
// Rerank all results together
const reranked = await reranker.rank(query, [
...vaultEntries,
...codeGraphResults,
...onyxResults,
]);
return {
entries: reranked.slice(0, context.maxResults ?? 10),
index_summary: indexSurvey,
token_cost: estimateTokens(reranked),
};
};The agent receives a unified ranked list of memory entries regardless of which layer produced them. The source layer is recorded in each entry’s metadata for attribution, but the agent doesn’t need to know whether an answer came from the OKF vault, CodeGraph, or Onyx — it receives relevant, cited, confidence-scored knowledge.
The Team Memory Architecture
Individual developer memory is valuable. Team memory is transformative. When every developer’s agent shares knowledge through the Virtual Gateway, the swarm collectively knows more than any individual agent.
The Vault Namespace Model
s3://clawql-org-vault/
personal/
{developer_id}/
vault/ # Private to the developer — other agents cannot access
decisions/
context/
errors/
team/
{team_id}/
vault/ # Shared across the team — all team agents can read
decisions/
patterns/
runbooks/
entity_instances/
org/
vault/ # Shared across the organization — all agents can read
standards/
architecture/
domain/When memory_recall fires on an edge gateway, it queries:
- The developer’s personal namespace (full access)
- The teams the developer belongs to (read access, ATRClaims scoped)
- The organization namespace (read access, public organizational knowledge)
When memory_ingest fires, the agent specifies the namespace based on the knowledge type: personal decisions stay personal, architectural decisions shared with the team, organizational standards go to the org namespace.
Cross-Vault Recall in the Swarm
In the Agentic Fabric, the Virtual Gateway’s NATS JetStream can propagate memory_recall requests across the swarm when an individual edge gateway’s local cache misses — team vault answers without every edge node needing the full team vault locally. Same collaborative instinct as agent coordination: pool recall when the local cache misses.
Memory and the Token Efficiency Stack
The memory stack connects to all 12 layers of ClawQL’s token efficiency architecture:
Layer 1 (Context bloat): The cg_find_symbol, cg_find_callers, and cg_explain_module tools are search + execute operations — the agent calls typed tools rather than loading full files. CodeGraph is what makes that pattern maximally effective for development tasks.
Layer 2 (Verbose responses): memory_recall results are trimmed to the specific fields the agent depends on. A recall result includes title, excerpt, confidence, and related links — not the full entry unless the agent explicitly requests it.
Layer 3 (Redundant static context): The vault index.md is relatively stable across sessions — it changes only when new entries are added. The system prefix that includes the index survey is cacheable at Anthropic’s provider layer.
Layer 5 (Semantic cache): Recall queries are themselves cached. The same recall query in a different session returns the cached result. The embedding similarity threshold is configurable — 0.92 default means “nearly identical query” returns cached recall.
Layer 6 (Undistilled history): When a session’s conversation history grows long, ClawQL distills it into a type: context entry in the vault. The full transcript is gone. The semantic content is preserved.
Layer 7 (Cross-session rediscovery): The next session starts by recalling that context entry rather than rediscovering project state from scratch — memory replaces rediscovery.
Layer 12 (Flywheel): Every memory_ingest event is a potential training signal. The agent that writes a decision entry with verdict: passed is generating fine-tuning data for the domain-specific recall model. After enough cycles, the Frugal-tier model becomes better at predicting which vault entries are relevant for which queries — embedding recall gets cheaper because the model needs fewer candidates to find the right answer.
The WORM Audit Trail for Memory
Every memory operation — ingest, recall, update, deletion — writes a WORM entry. This makes the memory stack not just persistent but auditable.
// WORM events for memory operations
'MEMORY_INGEST'; // New knowledge entry created
'MEMORY_RECALL'; // Query executed + entries returned
'MEMORY_CACHE_HIT'; // Recall returned cached result
'MEMORY_UPDATE'; // Existing entry modified
'MEMORY_ARCHIVE'; // Entry moved to cold storage
'MEMORY_SYNC_PUSH'; // Entry synced to R2
'MEMORY_SYNC_PULL'; // Entry pulled from R2 to edge
'CODEGRAPH_INDEX_UPDATE'; // CodeGraph index updated
'ONYX_SOURCE_INDEXED'; // Onyx connector indexed new contentThe WORM trail for memory enables compliance queries that no other memory system supports:
“Show me every decision entry created by agent X in the last 90 days” — WORM query on MEMORY_INGEST events filtered by agent_id and type: decision.
“Show me every time the team’s authentication decision was recalled” — WORM query on MEMORY_RECALL events filtered by the entry’s correlation_id.
“Show me what information agent X had access to when it made decision Y” — WORM chain from the decision’s WORM entry back through the recall events that preceded it.
“Prove that the HIPAA-sensitive patient data was not accessed by the agent during this session” — WORM query on MEMORY_RECALL events for the session, filtered by pii_classification: PHI. If no PHI entries appear in recall events, the agent didn’t access them.
This last query — the negative proof — is what makes the memory stack compliant in regulated industries. You can prove not just what the agent remembered, but what it didn’t access.
What’s Missing From Every Alternative
The ecosystem has produced several agent memory approaches. Understanding why each falls short clarifies what a five-layer stack actually provides.
Plain Obsidian vault (no agent integration): The knowledge structure is right. The agent access is manual — the developer writes entries by hand, the agent reads files directly. No memory_ingest automation. No vector recall. No WORM audit. No team sync. No CodeGraph integration. A good starting point, not a production system.
MemoryGraph and similar graph-backed MCP servers: Graph-based storage for relational memory is conceptually sound. What’s missing for enterprise: no OKF format compatibility, no CodeGraph integration, no Onyx connector for enterprise sources, no WORM audit trail, no team sharing, no hot/cold tiering for large vaults, no Flywheel export. Single-developer tools without a path to team scale.
Claude’s built-in memory (Projects): Session context persistence within Claude’s platform. What’s missing: not portable, not accessible from other tools, not auditable, not yours. If you want to use Claude Code alongside Cursor alongside a custom agent, Claude’s Projects don’t help the other tools. And the memories belong to Anthropic’s platform, not your infrastructure.
RAG over documentation: Many teams implement naive RAG — chunk documents, embed them, retrieve by similarity. What’s missing: no structured knowledge format (raw text chunks, not typed entries), no CodeGraph for codebase structure, no cross-session capture (new knowledge never enters the index), no WORM audit, no team sharing. A retrieval system for static documentation, not a memory system for dynamic agent knowledge.
Vector databases (Pinecone, Weaviate, Qdrant) directly: Technically capable of storing and retrieving embeddings. What’s missing: the structured knowledge format, the OKF catalog for token-efficient index survey, the PageIndex fallback for embedding-free retrieval, the CodeGraph integration, the Onyx connector for enterprise sources, the WORM audit trail, the provenance fields. Vector databases are storage infrastructure. They’re Layer 2 of the five-layer stack, not the stack itself.
Getting Started
The memory stack is incremental. Each layer adds value independently. You don’t need all five to see benefits.
Day 1: OKF Vault + Basic Recall
# Initialize the vault
clawql memory init
# Configure your storage backend
export CLAWQL_MEMORY_BACKEND=r2
export CLAWQL_R2_BUCKET=your-org-vault
export CLAWQL_R2_ACCESS_KEY_ID=...
export CLAWQL_R2_SECRET_ACCESS_KEY=...
# Start the gateway with memory tools
clawql inference serve --port 8080
# Connect your IDE (Cursor/Claude Code MCP):
# URL: http://localhost:8080/mcp
# Ask the agent to remember something:
# "Remember that we chose JWT over sessions for auth — because stateless, horizontal scaling"
# Agent calls: memory_ingest({ type: "decision", title: "...", content: "..." })
# Next session:
# "What auth decisions have we made?"
# Agent calls: memory_recall("authentication decisions")Week 1: Add CodeGraph
# Install CodeGraph
npm install -g codegraph
# Initialize in your project
cd your-project
codegraph init
# CodeGraph now auto-syncs on file changes
# MCP tools: cg_find_symbol, cg_find_callers, cg_explain_module, cg_find_route, cg_dependency_pathWeek 2: Add Vector Recall
# Configure embedding model
export CLAWQL_EMBEDDING_MODEL=text-embedding-3-small
export OPENAI_API_KEY=sk-...
# Or use local embeddings (no API key)
export CLAWQL_EMBEDDING_MODEL=nomic-embed-text
export OLLAMA_BASE_URL=http://localhost:11434
# Enable semantic recall
export CLAWQL_MEMORY_VECTOR_RECALL=1
export CLAWQL_MEMORY_CACHE_THRESHOLD=0.92Month 1: Add Onyx for Enterprise Sources
# Deploy Onyx (Docker or Helm); configure connectors
export CLAWQL_ONYX_URL=http://onyx:3000
export CLAWQL_ONYX_API_KEY=...
# knowledge_search now queries across configured enterprise sources
# Permission-aware — respects source system authorizationMonth 2: Add Team Vault Sync
clawql sync init --provider r2 --bucket org-vault
clawql sync pull # pulls team/ and org/ namespaces
clawql sync push # pushes personal/ namespace entries to R2The Memory Stack as a Competitive Moat
The memory stack creates competitive advantage through accumulation. The longer it runs in an organization, the more the vault knows. The richer the vault, the better every subsequent agent session. The better each session, the more high-quality training data enters the Flywheel. The Flywheel produces a fine-tuned model that’s better at recall for this organization’s specific knowledge domain.
This is a flywheel that competitors cannot replicate even if they copy the architecture. Your knowledge entries contain your organization’s decisions, your codebase’s structure (through CodeGraph change entries), your domain’s patterns, your team’s accumulated wisdom. None of that is transferable. A competitor who builds an identical memory stack starting today is starting from zero. Your stack has months or years of compounded organizational knowledge.
The WORM audit trail adds the compliance dimension: not just “we remember” but “we can prove what we remembered, when, by whom, verified against what standard.” For regulated industries — healthcare, finance, legal — this proof is the difference between an AI system that can be deployed in production and one that cannot.
Agents have had amnesia since the beginning. They’ve re-discovered codebases, re-reconstructed decisions, re-explained context, re-benchmarked approaches that were already tried. The five-layer memory stack — OKF vault, vector recall, PageIndex, CodeGraph, Onyx — makes this amnesia optional rather than structural. The knowledge your agents accumulate today is available to every agent tomorrow, and the day after, and the year after. Permanently. Auditably. Yours.
OKF vault serialization and memory_ingest / memory_recall behavior: docs/memory/okf.md · memory-obsidian.md. The vault ties to the Enterprise Ontology — entity instances are knowledge entries with type: entity_instance / ontology types. Ontology architecture: docs.clawql.com/architecture/enterprise-ontology.