Memory architectures: semantic, procedural, and the conflation failures

Two kinds of memory an agent must get right

Every agent that deals with knowledge has two distinct memory problems that look similar but fail in completely different ways. The first is semantic memory — what the agent knows: facts, world knowledge, document content. The second is procedural memory — how the agent behaves: rules, skills, output formats, escalation policies. The distinction comes from cognitive science: Tulving's 1972 work on human memory separated semantic memory (context-free factual knowledge) from other memory types; Squire's subsequent research on declarative versus procedural memory extended this into the domain of skills and rule-governed behaviour. For AI agents the same split applies, and the engineering consequences are severe when the two types are conflated.

Semantic memory belongs in a retrieval corpus. Facts change — prices shift, products are discontinued, policies are updated. If semantic knowledge lives in the model's weights or is hard-coded in a system prompt, it cannot be updated without retraining or redeploying the agent. The standard remedy is Retrieval-Augmented Generation (RAG), formalised by Lewis et al. (2020): at query time, a retriever fetches relevant passages from an external, updateable corpus; the model generates its answer conditioned on those passages rather than recalling facts from weights. Knowledge lives in the corpus. Reasoning lives in the model. Updates to facts do not require touching the model at all.

Procedural memory belongs in code or structured configuration. Business rules, decision trees, output-format constraints, and escalation logic govern how the agent acts, not what it knows. These belong where authority and priority are explicit — not in a database where they are retrieved by similarity.

The two conflation failure modes

Conflating semantic and procedural memory produces two distinct failure modes that are worth naming precisely because they are easy to introduce and hard to detect.

Authority Vacuum is the failure that occurs when business rules (procedural memory) are stored in a vector database. Vector databases resolve queries by similarity — they return whichever stored text is most similar to the query embedding. But rules have authority and priority, not just content. When two rules conflict — "always escalate billing disputes" versus "resolve billing disputes autonomously under $100" — there is no similarity score that can adjudicate which rule wins. The retriever returns whichever rule is closest in embedding space to the current query, which may be the lower-priority rule. The business logic has no authority structure; it has a search result. The fix is to keep rules in code, in a decision table, or in an explicitly ranked configuration — anywhere that priority can be stated and enforced without a similarity lookup.

Silent Staleness is the failure that occurs when facts (semantic memory) are hard-coded into a system prompt. Hard-coded facts go stale invisibly: when the underlying reality changes, nothing in the agent loop signals the discrepancy. The model continues to answer confidently with outdated information because the prompt is static and the model has no mechanism to doubt it. A RAG architecture fixes this directly: the system prompt no longer holds facts, only retrieval instructions. Updated facts propagate to the agent the moment the corpus is updated, with no prompt or model change required.

RAG as the standard semantic-memory mechanism

The canonical RAG setup from Lewis et al. 2020 has three components: a document store, a retriever, and a reader (the language model). At query time, the retriever scores every document against the query and returns the top-k most relevant passages. The reader generates an answer conditioned on those retrieved passages plus the original query. The model never needs to memorise the facts — it reads them at inference time from a corpus that can be updated independently.

In practice, retrieval quality is the bottleneck. Anthropic's contextual retrieval work shows that prepending a short context description to each document chunk before indexing substantially improves recall — retrieved chunks carry enough surrounding context to be useful to the reader, rather than being semantically orphaned fragments. The retriever does not need to be a dense-vector model. For smaller corpora, a keyword or BM25-style retriever achieves the same architectural role with zero external dependencies, which is exactly what the reference implementation below demonstrates.

For production deployments that require semantic (dense) retrieval, a vector database like Chroma handles the embedding storage and approximate nearest-neighbour search. But the architectural principle is the same regardless of retriever type: facts live in the corpus, not in the model or the prompt.

Persistent memory across runs

Retrieval solves in-context semantic memory for a single query. But agents often need facts to persist across sessions — a user preference noted in one run should not have to be re-retrieved in the next. The standard approach is a durable key-value store: the agent writes discovered or confirmed facts to a JSON file (or a lightweight database) at the end of each run, and loads them at the start of the next. This is still semantic memory — the facts are about the world — but with a write path. The classification matters: because these are facts (not rules), they belong in the same retrievable tier as the corpus, not baked into the system prompt and not stored where authority ordering is required.

The reference implementation uses a single memory.json file. On the first run, the agent retrieves documents, writes the top result's metadata to memory, and persists an explicit demo fact. On every subsequent run it loads those keys at init time, proving cross-run recall without any network call, external dependency, or model invocation. The retrieval arithmetic and the JSON read/write are both deterministic — there is no judgment call in either operation, and neither should ever be delegated to the model.