When a language model starts to feel more like an assistant than a glorified chatbot, the difference usually lives in how it remembers. In early prototypes, developers tend to shove everything—state, tool descriptions, user preferences—into a single prompt that fits inside the model's context window. That approach works for demos, but it quickly hits a ceiling as interactions grow longer and tasks become more complex. To create agents that can operate reliably in production, we need a memory system that mirrors the way human cognition is organized: multiple layers, each with a distinct purpose and lifecycle.

---

The Four‑Layer Agentic Memory Model

Layer	What it stores	Analogy	Typical implementation
Working Memory	Current turn‑by‑turn context	RAM	The hot prompt that is sent to the LLM for every inference
Semantic Memory	General world knowledge, facts, conventions	Long‑term encyclopedia	Vector database or knowledge‑base that can be queried with embeddings
Procedural Memory	How‑to knowledge, tool signatures, APIs	Muscle memory	Index of skill identifiers that lazily load full implementations when needed
Episodic Memory	Summaries of past interactions, patterns, lessons learned	Personal diary	Compressed replay buffers or retrieval‑augmented stores with forgetting policies

1. Working Memory – The Fast Buffer

The simplest layer is the working memory, which is simply the prompt that the model sees at inference time. It is fast, transient, and must be kept under the model’s token limit (often 4k–16k tokens depending on the provider). Anything that isn’t required for the immediate reasoning step should be pushed out of this buffer.

Best practice: limit the prompt to the last few user messages plus a short system instruction. Use token‑count utilities to guarantee you stay within limits, and truncate older turns automatically.

2. Semantic Memory – The Knowledge Backbone

Semantic memory holds the “rules of the world.” This is where you store embeddings of documents, API specifications, or domain‑specific ontologies. When the agent needs factual grounding, it performs a similarity search against this vector store and injects the most relevant snippets back into the working memory.

Implementation tip: index documents with chunk sizes of 300–500 tokens and store both the raw text and metadata (source, version). Use hierarchical retrieval (e.g., first a coarse‑grained filter, then a fine‑grained rerank) to keep the number of injected tokens low.

3. Procedural Memory – The Skill Catalog

Procedural memory is a catalogue of capabilities the agent can invoke: code execution, web scraping, calendar updates, etc. Instead of describing each tool in the prompt, maintain a lightweight manifest that maps skill names to function signatures. When the reasoning chain decides a particular skill is needed, the full implementation (or a just‑in‑time import) is fetched and attached to the prompt.

Pattern to follow:

When the LLM outputs create_github_issue, the orchestrator loads the actual API wrapper and passes the arguments.

4. Episodic Memory – The Experience Archive

The hardest layer to get right is episodic memory. It must capture what actually happened and turn those raw logs into reusable insights. Two challenges dominate this space:

1. Compression: Raw conversation logs can be massive. Summarization models or extractive algorithms should distill each interaction into a concise vector representation (e.g., a 256‑dimensional embedding plus a short textual summary).
2. Forgetting: Not every experience is valuable. You need a policy that ages out noisy or redundant entries while preserving patterns that improve future decisions.

Suggested workflow:

1. After each turn, store the full transcript in a durable store (e.g., a relational DB).
2. Run a summarizer (such as an LLM with a summarize‑conversation prompt) to produce a 2‑sentence gist.
3. Embed the gist and store it in a separate vector index.
4. Apply a scoring function that combines recency, relevance, and usefulness (e.g., reinforcement‑learning‑derived reward) to decide which embeddings stay active.

---

Matching Architecture to Agent Type

Different applications require different subsets of the four layers.

• Reflex Agents – Simple react‑and‑reply bots that only need the current turn. Working memory alone suffices.
• Support Agents – Customer‑service bots that must look up policies and execute actions. Combine working and procedural memory, optionally adding semantic if policy documents are large.
• Coding Agents – Developers’ assistants that need to recall coding conventions, fetch library docs, and compose multi‑step plans. They benefit from the full stack, especially a robust episodic layer to remember past debugging sessions.

---

Practical Example: Building a Code‑Review Assistant

Below is a condensed snippet showing how the layers cooperate in a Node.js‑style orchestrator.

In this flow, each call to vectorDB.search pulls minimal background knowledge, while runLinter is fetched lazily from a procedural catalog. After the conversation ends, storeEpisodic runs a summarizer and updates the forgetting scores.

---

Key Takeaways

• Don’t treat the context window as the sole source of truth. Layered memory lets you keep prompts short while still accessing rich information.
• Separate “what” from “how.” Semantic memory holds facts; procedural memory stores capabilities.
• Episodic memory is about insight, not storage. Focus on summarization and intelligent pruning.
• Choose layers based on the agent’s role. Reflex bots need only working memory; complex assistants require the full stack.
• Implement forgetting logic early. Without it, episodic stores become noisy and degrade performance.

---

Conclusion

Moving beyond a monolithic prompt is essential for scaling LLM agents from sandbox demos to real‑world services. By adopting a four‑layer memory architecture—working, semantic, procedural, and episodic—you give your agent the tools to reason efficiently, access appropriate knowledge, execute actions on demand, and learn from past interactions. The hardest part remains building reliable forgetting policies, but even a simple recency‑weighted score can dramatically improve usefulness. Start by adding the layer that gives you the biggest ROI for your current use case, and iteratively flesh out the rest as the product matures.

---

Source: Moving Beyond the Context Window: The Agentic Memory Architecture