Agent Memory: From Context Window to Persistent Knowledge

The customer success agent at a B2B SaaS company went live in March. By June, support engineers were fielding a peculiar class of complaint: the agent gave solid answers but had no memory of the account. Every conversation started cold. "We discussed the Salesforce integration outage twice before," one customer wrote. "Why does your agent keep asking me to explain it again?" The agent had access to a knowledge base and could call APIs. What it lacked was any mechanism to remember what had already happened with this specific customer.

That is the memory problem in agents, and it is not a model capability problem. It is an architecture problem.

What "memory" means in a running agent

Memory in agents maps loosely to the same taxonomy cognitive scientists use for humans, and for once the analogy is productive rather than misleading.

Working memory is the context window — the full set of tokens the model can attend to right now. Everything in the window is instantly accessible with no retrieval step. This is your fastest tier, your most expensive per-token tier, and the one that disappears completely when the session ends.

Episodic memory is the record of what happened. A user asked a question on Tuesday. The agent called three tools, reached a conclusion. The user came back Thursday and asked a follow-up. Episodic memory stores those interaction logs — timestamped, attributed, searchable — so the agent can reconstruct what happened even weeks later.

Semantic memory is extracted knowledge. Not "on Tuesday, the user mentioned they use Python" — rather, the stable fact: this user uses Python. Semantic stores hold facts, entities, preferences, and their relationships, decoupled from the raw events that generated them.

Procedural memory is the how-to layer: task patterns, tool schemas, behavioral guidelines. This is largely the system prompt, but it can also include cached successful task decompositions and known failure patterns.

Most demo agents have only working memory. Most production agents need all four.

Working memory: the context window as RAM

The context window is the agent's RAM. Right now, mid-2026, frontier APIs offer 128k–200k tokens. That sounds enormous until you count what actually lives there for a typical production agent.

Token budget breakdown for a production agent (128k window)
──────────────────────────────────────────────────────────
System prompt (persona + constraints):      ~1,500 tokens
Tool definitions (10 tools × ~200 tokens):  ~2,000 tokens
Conversation history (last 30 turns):      ~15,000 tokens
Retrieved episodic chunks (top-5):          ~3,000 tokens
Retrieved semantic facts (top-10):          ~2,000 tokens
RAG context (3 document chunks):            ~6,000 tokens
─────────────────────────────────────────
Total consumed:                            ~29,500 tokens  (23%)
Available for output:                      ~98,500 tokens  (77%)

But as conversation history grows:
  100 turns × ~500 tokens/turn = 50,000 tokens
  Now consumed:                            ~64,500 tokens  (50%)
  Available for context and output:        ~63,500 tokens

{ "type": "context-window", "title": "Agent memory competing for context budget", "window": 128000 }

The math shows why long-running sessions run into trouble. A customer support agent handling a complex multi-hour debugging session will exhaust its history budget before the session ends unless you actively manage what stays in context.

Three strategies handle context overflow in working memory:

Sliding window eviction drops the oldest turns when the window fills. Cheap to implement, but the agent forgets earlier parts of its own session, which causes confusing behavior mid-conversation.

Recursive summarization compresses older turns into a running summary before evicting them. The agent loses granularity but preserves the gist. LangChain's ConversationSummaryBufferMemory implements this pattern. Cost: one additional LLM call per summarization cycle, typically ~$0.002–$0.005 for a 5k-token summary at current (mid-2026, illustrative) input rates.

Selective retrieval replaces the always-on history dump with an on-demand retrieval step: embed the current query, search the conversation history, inject only the top-k most relevant prior turns. More accurate than a sliding window for non-linear conversations; adds ~50–200ms latency per turn for the retrieval step.

The context window management article covers these in depth. The point here is that working memory alone cannot support a persistent agent — it needs to write to something durable.

Episodic memory: giving agents temporal recall

Episodic memory is the external log. Every significant event — a user question, a tool invocation, an intermediate conclusion, a final answer — gets written to a persistent store with a timestamp and enough metadata to retrieve it later.

The architecture is conceptually simple:

sequenceDiagram
    participant U as User
    participant A as Agent
    participant ES as Episodic Store
    participant LLM as LLM

    U->>A: "What did we decide about the API key rotation policy?"
    A->>ES: semantic search for "API key rotation policy"
    ES-->>A: 3 matching episodes from past sessions
    A->>LLM: [system + current query + retrieved episodes]
    LLM-->>A: "In our March 15 session, you decided to rotate every 90 days..."
    A-->>U: answer with temporal grounding
    A->>ES: write new episode (this turn's Q&A)

The retrieval step before each LLM call is what gives the agent its temporal awareness. Without it, every session is a fresh start. With it, the agent can answer "what did we decide last Tuesday?" — the query type where pure RAG on a knowledge base consistently fails because there is no document to retrieve, only an interaction to recall.

What goes in an episode record? Minimally:

{
  "episode_id": "ep_20260315_143022_abc",
  "session_id": "session_20260315",
  "user_id": "user_789",
  "timestamp": "2026-03-15T14:30:22Z",
  "turn": 3,
  "user_message": "What rotation period should we use for API keys?",
  "agent_response": "Given your compliance requirements, 90-day rotation with automated alerts at 75 days.",
  "tools_called": ["get_compliance_policy", "get_current_rotation_config"],
  "entities_mentioned": ["API key rotation", "compliance policy"],
  "embedding": [0.023, -0.147, ...]   # 1536-dim vector for semantic search
}

Store these in anything that supports both vector similarity search and timestamp filtering: Postgres with pgvector, Qdrant, or Redis Stack all work. For most teams starting out, Postgres is the right call — you already have it, and pgvector handles ~10M episodes comfortably before you need to think about a dedicated vector DB.

The critical operational concern is the write path latency. If you write episodes synchronously before returning the agent's response, you add ~10–50ms per turn. Write them asynchronously (fire-and-forget) and you risk losing episodes on crashes. The middle path: write synchronously to a fast queue (Redis Streams), return the response, and have a background worker persist to the durable store within seconds.

Semantic memory: from events to facts

Raw episodic records answer "what happened?" Semantic memory answers "what is true?" — and the difference matters enormously for agent reasoning.

Consider an agent that helps a software team. Over three months it has had 200 conversations that collectively establish: this team uses Python 3.12, their CI is GitHub Actions, their database is Postgres, the tech lead is Priya, and they have a strict no-dependencies-without-security-review policy. None of that is in a document. It emerged from conversations. But it should not require re-reading 200 conversation logs to answer "what database does this team use?"

Semantic memory solves this with a consolidation pipeline that runs episodic logs through an extraction step:

flowchart LR
    E[Episodic Store\nraw logs] -->|extraction LLM call| X[Fact Extractor]
    X -->|new or updated fact| M[Merge / Dedupe]
    M -->|contradiction?| R[Resolution LLM call]
    M -->|clean fact| V[Vector Store\nfact index]
    V -->|semantic search| C[Context Injection]
    C --> A[Agent Working Memory]
    style E fill:#0e7490,color:#fff
    style V fill:#a855f7,color:#fff
    style A fill:#00e5ff,color:#0a0a0f

The extraction LLM call looks roughly like this:

FACT_EXTRACTION_PROMPT = """
Given this conversation turn, extract any durable facts about the user, their organization,
their preferences, or their environment. Return a JSON array of facts.

Format: [{"fact": "...", "subject": "...", "confidence": 0.0-1.0, "replaces": "fact_id or null"}]

Only extract facts that would still be true in a week. Do not extract temporary context.

Conversation turn:
{turn_text}
"""

# Output example:
[
  {"fact": "User's team uses Python 3.12", "subject": "tech_stack", "confidence": 0.9, "replaces": null},
  {"fact": "CI system is GitHub Actions", "subject": "infrastructure", "confidence": 0.95, "replaces": null}
]

The merge and contradiction resolution step is where most DIY implementations break. If yesterday's episode says the team uses Python 3.11 and today's says 3.12, you need to detect the contradiction and resolve it (typically: newer fact wins, with confidence weighting). Systems like Mem0 handle this with a dedicated resolution pass; rolling your own requires explicit versioning and a contradiction-detection query at write time.

A rough cost estimate for the consolidation pipeline on a moderately active agent:

Cost of semantic consolidation at scale (illustrative, mid-2026 pricing)
────────────────────────────────────────────────────────────────────────
Assumptions:
  100 active users × 5 sessions/week × 10 turns/session = 5,000 turns/week
  Extract facts from 40% of turns (heuristic: only turns with new information)
  = 2,000 extraction calls/week
  Avg extraction call: 500 input tokens + 150 output tokens
  At $3/M input + $15/M output (illustrative mid-tier model):
  = 2,000 × (500 × $0.000003 + 150 × $0.000015)
  = 2,000 × ($0.0015 + $0.00225)
  = 2,000 × $0.00375
  = $7.50/week ≈ $30/month

Contrast with re-sending full history on every call:
  Turn k re-sends all k−1 prior turns, so a 10-turn session re-sends
  45 × 500 = 22,500 history tokens on top of the new turns.
  500 sessions/week × 22,500 × $0.000003 ≈ $34/week — and that is
  session-local only. Carry history across sessions and per-user cost
  grows quadratically with total turns: a user 200 turns deep pays
  ~100k history tokens ($0.30) on every single call.
  Consolidation reads each turn once, ever — linear in turns.

Procedural memory: the system prompt is memory too

Procedural memory is the most underrated tier. It is the agent's knowledge of how to do things rather than what it knows or what happened. This lives primarily in the system prompt and tool definitions, but it can be far more sophisticated.

A naive system prompt gives the agent a persona. A well-engineered procedural memory gives it:

• Canonical task decomposition patterns for its domain ("when asked to debug a deployment, always check X before Y before Z")
• Known failure patterns and how to handle them ("if the database query returns empty, do not assume success — verify with a count query")
• Tool usage policies ("always call get_policy before execute_action for any write operation")
• User-specific preferences loaded from semantic memory at session start

The injection pattern for dynamically loaded procedural memory looks like this:

async def build_system_prompt(user_id: str, base_prompt: str) -> str:
    # Load user preferences from semantic store
    user_facts = await semantic_store.get_facts(
        subject=user_id,
        categories=["preferences", "constraints", "working_style"],
        top_k=15
    )

    # Load relevant task patterns
    task_patterns = await procedural_store.get_patterns(
        context=current_session_context,
        top_k=5
    )

    facts_block = "\n".join(f"- {f['fact']}" for f in user_facts)
    patterns_block = "\n".join(f"- {p['description']}" for p in task_patterns)

    return f"""{base_prompt}

<user_context>
Known facts about this user:
{facts_block}
</user_context>

<task_patterns>
Relevant patterns for this session:
{patterns_block}
</task_patterns>"""

Tool schemas are procedural memory too — and they consume tokens. Ten tools at ~200 tokens each cost 2,000 tokens on every request. Tools that won't be needed in this session are dead weight. The designing tool schemas article covers trimming this budget; the memory angle is that selectively loading only session-relevant tools (based on user context from semantic memory) can cut tool-definition overhead by 40–60%.

Memory in the agent loop

These tiers do not operate independently. They interact at specific points in the agent's perceive-reason-act loop:

sequenceDiagram
    participant U as User turn
    participant R as Retrieval layer
    participant W as Working memory (context)
    participant LLM as LLM
    participant T as Tools
    participant W2 as Write-back

    U->>R: new user message
    R->>R: 1. semantic search episodic store (top-3 episodes)
    R->>R: 2. semantic search fact store (top-10 facts)
    R->>W: inject episodes + facts into context
    W->>LLM: full context: system + facts + history + message
    LLM->>T: tool call(s)
    T-->>LLM: tool result(s)
    LLM-->>W: final response
    W-->>U: response
    W2->>W2: async: write episode to episodic store
    W2->>W2: async: run fact extraction on new turn
    W2->>W2: async: merge new facts into semantic store

The pre-call retrieval adds latency. On a warm vector DB (Qdrant or pgvector with a cached index), two parallel semantic searches take ~20–80ms. That is the price of memory. For most applications it is acceptable. For sub-100ms latency requirements, you need to preload memory at session start rather than per-turn.

The widget below steps through the bare loop — reason, tool call, observe — with no memory attached. The retrieval and write-back stages in the sequence diagram above wrap around exactly this cycle.

{ "type": "agent-loop", "scenario": "weather-trip", "title": "The base agent loop that memory hooks into" }

Systems that implement this: Mem0, MemGPT, Zep

Three open-source systems have productionized this architecture, each with different tradeoffs.

Mem0 (which grew out of the Embedchain project) focuses on the episodic-to-semantic consolidation pipeline. It maintains a managed memory graph that extracts facts, resolves contradictions, and provides a unified retrieval interface. Mem0's published evaluations show the largest accuracy gains for memory-augmented agents on temporal queries and multi-hop reasoning — the exact cases where raw chunk retrieval degrades. The Mem0 managed API works well for fast prototyping; the open-source version gives you control over the store backends.

MemGPT (now Letta) introduced the operating-system abstraction: the LLM is a CPU with main memory (context window) and disk (external store). The agent can explicitly call read_from_disk and write_to_disk functions, making memory management an explicit part of the agent's reasoning rather than a hidden infrastructure concern. This is elegant conceptually, but adds function-calling overhead to every turn where memory access is needed.

Zep takes a graph-first approach, storing episodic memory as a knowledge graph of entities and relationships rather than flat vector embeddings. Multi-hop queries ("what did the user say about their Postgres setup after we discussed the migration?") run as graph traversals, which can be faster and more accurate than pure vector search on relationship-heavy queries.

The comparison in practical terms:

System	Best for	Key tradeoff
Mem0	General-purpose agents, fast setup	Managed vs self-hosted cost; API adds a network hop
Letta/MemGPT	Research, full transparency of memory ops	Higher token overhead per turn; more agent complexity
Zep	Relationship-heavy domains (CRM, legal)	Graph query expertise required; more infra to operate
DIY (pgvector)	Full control, existing Postgres infra	You build the consolidation pipeline yourself

What breaks

Retrieval quality compounds

Every memory retrieval has an accuracy below 100%. If your episodic retrieval is 85% accurate (a realistic number for a broad query over months of history), and a single agent task requires five relevant facts from memory, your probability of retrieving all five correctly is 0.85^5 ≈ 44%. The agent is operating on a materially incomplete picture of the past nearly half the time. This is the same compounding error problem that destroys multi-step task success rates.

The mitigations are familiar: retrieval with reranking, hybrid sparse+dense search, and chunking episodic records into smaller, more precisely targeted units. They help but do not solve the fundamental probability math. Design your agent to degrade gracefully when memory is incomplete — prompt it to acknowledge uncertainty on temporal claims rather than confabulate.

{ "type": "error-compound", "accuracy": 0.85, "steps": 10, "title": "Compounding memory retrieval error" }

Staleness and contradiction

Semantic stores accumulate outdated facts. A user who was on a team using Python 2.7 is now using Python 3.12. The old fact is still in the store. Naive retrieval returns it alongside the new one, and the agent gets confused.

Explicit versioning with timestamps and a "valid until" field on facts is the right defense. When a new fact contradicts an existing one, mark the old fact as superseded rather than deleting it (you may need the audit trail). Resolution rules need to be explicit: most recent wins for technical facts, explicit user corrections always win, high-confidence automated extractions win over low-confidence ones.

Memory leakage across users

Multi-tenant agents that store memory per user but share a database are a security hazard. A retrieval bug, a missing user ID filter, or a misconfigured index can surface one user's private data in another's session. This is not theoretical — it is the failure mode that causes teams to either skip memory entirely or over-engineer isolation.

Partition episodic and semantic stores by user ID at the database level, not the query level. Row-level security in Postgres, or separate Qdrant collections per user. The query-level filter (WHERE user_id = ?) is not enough: schema mistakes and ORM bugs have a way of dropping those filters.

Deletion requests propagate badly

A "forget me" request sounds simple until you trace where the data actually lives: raw episodes in the episodic store, facts extracted from those episodes in the semantic store, and embeddings of both. Three stores, three deletion paths. Deleting the episode does not delete the fact it produced — "user's team uses Postgres" survives the erasure of the conversation that established it, which is exactly what a GDPR-style deletion request says must not happen.

The defense is lineage: every semantic fact records the episode IDs it was derived from, so a deletion cascades from episodes to facts to their vectors. Add TTLs on low-confidence facts so weak extractions age out instead of persisting forever. And notice the tension with the staleness advice above — "mark superseded, don't delete" is right for audit trails, but a deletion request overrides it: superseded facts derived from deleted episodes have to go too. Build the cascade before you need it; retrofitting lineage onto a store with six months of unattributed facts means reconstructing provenance you never recorded.

Write amplification on high-volume agents

An agent handling 10,000 conversations per day generates 10,000 episodic records minimum. If each record triggers a fact extraction call (one LLM call per turn, even a cheap one), that is 10,000 additional LLM calls per day — a background cost that quietly rivals the main agent's API spend. Batch and gate the consolidation pipeline: extract facts only from turns that meet a relevance heuristic (new entities mentioned, user corrections issued, significant decisions made), and run consolidation in nightly batches rather than synchronously.

Context poisoning from bad memories

If the fact extraction step makes errors — and it will, particularly on ambiguous or ironic statements — it pollutes the semantic store with false facts that then get injected into future sessions as ground truth. "User prefers TypeScript" extracted from "I hate TypeScript but my team forces it" is the kind of extraction failure that leads to subtly wrong agent behavior that is hard to debug.

Two defenses: confidence thresholds (only persist facts above 0.8 confidence), and explicit user-confirmation flows for high-stakes facts. Some agent systems surface extracted facts back to the user: "I've noted that your team uses Python 3.12 — does that look right?" It adds a turn but catches extraction errors before they compound.

The decision in practice

Start with a clear question: what does your agent need to remember, and for how long?

If your agent only needs to maintain coherence within a single session, working memory management — a rolling window plus selective retrieval from earlier turns — is sufficient. Add a ConversationSummaryBuffer and call it done. This handles the majority of chatbot and assistant use cases.

If your agent needs cross-session continuity for individual users (remember their preferences, recall past decisions, track ongoing projects), add episodic memory. Start with Postgres + pgvector for the store, write episodes asynchronously, and retrieve the top-5 most recent and top-5 most semantically similar episodes at session start. Test whether this alone solves the continuity problem before building a full consolidation pipeline.

Add semantic memory when you observe that episodic retrieval is returning noisy, voluminous results for queries that should be answerable with a simple fact. The signal is agents giving long, hedged answers when a direct answer exists — they are wading through episodic logs to find something that should be a simple fact lookup. At that point, the consolidation pipeline earns its cost.

Procedural memory optimization is available at any stage: audit your system prompt and tool definitions for dead weight, and build the injection mechanism to load user-specific preferences from semantic memory into the system prompt at session start.

The broader agent memory architectures article covers the episodic/semantic/procedural taxonomy in finer detail, including knowledge graph alternatives to vector stores and the specific mechanics of MemGPT's paging model. The agentic RAG article covers the RAG-as-memory pattern, where retrieval itself becomes part of the agent's reasoning loop rather than a pre-call injection step.

One last thing worth saying plainly: the field does not have this solved. Consolidation pipelines that accurately extract facts without hallucinating them, contradiction resolution that handles nuanced updates, memory systems that remain accurate across months without drift — these are active engineering challenges, not solved problems with off-the-shelf answers. Mem0, Zep, and Letta give you good starting points; plan to build application-specific validation on top of whatever you choose.