Agent memory: short-term, long-term and episodic
A model has no memory between calls, so everything an agent "remembers" is something your system stores and puts back into the context. Short-term (working) memory is the current task's history in the context window. Long-term memory stores facts, preferences and knowledge outside the model, in a database or vector store, and retrieves what's relevant. Episodic memory records past runs, what was tried and what worked, so the agent can learn from experience. Store less than you think, retrieve only what's relevant, let users see and delete what's stored, and never let memory become a way to inject instructions.
Why agents need memory
A model forgets everything when a call ends. Within one task, an agent needs to remember what it has done so far. Across tasks, a useful assistant should remember that you prefer short answers, that your company uses PostgreSQL, or that it already tried a fix yesterday and it failed.
None of this is stored in the model. It is stored by your application and added to the context when needed, which makes memory a context engineering problem: what to store, where, and what to retrieve for each step.
Types of memory
The common way to describe agent memory borrows from how people's memory is described.
Working memory (short-term)
The current task: the instructions, the user's request, the tool calls and results so far, the agent's reasoning. It lives in the context window, and it disappears when the task ends unless you save it. The main problem is size: long tasks fill the window, so working memory needs trimming and summarising. See context compression.
Semantic memory (facts and knowledge)
Facts about the user, the organisation or the world that stay true across tasks: "the user's name is Asha", "the team deploys on AWS", "the company's fiscal year starts in April". Stored in a database or vector store, retrieved by relevance or loaded for every task if small. RAG over your documents is a large form of semantic memory.
Episodic memory (experiences)
Records of past tasks: what the goal was, what the agent tried, what worked, what failed. Useful for agents that repeat similar tasks, like a support agent recalling how a similar issue was resolved, or a coding agent recalling that a certain approach broke the build last time.
Procedural memory (how to do things)
Learned instructions and skills: the system prompt, saved workflows, tool usage notes, or reusable "skills" the agent can load. Some systems let the agent update its own instructions from experience, which is powerful and needs careful review.
Where memory lives
| Memory | Typical storage | How it's used |
|---|---|---|
| Working | Context window, plus a state object | Sent on every step of the task |
| User facts and preferences | Database table or key-value store | Loaded at the start of a session |
| Knowledge | Vector database, search index | Retrieved per question |
| Episodes | Database or vector store of task summaries | Retrieved when a similar task starts |
| Notes and plans | Files the agent reads and writes | Read at the start of each step or session |
Writing to memory
The hard part is deciding what's worth remembering. If you store everything, retrieval returns noise and old, wrong information.
- Extract facts deliberately at the end of a session or task, with a prompt that says what to keep (stable facts, preferences, decisions) and what to ignore (small talk, one-off details).
- Store facts in a structured form where possible, with a source and a date.
- Update instead of duplicating: when a fact changes, replace the old version.
- Expire or review old memories.
Reading from memory
- Load small, always-relevant memory (user preferences, key facts) at the start.
- Retrieve larger memory by relevance to the current task, like RAG.
- Label memory clearly in the context ("Known facts about this user:"), separate from instructions.
- Prefer recent and frequently confirmed memories when they conflict.
Reflective memory
Some agents add a reflection step: after a task, the agent reviews what happened, writes a short lesson ("the payments API rejects dates in this format, convert first"), and stores it. Next time, relevant lessons are retrieved and the agent avoids the same mistake. This is a simple way for an agent to improve over time without retraining the model, and the lessons need review, because a wrong lesson repeats a mistake instead of preventing it.
Risks
- Privacy: memory stores personal information. Tell users what's remembered, let them see and delete it, and follow your data rules.
- Wrong memories: a misunderstood fact, stored once, affects every future task.
- Memory poisoning: if content from users, documents or web pages can be written into memory, an attacker can plant instructions that the agent will read later. Store facts, not instructions, and treat retrieved memory as data.
- Cost and context: every memory loaded into context costs tokens on every call.
From the floor
In the PILOT session where I worked through building a stateful AI agent framework with reflective memory, I started with research: what the topic covers and what I'd need to learn to build it. The research mentioned working memory and the context window, semantic memory, skill libraries, and a reflective cycle where the agent reflects on what it generates so it improves over time, and it showed that conversational memory is a crowded space while areas like procedural memory consolidation were less explored. That is how I approach a topic like this: research the problem first, then decide which parts are worth building.