Memory (AI Agents)

LLMs are stateless — they only see what's in the current context window and forget everything between calls. For an agent to handle long tasks, remember past interactions, and personalise, it needs memory. Agent memory adds a persistent brain around the model, so the agent stays coherent across many steps and sessions.

💡 In one line: Memory lets an agent store and recall information beyond the context window — short-term for the current task, long-term across sessions.

Why Agents Need Memory

An LLM's context window is limited and resets every call. Without memory, an agent can't remember what it did three steps ago, or who you are from yesterday. Memory provides continuity — the thread that makes multi-step, multi-session work possible.

Short-Term vs. Long-Term Memory

  • Short-term (working) memory — the current context window: recent turns plus a scratchpad of the ongoing task. Fast, but limited and ephemeral.
  • Long-term memory — a persistent store (usually a vector database) of facts and past interactions, retrieved by similarity when relevant.
Whiteboard
Whiteboard diagram


Types of Long-Term Memory

A useful breakdown:

  • Episodic — past events and interactions.
  • Semantic — facts and knowledge.
  • Procedural — how to perform tasks.

How Long-Term Memory Works

Long-term memory is essentially RAG applied to the agent's own history: store memories as embeddings, then retrieve the relevant ones and inject them into context. 

Whiteboard
Whiteboard diagram


Memory Operations

  • Write — store a new memory.
  • Read — retrieve relevant memories.
  • Update — revise existing ones.
  • Forget / prune — drop stale or irrelevant memories.

Managing the Context Window

Context is scarce, so agents:

  • Summarise old turns to compress history.
  • Use a sliding window of recent messages.
  • Retrieve only relevant long-term memories, not everything.
  • Reflect — periodically consolidate key facts into long-term memory.

Example

A personal assistant remembers your preferences across sessions; a coding agent recalls the project structure; a long-running research agent tracks its findings over many steps.

Challenges

  • What to remember — relevance and importance.
  • Retrieval quality — surfacing the right memory at the right time.
  • Staleness, cost, and privacy.

Best Practices

  • Separate short- and long-term memory.
  • Summarise and consolidate regularly.
  • Retrieve only relevant memories into context.
  • Prune stale ones and scope per user for privacy.

Summary

  • Memory gives a stateless LLM continuity across steps and sessions.
  • Short-term = the context window; long-term = a persistent vector store.
  • Long-term memory works like RAG: embed → store → retrieve → inject.
  • Manage context with summarisation, windows, and selective retrieval.
  • Mind relevance, staleness, cost, and privacy. EOF echo created