Module 6 — Agent Memory 🧠💾

Goal of this module: Understand how agents remember things — within a task and across sessions. We’ll cover short-term vs. long-term memory, vector databases, embeddings, and RAG (the most important memory pattern). All explained from zero.

6.1 The problem: LLMs are forgetful

Recall from Module 2: an LLM only knows what’s currently in its context window (the desk). The moment something falls off the desk — or the conversation ends — it’s gone. The model has no built-in memory between calls.

This causes real problems:

A long task overflows the context window, and early steps get forgotten.
Tomorrow’s conversation doesn’t remember today’s. (“Who are you again?”)
The agent can’t learn your preferences over time.

Memory is the system we bolt on to fix this — a way to store information outside the context window and bring it back when needed.

Analogy: The context window is what’s on your desk right now. Memory is your filing cabinet and notebook. You can’t fit everything on the desk, so you file things away and pull out the relevant folder when you need it.

6.2 The two kinds of memory

Short-term memory (working memory)

This is just the context window — the current conversation and recent steps. It’s fast and immediate, but limited and temporary. When the task ends or the window fills, it’s gone.

Holds: the current goal, recent thoughts/actions/observations.
Lifespan: this session only.
Analogy: what you’re actively holding in your head right now.

Long-term memory (persistent memory)

This is information stored outside the model (in a database or files) that survives across sessions and can be huge.

Holds: facts, past conversations, user preferences, documents, learned lessons.
Lifespan: forever (until deleted).
Analogy: your filing cabinet, notebook, or photo album.

┌─────────────── SHORT-TERM (context window) ───────────────┐
│  Current goal + recent steps. Fast. Small. Temporary.     │
└────────────────────────────────────────────────────────────┘
              ▲ pull relevant info in        │ save important info out ▼
┌─────────────── LONG-TERM (external store) ────────────────┐
│  Everything else. Slow-ish. Huge. Permanent.              │
└────────────────────────────────────────────────────────────┘

The art of agent memory is moving the right information between these two at the right time.

6.3 Types of long-term memory (a useful breakdown)

Borrowing loosely from how humans remember, agent memory is often split into:

Episodic memory: Records of past events/conversations. “On June 3rd, the user asked about flights to Rome.”
Semantic memory: General facts and knowledge. “The user is vegetarian.” “Our refund policy is 30 days.”
Procedural memory: How to do things — learned skills or routines. “To reset a password, follow these steps.”

You don’t need to over-engineer this as a beginner, but it helps to know that “memory” isn’t one thing — it’s facts, events, and know-how.

6.4 The simplest memory: just save text

The most basic memory is literally saving notes to a file or database and loading them back later.

End of session:  save "User prefers window seats" → memory file
Next session:    load memory file → include "User prefers window seats"
                 in the context window

This works! For small amounts of info (a few facts, user preferences), a simple key-value store or text file is perfectly fine. Don’t over-complicate it when a notebook will do.

The problem comes when memory grows huge — thousands of notes, documents, past chats. You can’t fit them all on the desk. You need to fetch only the relevant bits. That’s where embeddings and vector databases come in.

6.5 The key concept: embeddings (search by meaning)

Imagine you have 10,000 saved notes and the user asks about “refund policy.” You need to find the relevant notes — but they might say “money-back guarantee” or “returns,” not the exact words “refund policy.” Plain keyword search would miss them.

Embeddings solve this. An embedding is a way to turn a piece of text into a list of numbers (a “vector”) that captures its meaning. Texts with similar meaning get similar numbers — even if they use different words.

"refund policy"        → [0.21, -0.08, 0.55, ...]   ┐
"money-back guarantee" → [0.20, -0.07, 0.54, ...]   ┘ very close numbers!
"pizza toppings"       → [-0.61, 0.33, 0.02, ...]     far away

So “refund policy” and “money-back guarantee” end up near each other in number-space, while “pizza toppings” is far away. This lets us search by meaning, not exact words. This is called semantic search.

Plain version: An embedding turns text into coordinates on a giant “map of meaning.” Similar ideas sit close together on the map.

6.6 Vector databases: storage for embeddings

A vector database is a special database built to store embeddings and quickly find the ones closest (most similar in meaning) to a query.

How it works:

1. Take each note/document → turn into an embedding → store in vector DB.
2. When the agent needs info, turn the QUESTION into an embedding too.
3. The vector DB finds the stored embeddings closest to the question.
4. Return those few most-relevant notes.

Popular vector databases you’ll hear about: Pinecone, Weaviate, Chroma, FAISS, Qdrant, Milvus. They all do the same core job: store meaning-vectors and find the nearest matches fast.

Question: "How do I get my money back?"
    │  (turn into embedding)
    ▼
Vector DB searches its "map of meaning"
    │
    ▼
Returns the 3 closest notes:
  - "Refund policy: 30 days..."
  - "Returns are accepted if..."
  - "Money-back guarantee covers..."

6.7 RAG: the most important memory pattern

RAG = Retrieval-Augmented Generation. It’s the single most important memory technique, and the term you’ll hear constantly. Despite the scary name, it’s simple:

RAG = before answering, go fetch (retrieve) relevant info and put it in the context window, so the model answers using real, specific knowledge instead of guessing.

The flow:

1. User asks a question.
2. RETRIEVE: search memory (vector DB) for relevant info.
3. AUGMENT: paste that info into the context window alongside the question.
4. GENERATE: the model answers using that info.

Example:

User: "What's our company's parental leave policy?"

Step 1 (Retrieve): search the HR document store → finds the parental leave section.
Step 2 (Augment): add that text to the prompt:
       "Using this policy text: [...16 weeks paid...], answer the question."
Step 3 (Generate): "Our policy offers 16 weeks of paid parental leave."

Why RAG is everywhere:

It lets agents use private/company data the model was never trained on.
It keeps answers current (just update the documents).
It reduces hallucination — the model answers from real retrieved text, not vibes.
It’s cheaper than retraining the model on your data.

Remember: RAG = Retrieve relevant info → Add it to the prompt → Generate the answer. If you remember one thing from this module, remember RAG.

6.8 Memory management: keeping the desk tidy

For long-running agents, you actively manage what stays in the context window. Common techniques:

Summarization: When the conversation gets long, replace old messages with a short summary. (“Earlier, we established the user wants a budget trip to Italy in July.”) This frees up desk space while keeping the gist.
Sliding window: Keep only the most recent N messages, drop the oldest.
Selective retrieval: Don’t keep everything in context — store it in long-term memory and pull back only what’s relevant for the current step (RAG-style).
Importance filtering: Save only meaningful facts to long-term memory, not every trivial message.

Long conversation → [summarize old parts] + [keep recent parts] + [retrieve relevant facts]
                  → fits comfortably on the desk

6.9 A mental model of the whole memory system

                  ┌──────── The Agent ────────┐
                  │      (LLM + loop)          │
                  └────────────┬──────────────┘
                               │ needs info
            ┌──────────────────┴───────────────────┐
            ▼                                       ▼
   SHORT-TERM MEMORY                       LONG-TERM MEMORY
   (context window)                        (vector DB + files)
   - current task                          - past chats (episodic)
   - recent steps                          - facts/preferences (semantic)
                                           - how-tos (procedural)
            ▲                                       │
            └────── RAG: retrieve & inject ─────────┘

The agent thinks in short-term memory, and uses RAG to pull relevant long-term memories onto the desk exactly when needed.

6.10 Common mistakes with memory

Mistake	Result	Fix
Storing everything forever	Bloated, noisy memory	Save only meaningful info
Retrieving too much	Context overflow, confusion	Retrieve just the top few relevant items
Keyword-only search	Misses differently-worded matches	Use embeddings / semantic search
Never summarizing	Long chats overflow the window	Summarize old history
Trusting retrieved text blindly	Wrong/outdated docs mislead the agent	Keep the knowledge base clean and current
Over-engineering early	Wasted effort	Start with a simple text store; add a vector DB only when you actually have lots of data

6.11 Check yourself ✅

What’s the difference between short-term and long-term memory for an agent?
In one sentence, what does an embedding do?
What do the letters R-A-G stand for, and what are the three steps?
Why does RAG reduce hallucination?

Answers

Short-term = the context window (current task, temporary, small); long-term = external storage (facts, past chats) that persists and can be huge.
It turns text into numbers that capture meaning, so similar ideas have similar numbers (enabling search by meaning).
Retrieval-Augmented Generation: Retrieve relevant info → Augment the prompt with it → Generate the answer.
Because the model answers from real retrieved text instead of guessing from memory, grounding it in actual sources.

6.12 Summary

LLMs forget everything outside the context window; memory systems fix this.
Short-term memory = the context window (temporary). Long-term memory = external storage (persistent): episodic (events), semantic (facts), procedural (how-tos).
Embeddings turn text into meaning-vectors; vector databases store them and find the closest matches (semantic search).
RAG (Retrieve → Augment → Generate) is the key pattern: fetch relevant info and inject it into the prompt before answering. It enables private data, freshness, and less hallucination.
Manage memory with summarization, sliding windows, and selective retrieval. Start simple.

Next up: Module 7 — Reflection & Self-Correction. How does an agent catch and fix its own mistakes? 👉 07_Reflection_And_Self_Correction.md