All thoughts

How AI Agents Remember

Why your AI tool seems to know you, and the small gate that keeps its memory short.

How it works

One chat turn, and how it becomes memory

Six blocks joined by arrows. Only the gate is gold.

  1. You ask (a new message): Your new message starts a turn, and the model remembers nothing from earlier calls unless the app puts it back in front of it.
  2. Context (working memory for this turn): Working memory is rebuilt for this turn: the instructions, the chat so far and only the closest few matches pulled from long-term stores.
  3. Reply (the model answers): The model answers from what it has in view right now.
  4. Episodic log (every chat, dated): The exchange is saved in the log in the order it happened, and the full record stays searchable.
  5. Gate (waits for N chats): A set amount of activity, a schedule or a manual call opens the gate, and a summarizer turns the log into a few lasting facts. This is the key step.
  6. Facts (read again next chat): Those facts go back into the context at the start of the next chat, which closes the loop.
Every chat lands in the log. The gate is gold because it decides when a summarizer turns the log into a few lasting facts. Those facts go back into the context at the start of the next chat, which closes the loop.

Language models are stateless. Nothing carries over between calls unless the app puts it back in front of the model.

Working memory

Working memory is what the model has in view right now: the instructions it runs under, the conversation so far and anything pulled in for this step. It's a scratchpad that gets rebuilt every turn.

A long chat outgrows that space fast. So the agent leans on long-term stores that keep living after the conversation closes.

The three pillars

  • Procedural: the rules for doing the job. The agent's instructions, prompts and code.
  • Semantic: facts. What's true about you, your work and your preferences.
  • Episodic: experiences. Past conversations and actions, kept in the order they happened.

Researchers borrowed these three labels from cognitive science and mapped them onto language agents. I find them a useful filing system.

Fetch only what you need

Long-term stores get big, so the agent never loads them whole. It searches them by meaning (text turned into embeddings, which are lists of numbers) and pulls only the closest handful of matches into working memory. That's retrieval, and the "top-K" setting decides how many matches come back.

Bigger context windows don't fix this on their own. A crowded context gets more expensive, slower, and easier for the model to get distracted by.

The consolidation gate

The episodic record grows with every exchange. Older messages can be pushed out of view and condensed into a running summary, while the full record stays searchable.

Turning that record into lasting facts is a separate job, and timing is the hard part. You can write memories during the conversation, which adds delay to every reply. Or you can run it in the background on a trigger: a set amount of activity, a schedule, or a manual call. Run it too often and you redo work. Run it too rarely and the facts go stale. That trigger is the gate.

My take

This is the same model I use in my Agent OS. In Part 1, memory is working memory plus three pillars: procedural, semantic and episodic, with a consolidation gate between the log and the facts.

The gate is the part I'd underline. Saving everything is easy. Deciding when to compress, and what deserves to become a fact, is where the token bill and the answer quality get decided.

I also like that the three pillars force a label on every piece of memory. A rule goes in the instructions. A fact goes in the semantic store. An event goes in the log with a date. If something doesn't fit, it probably shouldn't be saved.

One idea I want to try: let the summarizer write facts with a source date, so an old fact can be flagged when the log says it changed.

Try this

  1. Open the memory page in the AI tool you use most. Sort each saved line into procedural, semantic or episodic, and delete what's out of date.
  2. If you build agents, pick a trigger for your gate (a count of conversations or a nightly run) and have a cheaper model write the summary, instead of pasting the whole history into every turn.

Sources