AI Agent Memory: Short-Term vs Long-Term Context

AI agent memory is the mechanism by which an agent retains information across a task or across sessions, since the underlying model itself is stateless — it has no memory of anything that isn’t included in the current prompt. Everything an agent “remembers” has to be re-fed to the model somehow, and the two dominant strategies for doing that are short-term memory and long-term memory.

Short-term memory: the context window

An agent’s short-term memory is whatever fits in its context window — the running conversation history, recent tool outputs, and any instructions included in the current call. It’s fast, requires no extra infrastructure, and is exactly what the model attends to when generating its next response.

The catch is capacity. Context windows are large but finite, and every token counts against both the limit and the bill — see our guide to prompt caching and LLM costs for how providers discount repeated context. A long-running agent that keeps appending every tool call and result to its history will eventually either hit the context limit or start pushing early, possibly important, information out of the window entirely.

Most agent frameworks manage this with some form of trimming: summarizing older turns, dropping verbose tool outputs once they’ve been acted on, or keeping only the last N exchanges verbatim. That summarization is itself a design decision — compress too aggressively and the agent loses details it needed; compress too little and you’re back to running out of room. You can estimate how much headroom a given conversation leaves with a token cost calculator before deciding how much history to keep.

Long-term memory: external storage

Long-term memory moves information outside the context window entirely, into a database the agent can query on demand. The most common implementation embeds text into vectors and stores them in a vector database, so the agent can retrieve semantically relevant memories rather than everything it has ever seen.

This is structurally the same pattern used in retrieval-augmented generation: instead of retrieving from a fixed document corpus, the agent retrieves from a growing store of its own past interactions, facts it’s been told, or user preferences it’s picked up. A customer-support agent might store “this user prefers email over chat” as a long-term memory and retrieve it at the start of every future conversation, without that fact ever occupying space in the current context window until it’s actually needed.

Long-term memory isn’t limited to vector search. Simpler agents get real mileage out of structured stores — a key-value store of user facts, a running summary document the agent rewrites after each session, or even a plain log file the agent can grep. The vector-database approach shines when the number of stored memories is large and the query is fuzzy (“what does this user care about”); simpler stores work better when the lookup is exact (“what’s the user’s account ID”).

Short-term vs long-term memory

	Short-term (context window)	Long-term (external store)
Location	Inside the current prompt	Outside the model, in a database
Capacity	Bounded by the model’s context limit	Effectively unbounded
Retrieval	Automatic — the model attends to everything present	Explicit — the agent must query and re-inject relevant results
Cost	Charged as input tokens on every call	Storage cost plus retrieval tokens, only when queried
Persistence	Gone once the session or window resets	Survives across sessions indefinitely
Failure mode	Truncation or eviction of older context	Retrieval mismatch — the wrong or no memory surfaces

Most production agents use both. Short-term memory handles the immediate task; long-term memory handles facts and preferences that need to outlive a single conversation. The ReAct pattern that many agents follow — reason, act, observe, repeat — typically keeps its reasoning trace in short-term memory while treating any retrieved long-term memories as just another tool result to reason over.

How agents decide what to remember

Deciding what graduates from short-term to long-term memory is one of the harder design problems in agent systems, and there’s no universal answer. Common approaches include:

• Explicit write actions. The agent has a remember(fact) tool it calls deliberately, the same way it would call any other tool — this keeps memory writes auditable but depends on the model reliably choosing to use it.
• Periodic summarization. After every N turns, a separate call summarizes the session and writes the summary to long-term storage, independent of whether the agent “decided” anything was worth keeping.
• Post-hoc extraction. A background process reviews completed sessions and extracts durable facts, decoupling memory writes from the live conversation entirely.

Each approach trades control for reliability. Explicit write actions are the most agent-driven, and also the most likely to miss things the agent didn’t think were important at the time. If you’re prototyping an agent, our build-your-own-AI-agent guide walks through wiring up a basic tool-use loop, which is the natural place to add a first memory tool.

Failure modes worth designing around

Short-term memory fails by truncation — older, sometimes still-relevant context silently falls out of the window, and the model has no signal that anything is missing. Long-term memory fails differently: retrieval can surface an irrelevant memory, miss a relevant one because the query phrasing didn’t match well semantically, or serve a stale fact that’s since changed. Neither failure mode announces itself; the agent just acts on incomplete or wrong information with full apparent confidence.

The takeaway

Short-term memory is free, automatic, and bounded by the context window; long-term memory is unbounded but requires the agent to explicitly store and retrieve it, usually through a vector database or a simpler structured store. Neither replaces the other — durable facts belong in long-term storage so they don’t compete for space with the immediate task, while the immediate task itself is best kept in the context window where the model can attend to it directly. Designing agent memory well is mostly about deciding, deliberately, which information belongs in which tier.