Agent memory and context engineering

A language model has no memory between calls. Everything an agent “remembers” while working has to be placed into its input each time the model runs. On a short task that is easy. On a task with dozens of tool calls, file reads and search results, deciding what the model sees becomes one of the most important design problems. That discipline is often called context engineering.

The context window is working memory

The context window holds the system prompt, tool definitions, conversation history, tool results and anything retrieved. Think of it as working memory: whatever is in there can shape the next step; whatever is not, the model cannot use (beyond what it learned during training).

Windows have grown enormously, but three limits remain:

• Capacity. Long-running agents still fill them.
• Cost and latency. Every token is processed on every call.
• Attention quality. Models do not use all tokens equally well.

That last point is easy to underestimate. The 2023 paper Lost in the Middle (Liu et al.) found that on retrieval-style tasks, models used information best when it appeared near the start or end of the input and noticeably worse when it was buried in the middle, even for long-context models. Newer models have improved, but the general lesson has aged well: more context is not automatically better, and irrelevant material can distract from what matters.

Treat context as a finite budget. Aim for the smallest set of high-signal tokens that makes a good result likely.

Curating what goes in

Good context engineering starts before any clever memory system:

• Clear, compact instructions. Specific enough to guide behaviour, not a wall of edge-case rules.
• Focused tools. A small set of well-described tools with little overlap is easier to use than a sprawling catalogue.
• A few representative examples rather than every possible case.
• Just-in-time loading. Keep lightweight references (file paths, URLs, query strings) and let the agent fetch full content only when needed, instead of pasting everything up front.

Short-term and long-term memory

It helps to separate two kinds of memory:

• Short-term memory is the current context: what the agent is actively working with during this task.
• Long-term memory lives outside the model, in files, databases or search indexes. It only matters once something retrieves it back into context.

Most memory techniques are ways of moving information between these two tiers. The MemGPT paper (Packer et al., 2023) made the analogy explicit, treating the context window like an operating system’s RAM and letting the model page information in and out of larger external storage through function calls.

Summarisation and compaction

When a conversation nears the limit, an agent can compact it: summarise the history so far, then continue in a fresh context that starts with the summary. A good summary keeps goals, decisions, unresolved problems and key facts, and drops things like raw tool output that has already been acted on.

A lighter variant is simply clearing old tool results, for instance a huge directory listing the agent has already processed.

The honest caveat: summarisation is lossy. A detail that looked irrelevant at summary time can turn out to be crucial later, and once dropped it is gone. Aggressive compaction is a common source of agents “forgetting” constraints the user stated early on. Mitigations include tuning the summarisation prompt on real transcripts and saving important details somewhere retrievable.

Retrieval-based memory

Retrieval lets an agent store far more than fits in context and pull back only what is relevant, using keyword search, embedding similarity, structured queries, or plain tools like grep over files. It is powerful but brittle in its own way: if the search misses the right item, or surfaces a stale or misleading one, the agent proceeds confidently on the wrong basis. Retrieval quality deserves its own testing.

Scratchpads, notes and files

A simple, robust pattern is letting the agent write notes outside its context window: a to-do list, a progress log, a NOTES.md file, or a memory directory. Because the notes persist independently, they survive compaction and can carry across sessions. On a long task the agent can re-read its plan instead of relying on a summary of a summary.

Notes need hygiene too. Stale or wrong notes get re-read and trusted, so it helps to have the agent update and prune them, not just append.

Sub-agents that return condensed results

Another way to protect the main context is delegation. A lead agent hands a focused job, such as “find where authentication is handled in this codebase”, to a sub-agent with its own fresh context. The sub-agent may read dozens of files, but it returns only a short summary. The noisy exploration never enters the lead agent’s context.

Trade-offs:

• Total token use goes up, sometimes a lot.
• The summary can omit something the lead agent needed.
• Coordination adds complexity and new ways to fail.

Sub-agents work best when the delegated task is well defined and its useful output is much smaller than the work required to produce it.

Putting it together

A typical long-running agent combines several of these: a lean system prompt, tools that fetch content on demand, a notes file for progress, compaction when context grows, and sub-agents for broad searches. None is magic. The right mix depends on task length and how costly it is to forget something, and the only reliable way to tune it is to read real transcripts and see what the agent actually had in front of it when it went wrong.