How to Build Agent Memory: The Building Blocks of Agent Memory

In the first episode of our new webinar—What do agents know?—we break down "Are We Ready for an Agent-Native Memory System?", a survey paper that describes the taxonomy of existing agent-native memory systems.

There's a lot of noise right now about agent memory, agent knowledge, and "company brains." Everyone agrees agents should learn from their interactions. Almost nobody agrees on what that actually means.

At its simplest, memory is how agents capture and learn from their interactions over time. But even that definition is shifting. Memory historically meant user personalization - how does the agent learn about you? The trend we're seeing now is toward task-oriented memory: how does the agent learn how to do things?

To make sense of the space, we started a journal club at Dosu. Our first pick was a survey paper called "Are We Ready for an Agent-Native Memory System?" - chosen because it lays out a clean typology for decomposing any memory system into four components: representation and storage, extraction, retrieval and routing, and maintenance.

If you've worked with databases, this should feel familiar. It's CRUD: how data is laid out on disk, how you write to it, how you read from it, and how you update it. The analogy is a useful lens for almost every design decision below, and we'll keep coming back to it.

Representation and storage: what shape does memory take?

The paper separates logical representation (how you organize information) from physical storage and indexing (how you implement it), and identifies three rough categories:

Token-level sequence representation. Memory as text that can be injected directly into a prompt - think of how Claude Code injects a CLAUDE.md file. Simple, transparent, and the thinnest possible representation layer.

Graph and tree-based representation. Hierarchical or relational structure, usually paired with a graph database or graph-like query engine. Great for precise, low-latency retrieval of specific facts.

Heterogeneous composite representation. A little bit of everything. And notably, this is where most real memory tools land today. Some graph parts, some tree parts, some plain text. It's hard to do just one thing.

That's a theme you'll see throughout this post: there is no one-size-fits-all memory system. The quality of what you retrieve depends on how you organized it; the latency of retrieval depends on your representation layer. Everything touches everything.

One finding from our own work that surprised us: the optimal representation for agents is not the optimal representation for humans. Human knowledge at most organizations lives in documentation - Confluence pages, Notion docs - designed to be read front to back. Agents don't want that. They're happy to traverse graph relationships, fan out in parallel, and grind through needle-in-a-haystack searches that humans suffer on. They even parse aggressively compressed, semi-structured representations ("caveman English") remarkably well. Consistency between the human view and the agent view matters, but forcing agents to read like humans leaves capability on the table.

Extraction: how opinionated should your writes be?

Extraction is the write path: turning raw data - a Slack conversation, a trace from a previous agent session - into a memory you can access later.

The central design question is how schema-constrained your writes are. How much of an opinion do you bake into the system about the structure of the world? The paper describes three options:

• Schema-constrained extraction. Strong assumptions, easy graph construction, precise retrieval - but you've limited the universe of things you can know about.
• Schema-free extraction. Discover the relational structure bottom-up from data. A very active research direction right now (Microsoft's Memora is a good example), but nobody has the ideal answer yet.
• Raw extraction. Skip the abstraction entirely and just store previous session logs. Lightweight tools in this category essentially say: here's the last time something similar came up, use it as a direction.

If this sounds like SQL versus NoSQL all over again, that's because it is. We're in the pre-relational-database era of memory: nothing has emerged as the default yet, and the right choice depends heavily on how the system will be used.

One factor that's underdiscussed: operational cost. A memory system front-loads compute. Strict schemas and complex graphs are expensive to build and even more expensive to keep current. So the real question at write time is: what's worth precomputing, and what's a waste of time?

Retrieval and routing: reading it back

Retrieval has been through a whole trend cycle. First everything was RAG - vector retrieval over embedded chunks. Then everything swung to agentic retrieval - give the model a file system and some tools, let it grep and ls its way to an answer. Models have been heavily post-trained to be good at exactly that, and it shows.

Now the pendulum is settling somewhere in the middle: fusion retrieval. File system operations have their place. So does vector search. So do MCP reads for external state that isn't in the code. Different questions want different retrieval strategies, and none is universally ideal.

A database has many index types, but it also has a query planner that decides how to execute a given query against your storage - and nothing compels it to use an index if a scan is faster. We expect something similar to emerge in memory: queries routed to different retrieval paths optimized for the task, rather than one path for everything.

There's also a step before any of that: the model first has to decide it wants to retrieve at all - that it doesn't already know the answer from model knowledge. The job of a memory product is to provide rich affordances so that when the model does reach for retrieval, it can pick the right path.

Maintenance: the hardest problem in memory

Representation and retrieval get most of the attention. Maintenance is where the hard, valuable, unsolved problems live: how does memory stay true as the world changes underneath it?

The strategies in circulation map neatly to systems concepts you already know:

• Time-based decay. Facts lose importance as they age. Useful, but incomplete - different kinds of facts have different characteristic rates of decay, so staleness and decay are related but not the same idea.
• Capacity-driven eviction. Treat memory like a cache with a fixed budget. Least-recently-used, first-in-first-out - the Redis playbook.
• Recompute-cost awareness. Some knowledge is trivial to rederive (just read the one code file), so it's not worth the carrying cost of maintaining. Low ROI? Evict it.
• Semantic consolidation. On a schedule or on upsert, reconcile new information with what you already know: merge similar memories, delete what's been invalidated.

Two ideas stood out to us as especially important.

First: localize the blast radius of an update. When a new event arrives - a commit, a Slack thread - you want to update only the memories that actually depend on it, not relearn everything you know. You can't afford to ask "is this stale?" of every memory on every push to main. Localized updates are what make a memory system stable and affordable over the long term.

Second: separate your episodic memory from your semantic memory. Raw artifacts - session logs, historical events - are records of what happened. They don't need updating; they're the past, and sometimes the past is exactly what you want (say, when debugging a change). Keep those raw, and maintain a much smaller layer of current understanding on top that references them. This also sidesteps a known weakness: summarization is inherently lossy, and agents consistently do better with raw source material than with compressed versions of it.

And there's a third kind of memory beyond those two that most systems barely touch: procedural memory. Knowing about things is not the same as knowing how to do things. Capturing how to accomplish a task - not just facts about the world - is one of the most interesting open areas in the space.

What the evaluations actually showed

The paper benchmarks a range of memory systems against each other, and the headline result surprised nobody who's been paying attention: no system won across all task types. Memory design is always tied to what you're optimizing for.

But the more interesting finding might be about the benchmarks themselves. Most public memory benchmarks are conversational - LoCoMo, LongMemEval, BEAM, and friends test whether a system remembers that a certain person is a vegetarian across a long chat history. That was a fascinating NLP question ten years ago. It's not what a coding agent needs to work effectively. Task performance - is the outcome high quality, does it happen fast, does it happen at low cost - is what end users actually care about, and datasets measuring memory's effect on it are only just starting to appear. Some benchmarks even use exact keyword matching as the evaluation criterion, which quietly biases results toward systems that retain raw details. To some degree, the whole field is bottlenecked by its benchmarks.

A few other findings worth sitting with:

You have to convince the agent your memory is true. If you return something close to the answer - or something the agent feels compelled to independently verify - you've undermined the value of the memory system. Agents today deeply trust the file system; hand them a memory and they'll often go double-check the code anyway. Sometimes the work is presentation: being explicit that "this is true, you don't need to re-verify." And this is an area where memory builders are somewhat beholden to the foundation model providers - what Anthropic and OpenAI choose to do around memory and tool use will shape what agents are trained to trust.

Evidence needs provenance - and the right kind. Knowing what evidence a memory was built on tells you when its ground truth has shifted. But it's also worth asking what evidence is dispositive for a claim. In software it's tempting to say code is always the source of truth. Not always: if you're writing firmware, your code can have an opinion about what setting a register value does, but the hardware datasheet is what actually decides.

Structure buys robustness. The more schema-driven your extraction, the more you can tolerate variability in the model that builds the memory. That's a real cost lever: bake in stronger assumptions about structure, and you may be able to build and maintain memory with a smaller, cheaper model. It's a genuinely complicated optimization surface.

Long horizons demand structure. Simple approaches - just saving session logs - break down as volume grows. Borges territory again: The Library of Babel contains all the world's knowledge, all the world's misinformation, and mostly just noise. Without abstraction, an ever-growing pile of raw text becomes intractable. And repeatedly feeding LLM output back into LLMs compounds the problem: every re-summarization is lossy, and the losses accumulate as drift away from ground truth. It's a game of telephone, and even very good models play it.

If you're building one of these

Three takeaways we'd offer anyone starting on agent memory:

Start from the task. If you're building a customer support bot where personalization matters, look at systems and benchmarks that are good at the "identify the vegetarian" class of problem. If you want agents that get better at work across your organization, look for task-oriented benchmarking instead.

Budget for maintenance, not just storage. Everything you index, you pay to maintain - exactly like a database, where indexing everything means expensive writes and a bigger machine. The best systems are deliberate about what they index based on the tasks they're optimizing for. And expect a spectrum of viable products: dirt-cheap trace search with keywords on one end, full-blown memory systems that handle high-abstraction questions on the other. There's room in the market for both.

It's harder than it looks. Code generation is cheap now, and it's tempting to conclude you can vibe-code a memory system in a weekend and replace your external tools. Our experience - and this paper - suggest otherwise. The write path, the read path, and especially the maintenance loop are each deep problems, and they all depend on each other.

We'll keep breaking down research like this as the space evolves. And if you'd rather not build the maintenance loop yourself - well, that's what we're here for.