Why Memory Is Unsolved

We often get asked: how is memory different from RAG? Retrieval-Augmented Generation (or RAG) was a buzzword in the pre-agent era of LLMs that created massive companies like Glean. Despite that word being phased out in more agent-native circles, it remains a dominant mental framework for most. And while those perpetually on Twitter have moved on to “memory” or more recently “continual learning”, the RAG question is a sharp one. Most memory systems are not that different from RAG under the hood. The advancements we have seen are not due to better methods for memory but because the models using them have gotten better.

2026 memory systems don’t look that different from ones of yesteryear: retrieval over a set of markdown files.

They are still essentially a form of retrieval over a corpus of text. The exact retrieval mechanism and background architecture may vary: grep over a file system of markdown files, knowledge graphs, vector search + BM25 with reranking, etc. However, they are all functionally the same: better search. The claim we make here is that perfect retrieval is not enough to solve the “memory” or “continual learning” problem. Solving this requires a paradigm shift over what memory actually is and the role it plays within agentic systems.

Memory is clearly an important problem at the moment. We see companies building their own context layer, Astra’s memory being reverse-engineered, and the rise of Instinct being attributed to its memory system. At the core is the prevailing belief in the mis-managed genius hypothesis: the idea that modern models can do almost any task if you provide them the right context or harness.

It’s also apparent to anyone who has used agents that memory is still unsolved. You cannot treat a modern agent like you would a normal human. The barrier is that memory is not retrieval, but the entire ecosystem still treats it largely like a retrieval or search problem.

Existing benchmarks are insufficient

Benchmarks and evals are likely the most important part of any AI system. Having one means you can hill-climb against it and eventually solve it. Without a good benchmark, it is hard to make progress.

The issue is that all of the existing popular benchmarks such as LoCoMo, LongMemEval, or BEAM 10M are largely sophisticated versions of the needle-in-the-haystack problem: find the right information (or lack thereof) in a large corpus. This is essentially retrieval. Agents have saturated many of these benchmarks and yet memory doesn’t feel qualitatively better.

One main issue here is that needle-in-the-haystack problems are only a small component of what it means to remember things. Most of your day-to-day isn’t about struggling to think back to remember a distant memory in the past. Instead, it’s about what you expect to happen based on an internal model of the world. Memory in this sense is a “world model” that is merely grounded in the past rather than of the past, which helps you make better decisions.

The implication is that most memory benchmarks are measuring the wrong thing. They emphasize good search over large corpuses that contain sparse signals, while most day-to-day memory use cases relate to more frequently occurring tasks and insights. They ignore the time-dependent process of a memory system evolving over time in favor of a single static corpus to ask questions against. As we’ll see later, this is a critical barrier to good memory systems today.

Memory benchmarks are also hard to build because they inherently involve large corpuses or long-horizon tasks. This creates a three-fold problem: 1) it is hard to curate the benchmarks in sufficient quantity to have good statistical power, 2) it is expensive to run the benchmark, and 3) it is hard to perform credit assignment to hill-climb. As models become better, this is ever more important. LoCoMo, the most popular memory benchmark, is largely useless today because its largest corpus is <1M tokens. This means a modern frontier model can fit the corpus into its context window and saturate the benchmark. (NB — this does not mean context windows will solve the memory problem.) The strength of modern methods combined with the compound nature of memory systems also means it is hard to evaluate like-for-like. Using a strong frontier model vs a weaker last-generation model can sometimes swing the benchmarks 10–15% even with no other changes made. However, companies reporting benchmark results rarely report what models were used in their proprietary system. Each benchmark also tends to emphasize different failure modes, from abstentions to multi-hop questions. It’s easy to build a memory system which performs well on one of these benchmarks at the expense of the others or real-life user performance.

These benchmarks are not completely useless though. They do a good job of testing for retrieval quality. They are a necessary but insufficient step to solving memory. Due to modern context windows, we believe that the only really meaningful ones today are BEAM 10M and LongMemEval-v2, both of which have corpuses beyond a modern frontier model’s context window. However, as the data generated by agents increases, these are not sufficient. BEAM 10M conversations are only about 10M tokens in length. For context, that is less than the amount of tokens our intern generated in his week of using Claude Code with us.

He generated 14M tokens in 1 week. Modern benchmarks are not enough.

Near perfect on the STALE benchmark

To explain this point on benchmarks further, we show that we were able to essentially saturate the STALE benchmark, which was released only in May 2026.

Taken from arXiv:2605.06527.

The benchmark measures an important failure mode of memory around staleness: existing facts are often invalidated by seemingly unrelated new facts. For example, if you break your leg, that may mean you can no longer bike to work. However, biking to work is largely unrelated to breaking your leg at the semantic level, so most memory systems would not be able to pick up on this.

Gemini 3.1 Pro was only able to achieve 55.2%, while the authors’ custom system achieved 68%. Our system achieved ~95.6%, or ~99.4% if you allow for a bit of benchmax prompting (with the caveat that we only ran this on random samples containing 15% of the full benchmark for speed reasons; we believe this is enough statistical power to make our point).

Blended average of 95.6% over T1 and T2 questions.

To reach this number, we modified our existing memory system in a very simple way: each piece of information ingested into the system also became a query. We then updated the returned results based on the new ingested information. For example, if we inserted the information that the user broke their leg, our query would look for and find related, stored information about topics like user health. This would enable it to find and then update the piece on bike commuting.

However, it’s unclear whether this change would cause any downstream effects on other types of failure modes. For instance, this may muddle the memory store for multi-hop questions. Ultimately, this goes beyond the standard benchmaxxing concerns: these benchmarks disguise a vibes-based memory approach as an objective benchmark. The benchmark itself may be a reasonable measurement under ideal conditions, but what the authors choose to measure is mostly a subjective judgement call. More often than not, that subjective call turns into a retrieval rather than a memory benchmark.

What memory requires beyond retrieval: why agentic scaling over search and integrations is not sufficient

Another common refrain we hear is that agentic grep + integrations with data is sufficient for a good memory system. This is not true. To see this, imagine an agent running on the next generation of Astra, but its only memory system is keyword search over a corpus the size of the entire internet. Obviously, this would present some issues. It would likely take too long and cost too much to find anything useful. Moreover, there’s a high likelihood such an agent wouldn’t be able to find the relevant information at all. As the data corpus from agents becomes larger, these concerns are very real. It’s simply not feasible for an agent to search over the entire corpus every time. We also see decreasing performance as corpus size grows.

Traditional agentic grep performance falls rapidly as corpus size increases.

In particular, search is a problem that scales with the size of the corpus. An ideal memory system does not. Instead, it scales to the size of the problem by presenting the agent with the right information for the task at hand. A very experienced chess player doesn’t have a harder time finding previous chess positions that are similar to the current board compared to an intermediate player. They have an easier time. This means that thinking of memory systems as simply an index or a cache over an agentic search process is limiting. There are three things beyond search and retrieval which a memory system has to do:

• Synthesis. Developing an integrated model of the world is important for a memory system. It has to combine different threads into a coherent picture which the agent can then pull from.
• Proactive push.It’s not sufficient to have the agent decide when it needs to find something. Instead, relevant information has to be pushed to the agent when it is useful. Otherwise, how could the agent know to even begin a search?
• Metadata. The important pieces of information are often the commentary instead of the underlying data itself. As an example, the nuances of the exceptions and process around a database table and its schema are needed to do any proper analysis.

The two key problems holding memory back

If retrieval is mostly solved, what is the bottleneck to memory then? We believe that there are two main ones:

• Blast radius. A generalization of the staleness problem from above. Information is bounded by time, scope, or priority. For instance, you can scroll through Instagram ads without letting them affect you (that much), but this is much harder for an agent that needs to reason about whether that information is applicable. Methods such as constitutional AI help, but they are limited in scope. Humans do this kind of implicit prioritization fluidly across many dimensions.
• Stability. As corpuses grow larger, an agent needs to have a stable view of the world and the past to behave reliably and effectively. The standard retrieval problem is too stochastic over large corpuses, while maintaining a world model of sorts can lead to regression toward the mean and agent slop very quickly.

Internally, we have developed evaluations to enable hill-climbing on each of these. We measure blast radius with evals focused on its two sides: when an agent fails to consider a piece of information as relevant even though it is (e.g. the bike vs broken leg problem above), and when an agent over-extends the blast radius of some information (e.g. the advertising example). On stability, we believe in more of a world-modeling approach over unbounded search. Thus, we can measure stability through a “PCR-like” approach: iteratively update our memory store in place many times. This amplifies any slop created through the process and makes measuring it much easier. Through these two eval methods, we’ve been able to make our memory system behave more naturally than benchmarking over retrieval alone would allow.

Despite the interest in memory, there are still many unsolved problems! If you have been building or thinking about memory, please send over a DM or book a call.