What Memory Benchmarks Don't Test

Every comparison of AI memory systems ranks on retrieval accuracy. None rank on what happens when the system retrieves confidently wrong information, holds contradictory beliefs simultaneously, or trusts stale knowledge as if it were current. Here's the evaluation framework they're missing. In March 2026, three independent comparison posts evaluated AI agent memory systems. All three used LoCoMo as their benchmark. All three ranked systems by retrieval hit rate. All three declared a winner. None of them asked the question that actually matters in production: what does the system do when it's wrong? This isn't a criticism of LoCoMo. It's an excellent benchmark for what it tests: whether a system can surface a relevant memory given a query. But retrieval accuracy is a necessary condition for useful memory, not a sufficient one. A system that retrieves the right fact 90% of the time and confidently hallucinates the other 10% — with no mechanism to distinguish between them — is not a produ