Memory judgment, measured.
A matched comparison of four systems ranking persistent-memory evidence for tool-using agents. Every arm receives the same ten retrieved candidates for each of 1,986 LoCoMo questions.
Leaderboard
Ranked by the selected measure. Recall@10 is a candidate-coverage invariant; reranking cannot retrieve a memory absent from the candidate pool.
| Rank | System | MRR@5 | Recall@1 | Recall@10 |
|---|
AtMem-MJM paired differences
Difference is AtMem-MJM minus the comparator. Intervals are 95% percentile bootstrap intervals, resampling conversations (10 clusters; 10,000 replicates).
How to read this
AtMem-MJM's MRR@5 is 0.0092 above off-the-shelf Skywork, with a positive conversation-cluster interval. This is one exploratory benchmark and the ten conversation clusters make uncertainty coarse.
Protocol and limitations
The Jev API rerun used the same frozen dataset and candidate IDs, with 40 batches of 50 cases to match the historical request grouping. All 1,986 cases returned scores for all ten candidates. Four historical evidence-reference anomalies remain in the primary all-case denominator.
AtMem-MJM was trained on 72 synthetic pairs, validated on 24, and audited on 24 examples from two held-out template families. The pilot is small and highly templated. The leaderboard measures evidence ranking only; it does not establish answer accuracy, tool-action quality, authorization, or general memory quality.
No questions, candidate memories, evidence IDs, private histories, synthetic training records, or model weights are included in this Space.