LongMemEval
From the FaultLine Wiki
What LongMemEval is
LongMemEval is a benchmark for long-term interactive memory in chat assistants, by Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang and Dong Yu, published at ICLR 2025.[1] It targets five memory abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.[1] The dataset is 500 curated questions embedded in freely scalable user–assistant chat histories, so the evidence a question needs sits in an earlier session of a long transcript rather than in the prompt.[1]
Question types
The dataset's question_type field takes six values: single-session-user,
single-session-assistant, single-session-preference, multi-session,
temporal-reasoning and knowledge-update.[2]
Abstention is not one of them. An abstention question is marked instead by its identifier: if
question_id ends in _abs, the question is an abstention
question.[2]
Dataset variants
LongMemEval_S and LongMemEval_M differ in haystack size, not question count. Both carry the same 500 questions; S gives each question a history of roughly 115k tokens (about 40 sessions), while M scales that to around 500 sessions.[2]
FaultLine results
FaultLine has not published a LongMemEval result. The measurement instrument is still being built, and results will appear on this page when they exist.
References
- Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, Dong Yu. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. ICLR 2025. arXiv:2410.10813 — arxiv.org/abs/2410.10813
- LongMemEval dataset documentation — github.com/xiaowu0162/LongMemEval
Last updated: 2026-08-10