New benchmark shows how reader-facing memory formats affect LLM evaluation scores

Researchers from Yuan Si, Simeng Han, Daming Li, and Jialu Zhang introduce RENDER, a benchmark designed to control for the format of reader-facing memory artifacts in LLM evaluations. The core insight is that systems may render the same conversation history in different ways—such as raw dialogue excerpts, ChatGPT-style entries, LangChain summaries, MemGPT-style typed records, or structured ledgers—and these formats can significantly influence measured performance.

RENDER combines a five-level "packet ladder" that localizes when answer-bearing content enters the input with deterministic templates mimicking common deployment formats. The benchmark evaluates nine models on 500 LongMemEval questions, finding that matched-budget resolved packets outperform recency-truncated raw dialogue by between 42.4 and 72.6 points, depending on the model.

In deployed-style templates, the best-worst spread across models ranges from 24.6 to 48.8 points per model. Under the primary scorer, ChatGPT-style entries yield higher point estimates than raw conversation for seven of the nine models. Judge rescoring preserves the positive aggregate effect, though model-specific significance is mixed.

Notably, three models that score 0% on formal ledger packets answer the same facts correctly between 45.4% and 53.4% of the time when those facts are presented as natural-language entries. The effect remains robust under retrieval noise and transfers to HotpotQA, indicating that evaluation protocols should explicitly report or control for the reader-facing artifact format.

The authors argue that current memory and RAG evaluations often treat the answering model's input as an implementation detail, which can obscure true model capabilities and lead to inconsistent or misleading comparisons across studies.