Benchmark finds frontier LLMs leak sensitive memory data in up to 69% of tests

A new benchmark called CIMemories evaluates whether large language models (LLMs) appropriately control the flow of information from their persistent memory based on task context. The benchmark uses synthetic user profiles containing over 100 attributes per user, paired with diverse task contexts where each attribute may be essential for some tasks but inappropriate for others.

In evaluations of frontier models, researchers observed up to 69% attribute-level violations, where models inappropriately disclosed sensitive information from memory. Lower violation rates were often achieved at the expense of task utility, indicating a trade-off between privacy and performance.

Violations accumulated with repeated use: for GPT-5, violations increased from 0.1% after a single task to 9.6% after 40 tasks. When the same prompt was executed five times, violations reached 25.1%, revealing arbitrary and unstable behavior where models leaked different attributes for identical prompts.

Privacy-conscious prompting did not resolve the issue. Models tended to overgeneralize, either sharing too much information or too little, rather than making nuanced, context-dependent decisions about disclosure.

The researchers conclude that these findings reveal fundamental limitations in current models’ ability to reason contextually about information sharing. They argue that addressing these issues will require capabilities beyond better prompting or scaling, such as contextually aware reasoning mechanisms.

A separate paper proposes a method to improve contextual integrity by combining explicit reasoning with reinforcement learning. Using a synthetic dataset of 700 examples with diverse contexts and disclosure norms, the approach substantially reduced inappropriate information disclosure while maintaining task performance across multiple model sizes and families. Improvements transferred to established contextual integrity benchmarks like PrivacyLens, which uses human annotations to evaluate privacy leakage in AI assistants.