Paper proposes parametric multimodal memory for personalized agents to store perceptual identity signals

Researchers from arXiv propose a method to store perceptual user traits—such as voice tone, facial appearance, and expressions—within a parametric memory bank for personalized agents. The approach addresses a key limitation of current systems, which rely heavily on text captions to represent users. These captions capture nameable facts (e.g., "my cat is named Bibi") but fail to encode non-nameable perceptual signals that define identity across modalities.

The proposed method decomposes recall into two subproblems: a vision-language model (VLM) grounds the referent in context (what and where), while a dedicated encoder extracts an identity key (who). The identity key is stored as a single inline token, read by attention during generation without requiring an external round-trip. The authors demonstrate that neither component alone suffices: a caption-based re-identifier recovers as little as 0.11 of a dedicated encoder's recall on non-nameable signals, collapsing toward chance. A VLM alone identifies cross-age faces with 0.54 recall, while a face encoder reaches 0.81; an ungrounded encoder recognizes a two-person scene referent at 0.05 recall. Combined, the system achieves a correct-region oracle recall of 0.96, generalizing across multi-speaker audio and video.

The recognition core of the method is training-free and reproduces the encoder's recall on any frozen model at O(1) registration cost, meaning no additional training is required for new models. The authors introduce PerceptMem, a benchmark spanning 12 domains and 1,080 tasks, to evaluate perceptual identity memory. They find that perceptual identity is capacity-limited, while exact facts are binding-limited, suggesting that identity traits belong in a parametric bank and factual knowledge in a text store. The two memory systems compose cleanly, enabling agents to remember not only what users say but also what they are like.