Researchers introduce benchmark testing multimodal models’ abstract perceptual reasoning

Researchers propose The Unwritten Benchmark to probe abstract perceptual reasoning in multimodal models, focusing on acousto-kinematic word inference. In this task, models must infer words being written across three writing styles using only the audio of pen scratches and video of hand movements, without visible ink.

Evaluation results show a stark contrast between human and machine performance: human participants achieve over 80% ordered letter accuracy, while leading multimodal models, including GPT-4o and Gemini 2.5-Pro, fail to surpass 10% accuracy.

The study also identifies a paradoxical fusion effect, where providing both audio and video modalities often degrades model performance instead of improving it, indicating a breakdown in the ability to synthesize complementary perceptual cues for this cognitive task.

The findings underscore significant limitations in current models’ cross-modal causal reasoning and their understanding of micro-kinematics essential for intuitive perceptual reasoning.