Researchers propose SciReC benchmark and DMRA diagnostic framework to evaluate multimodal relational reasoning in large language models

Researchers introduced SciReC, a model-adaptive multimodal academic dialog benchmark designed to evaluate multimodal large language models (MLLMs) on relational reasoning tasks. The benchmark spans multiple relational categories—including analogical, structural, and cause-effect reasoning—reflecting higher-order cognitive processes that require perceptual understanding, comparison, and integration of relationships between concepts.

To analyze failure modes, the team proposed DMRA, a deficit-based diagnostic framework that quantifies the contribution of key components such as visual understanding, knowledge recall, and memory limitations. This framework helps isolate the primary causes of unsuccessful cases, revealing broader patterns across models and domains.

In reported results, Claude 4.6 achieved the highest overall relational score at 73%, followed by GPT 5.4 at 68%. Performance varied by model type: open-source models performed worst on spatial relations, while proprietary models struggled more with hierarchical and sequential relations. Domain-level analysis showed the lowest performance in Astronomy and the highest in Psychology.

The DMRA analysis indicated that relational reasoning deficits were the dominant source of errors across all models, with memory limitations as the second most significant factor. These findings suggest that current MLLMs face consistent challenges in integrating and reasoning over complex, multi-modal relationships, particularly when domain knowledge and memory demands are high.