Research proposes diagnostic method to separate decision-rule misalignment from readout-coverage gaps in speech language models

Researchers introduced a generation-aligned diagnostic ladder to isolate sources of accuracy loss in speech language models when evaluated on paralinguistic tasks such as emotion recognition. The method compares four stages: the emitted answer, the option logits, an affine readout of those logits, and a linear readout of the hidden state at the same answer token. By measuring successive differences between these stages, the framework separates endpoint errors, decision-rule misalignment, and readout-coverage limitations.

Across evaluations involving five systems and two emotion corpora, state decoding exceeded generation accuracy by an average of 27.8 points, indicating substantial room for improvement in how models translate internal representations into final answers. Both decision-rule and readout-coverage gaps were positive in all ten tested conditions, suggesting these issues are pervasive rather than isolated to specific models or datasets.

The authors also report that a label-free logit correction improved generated accuracy in every condition, demonstrating that part of the decision-rule gap is actionable without additional labeled data. This implies that some performance losses stem from how decision rules are applied rather than from missing information in the model's representations.

In rank-matched comparisons, emotion information present outside the native readout generalized to held-out speakers and remained robust after controlling for measured acoustic descriptors. However, interventions that replaced selected readout-external directions typically had little effect on the emitted answers, highlighting a disconnect between available information and its behavioral use in decision-making.