Apple study finds LLMs unreliable for most machine translation evaluation tasks

Apple’s Machine Learning Research team evaluated the performance of large language models (LLMs) as alternatives to human annotators for machine translation (MT) quality evaluation using two established frameworks: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA). The study compared LLM outputs against human judgments across 70 language pairs, the WMT23 dataset, and the WMT25 dataset.

Results indicate that while LLM agreement with human annotators sometimes exceeds inter-human agreement in specific tasks, performance fluctuates substantially across annotation schemes, language pairs, and domains. Neither humans nor LLMs demonstrated consistent reliability across most evaluated settings.

The authors identify distinct challenges for each group: human annotators struggled particularly with fine-grained MQM annotations and low-resource language pairs, whereas LLMs faced difficulties with minor errors, incorrect language variants, and error span annotation tasks.

The findings suggest potential for improvement through human-LLM collaborative annotation pipelines to address the identified reliability issues in MT quality evaluation.