Researchers propose human-LLM collaboration framework to build culturally specific stereotype dataset for Spanish-speaking regions

Research on stereotypes in large language models (LLMs) has historically focused on English-speaking contexts due to the lack of datasets in other languages and the high cost of manual annotation in underrepresented cultures. To address this gap, researchers from arXiv:2607.07895 introduce a cost-efficient human-LLM collaborative annotation framework and apply it to construct EspanStereo, a Spanish-language stereotype dataset spanning multiple Spanish-speaking countries across Europe and Latin America.

EspanStereo captures both well-documented stereotypes from prior literature and culturally specific biases that are absent from English-centric resources. The dataset was developed using a framework where LLMs generate candidate stereotypes and in-culture annotators validate them, demonstrating effectiveness in identifying nuanced, region-specific biases.

The researchers evaluated Spanish-supporting LLMs using EspanStereo and found significant variation in stereotypical behavior across countries. This highlights the need for more culturally grounded assessments in stereotype analysis, as biases may manifest differently depending on regional cultural contexts.

Beyond Spanish, the proposed framework is adaptable to other languages and regions, offering a scalable path toward multilingual stereotype benchmarks. The work broadens the scope of stereotype analysis in LLMs and lays the groundwork for comprehensive cross-cultural bias evaluation.