Wazobia Eval introduces a 550-example Nigerian Pidgin benchmark for emotion, sarcasm, and cultural reasoning

Nigerian Pidgin is one of Africa’s most widely spoken languages, yet remains underrepresented in language model evaluation. Existing benchmarks tend to emphasize translation, transcription, or generic sentiment analysis, leaving culturally nuanced language understanding unmeasured.

Wazobia Eval introduces a benchmark specifically designed for Nigerian Pidgin emotion understanding, sarcasm detection, and cultural reasoning. The benchmark is built on a manually annotated dataset containing over 550 examples and a 16-category emotion taxonomy tailored to culturally specific emotional registers not captured by conventional sentiment frameworks.

The benchmark provides standardized evaluation protocols and benchmark tasks for assessing model performance on nuanced Nigerian language understanding. It also documents the annotation methodology, taxonomy development process, and preliminary pilot evaluation results.

The dataset and associated resources are publicly available on Hugging Face, with additional links to code repositories provided in the paper.