Falcon-Emirati-7B released on Hugging Face to specialize in Emirati Arabic dialect and culture

Falcon-Emirati-7B is a dialect-specialized model built on the Falcon-H1-Arabic base, designed to understand and generate Emirati Arabic with native-level vocabulary, tone, and cultural context. The model uses the Falcon-H1 hybrid architecture, combining State Space Models (Mamba) and Transformer attention in parallel within each block, with fused outputs before projection. This architecture aims to retain the linear-time efficiency of Mamba for long sequences while preserving the precision of attention for long-range dependencies, which is particularly relevant for morphologically rich languages like Arabic.

The team constructed a dedicated Emirati data pipeline using three complementary sources: authentic Emirati-dialect web data sourced from Emirati websites and forums, Modern Standard Arabic (MSA) data focused on Emirati culture and identity, and synthetic Emirati-dialect data guided by glossaries and style rules to ensure authenticity. The synthetic data was constrained to avoid generating content that sounds grammatically correct but culturally or linguistically off to native speakers.

Evaluation was conducted using both manual assessment by native speakers and automatic benchmarks. Native speakers judged outputs for correctness, naturalness, tone, and cultural fit, while automatic scoring tracked progress during training. The team also explored multiple training strategies through ablation studies, testing data mixes, training stages, and supervision approaches to determine the most effective adaptation recipe for Emirati Arabic.

The model was built on the 7-billion-parameter variant of the Falcon-H1-Arabic family, which spans 3B, 7B, and 34B parameters and supports context windows up to 128K and 256K tokens. The 7B variant was selected as a balance between quality and practicality for training and inference, offering sufficient capacity for dialect adaptation while remaining feasible to deploy as a chat model.