Hugging Face and NVIDIA release 2.8 trillion-token Indic pretraining corpus and open data pipeline

NVIDIA, IIT Madras’ AI4Bharat lab, and Hugging Face jointly released a 2.8 trillion-token synthetic pretraining corpus spanning 22 Indian languages and 12 romanized configurations, alongside an open GPU-accelerated data generation pipeline. The pipeline supports translation, quality assessment, and repair, with each output traceable to its original English source document.

The corpus includes native-script data for all 22 languages and romanized text for twelve, totaling 34 language-script configurations. Counts are reported using the Gemma-4 tokenizer: 2,601.0 billion native-script tokens and 224.6 billion romanized tokens. The data spans curated English content such as mathematics, code, and question-answer material translated into Indian languages.

The open pipeline implements a four-stage workflow for twelve languages: document translation, quality assessment, targeted repair, and romanization of a subset. For the remaining ten languages, a separate Indic Translate workflow is used. Quality is measured using a rubric inspired by Multidimensional Quality Metrics, retaining only outputs scoring at least 75 out of 100.

Nemotron 3.5 Super VL underwent continued pretraining with a 12-language subset of this corpus, adding 10 billion Indic tokens drawn from both synthetic and natural Indic text. The technical report accompanying the model documents these details.

The release emphasizes provenance and auditability: every record traces back to a curated source, and outputs are checked against inputs to ensure synthetic text remains grounded in real information. The authors highlight the scarcity of Indic language data as a core barrier to advancing Indian language technology.

The open-source release includes the full generation recipe—prompts, LLM workflows, and quality checks—so developers can adapt it to their own source material and generate additional data. This aims to make large-scale Indic data generation more accessible and reproducible for the community.