Open Source
NVIDIA and AI4Bharat release 2.8T Indic tokens
Hugging Face Bloghigh confidence
Why it matters
- →Solves the critical data scarcity problem for India's 22 constitutional languages.
- →Provides an open, reproducible pipeline for generating high-quality synthetic technical content.
- →Enables multilingual models to handle complex reasoning in math and code, not just translation.
NVIDIA and AI4Bharat have released a 2.8 trillion-token pretraining corpus covering all 22 Indian constitutional languages, accompanied by an open synthetic data generation pipeline. The dataset includes native-script and romanized variants for 12 languages, plus native-only data for the remaining 10, with specific focus on technical domains like mathematics and code. Built using NeMo Data Designer and Gemma-4-31B-it, the pipeline ensures provenance and quality control through automated translation and assessment workflows. This release provides the first large-scale, open resource for training models capable of complex reasoning in Indic languages.
Read originalThe story around this
Earlier coverage that leads up to this article, and what followed. Lines connect each piece to the closest one after it, converging here.
