Bilibili releases Index-1.9B series of open small language models with technical details and benchmarks

Bilibili released Index-1.9B, a series of four open small language models with 1.9 billion non-embedding parameters each. The series includes Index-1.9B-Base, a foundation model pre-trained on 2.8 trillion predominantly Chinese and English tokens; Index-1.9B-Pure, a control variant with instruction-like data filtered out; Index-1.9B-Chat, aligned via supervised fine-tuning and direct preference optimization; and Index-1.9B-Character, which adds retrieval-augmented generation for few-shot role-playing customization.

Training used a Warmup-Stable-Decay learning-rate schedule with increased curated data concentration during the decay phase and a Norm-Head output layer to stabilize training under large learning rates. On standard benchmarks covering examination, reasoning, mathematics, and code, Index-1.9B-Base achieved an average score of 64.92, reported as competitive with or exceeding larger open models.

The authors also report controlled studies on model depth, learning-rate magnitude and scheduling, the interaction between learning-rate decay and data quality, and the effect of including instruction data during pre-training. They document an unexplained performance surge midway through the constant-learning-rate phase.

All models and evaluation code are released under open licenses at the project’s GitHub repository.