Frozen 16 MB vocabulary achieves parity with learned embeddings in small language models while reducing size and compute

A new arXiv preprint proposes replacing the large learned embedding tables in small language models with a much smaller frozen vocabulary derived from principal-component projection and sign binarisation. In experiments on a Gemma-shaped 270M-parameter model trained on 2.05B tokens, the frozen 16 MB vocabulary table matched or outperformed the learned embedding table on LAMBADA, HellaSwag, and PIQA, while only underperforming on ARC-Easy.

The authors report that the learned embedding table contributes 0.20 nats of validation loss for 2.6 times the trainable parameters, whereas the frozen table eliminates those parameters entirely. When paired with a ternary BitNet b1.58 body, the entire model shrinks to a 46 MB file, yet still ties the floating-point model on LAMBADA and PIQA benchmarks.

On a laptop CPU, the ternary model with the frozen vocabulary generates 615 to 740 tokens per second within 62 MB of memory, which is five to six times faster than an 8-bit GGUF version of the floating-point model in llama.cpp while using one-seventh the memory.

The frozen vocabulary also shows stronger performance on rare words, improving by 1.7 to 2.3 nats over the learned table. It supports adding 10,000 new words after training with only a 3% cost, whereas the learned table’s performance collapses by 43% under the same scenario.

All code, data, checkpoints, and evaluation answers are released alongside the paper.