Hugging Face study finds hybrid models excel at predicting meaning-bearing tokens but trail on verbatim repeats

Researchers from the Allen Institute for AI (AI2), in collaboration with Hugging Face, compared the token-level prediction behavior of a hybrid model (Olmo Hybrid) against a closely matched transformer (Olmo 3) to isolate architectural differences. Both models were trained with the same data, tokenizer, and recipe, ensuring that observed differences stem from architecture rather than configuration.

The study evaluated predictions across diverse text types, including articles, Wikipedia entries, books, scientific papers, and structured code (Python, HTML, LaTeX). For each token, the models assigned probabilities to possible next tokens based on preceding context, and researchers computed the loss gap—the difference in loss between the two models—to quantify which architecture predicted the actual next token more accurately.

Results showed that Olmo Hybrid consistently outperformed Olmo 3 on meaning-bearing tokens such as nouns, verbs, and adjectives, with a loss gap near 0.04. The advantage was smaller for function words like "the" or "of," where the gap hovered around 0.02. The hybrid’s edge was particularly pronounced for adverbs, adjectives, and existential terms such as "there."

Conversely, the hybrid’s advantage diminished or disappeared in contexts where the next token was a verbatim repeat of an earlier phrase. The longer the repeated n-gram, the smaller the hybrid’s lead, approaching zero as repetition length increased. Attention layers, which can directly retrieve prior tokens regardless of distance, were found to excel in such cases, explaining the transformer’s relative strength.

To further validate these findings, the team conducted pretraining experiments using three 1B-parameter models: a transformer, a hybrid, and a pure recurrent model (no attention). On non-repeated, meaning-bearing tokens, both the hybrid and pure recurrent models outperformed the transformer, with the hybrid performing best overall. On repeated tokens, the pure recurrent model lagged due to its inability to retrieve prior tokens exactly, while the hybrid retained some attention layers and matched the transformer’s performance.

The authors propose using filtered token losses—evaluating loss only on specific token categories—as a more nuanced way to compare architectures during pretraining. They demonstrate that such metrics reveal differences in copying ability and content-word handling early in training that aggregate losses obscure.