RL fine-tuning yields more structured internal representations than SFT for mathematical reasoning, study finds

A new arXiv preprint compares reinforcement learning (RL) and supervised fine-tuning (SFT) for mathematical reasoning models, finding that RL fine-tuning yields more structured internal representations. Using linear probes on layer-wise hidden states, the authors report that RL models achieve higher accuracy in predicting answer correctness than SFT models, indicating more linearly separable representations.

Mean ablation studies further show that RL models develop a hierarchical architecture where deeper layers become progressively more critical, whereas SFT models distribute importance more uniformly across layers. The authors conclude that RL training fundamentally restructures how models represent and process reasoning problems.

The study also examines token-count variability under repeated sampling to assess adaptive compute allocation. While some RL-tuned models exhibit higher variability than their SFT counterparts, others show strong consistency, suggesting that token allocation depends more on the overall training pipeline than on the choice between RL and SFT alone.

The authors propose that token-allocation variability reveals the spread of plausible on-policy reasoning, highlighting which models exhibit stable policies versus those with under-determined or non-identifiable solution behavior.