Sparse autoencoders enable compact, prefill-trained topic monitors that generalize to decoding
TopicScope trains causal topic classifiers on SAE features to label tokens during generation, showing robustness to perturbations and efficiency gains over larger baselines.
1 source · cross-referenced
- TopicScope trains causal topic classifiers on sparse autoencoder (SAE) features to assign per-token topics during generation.
- Monitors trained on prefill hidden states generalized to decoding for Gemma-3 1B and 4B models.
- Classifiers lost at most 5 macro-F1 points under perturbations like off-topic asides or reordered sections.
- Ablations show SAE-based classifiers reached baseline performance while being several times smaller than alternatives.
- Code, dataset, and checkpoints are available on GitHub and Hugging Face.
Researchers introduce TopicScope, a framework that trains causal topic classifiers on sparse autoencoder (SAE) features to label each token’s topic during generation, allowing labels to change dynamically within a response.
The team trained three detector architectures on synthetic multi-topic conversations spanning seven topics and evaluated them on Gemma-3 1B and 4B models.
Monitors trained on features extracted from prefill hidden states generalized effectively to decoding, indicating that prefill-based training can support real-time monitoring without sacrificing accuracy.
Robustness tests showed that classifiers lost at most 5 macro-F1 points when subjected to perturbations such as inserted off-topic asides, removed topics, or reordered sections, demonstrating resilience to common conversational shifts.
Ablation studies revealed that SAE-based classifiers achieved baseline performance while being several times smaller than classifiers trained without SAE features, highlighting the efficiency gains from using SAE representations.
The authors release code, datasets, and model checkpoints on GitHub and Hugging Face, providing resources for further research and deployment of topic monitoring systems based on SAE features.
- Nov 11, 2026 · The Verge — AI
Google DeepMind unveils AlphaGenome 2 to predict genetic variant effects on cell types
Trust79 - Nov 11, 2026 · Google DeepMind — Blog
Google DeepMind unveils AlphaGenome 2 to predict DNA variant effects in specific cell types
Trust84 - Nov 11, 2026 · arXiv cs.AI
Researchers demonstrate auditable human-agent teams for autonomous synchrotron X-ray experiments
Trust79