Skip to content
Research · Nov 11, 2026

Sparse autoencoders enable compact, prefill-trained topic monitors that generalize to decoding

TopicScope trains causal topic classifiers on SAE features to label tokens during generation, showing robustness to perturbations and efficiency gains over larger baselines.

Trust79
HypeLow hype

1 source · cross-referenced

ShareXLinkedInEmail
TL;DR
  • TopicScope trains causal topic classifiers on sparse autoencoder (SAE) features to assign per-token topics during generation.
  • Monitors trained on prefill hidden states generalized to decoding for Gemma-3 1B and 4B models.
  • Classifiers lost at most 5 macro-F1 points under perturbations like off-topic asides or reordered sections.
  • Ablations show SAE-based classifiers reached baseline performance while being several times smaller than alternatives.
  • Code, dataset, and checkpoints are available on GitHub and Hugging Face.

Researchers introduce TopicScope, a framework that trains causal topic classifiers on sparse autoencoder (SAE) features to label each token’s topic during generation, allowing labels to change dynamically within a response.

The team trained three detector architectures on synthetic multi-topic conversations spanning seven topics and evaluated them on Gemma-3 1B and 4B models.

Monitors trained on features extracted from prefill hidden states generalized effectively to decoding, indicating that prefill-based training can support real-time monitoring without sacrificing accuracy.

Robustness tests showed that classifiers lost at most 5 macro-F1 points when subjected to perturbations such as inserted off-topic asides, removed topics, or reordered sections, demonstrating resilience to common conversational shifts.

Ablation studies revealed that SAE-based classifiers achieved baseline performance while being several times smaller than classifiers trained without SAE features, highlighting the efficiency gains from using SAE representations.

The authors release code, datasets, and model checkpoints on GitHub and Hugging Face, providing resources for further research and deployment of topic monitoring systems based on SAE features.

Sources
  1. 01arXiv cs.AI — Train on prefill, monitor at decode: Using sparse autoencoders for topic monitoring
Also on Research

Stories may contain errors. Dispatch is assembled with AI assistance and curated by human editors; despite the trust-score filter, mistakes happen. We correct publicly — every article links to its revision history. Nothing here is financial, legal, or medical advice. Verify before relying on any claim.

© 2026 Dispatch. No ads. No sponsorships. No paid placement. Reader-supported via Ko-fi.

Built by a person who cares about honest AI news.