Researchers propose TokenScope for token-level interpretability of code-generating LLMs

Researchers from the University of British Columbia have introduced TokenScope, an interactive interpretability tool designed to expose token-level metrics, attention patterns, and structural information during the generation process of decoder-based large language models (LLMs).

The tool is positioned as a response to limitations in existing interpretability methods, which often fail to provide decoding-time signals, fine-grained uncertainty measures, or interactive mechanisms for exploring alternative generation paths, particularly for code-oriented tasks.

TokenScope enables users to perform interactive token replacement and counterfactual branching, allowing them to explore how changes to individual tokens influence subsequent generation steps.

It also incorporates code-aware aggregation via abstract syntax trees (ASTs), unifying decoding-time signals with structural program analysis to support systematic investigation of LLM behavior during code generation.

The authors argue that this integration provides a more comprehensive view of model decisions at the token level, which is critical for debugging and improving code generation performance in LLMs.