Study finds semantic retrieval via LSP often costs more tokens than lexical grep for coding agents

Coding agents spend most of their context budget on retrieval, and the prevailing assumption has been that semantic retrieval via the Language Server Protocol (LSP) is more token-efficient than lexical grep. This new arXiv preprint formalizes the question with a single metric—tokens-to-success—and tests it across a five-arm ablation isolating semantic retrieval from confounds. The authors report a preliminary study spanning Python and TypeScript repositories and three frontier models: Claude Opus 4.8, Sonnet 4.6, and Haiku 4.5.

On symbol-named localization, LSP retrieval increased token use by 6% to 118% relative to grep, and agents ignored the LSP when it was free. On reference-completeness, LSP improved precision but did not reduce token consumption and could not raise the recall ceiling set by agent thoroughness; token savings were observed only for the weakest model. Tool choice was task-dependent: models defaulted to grep on localization (0–6% semantic use) but reached for LSP about half the time on reference tasks without prompting.

The starkest gap emerged on edits scored by real test execution. Grep solved multi-file renames perfectly, while a location-only LSP failed three-quarters of them by missing call sites. Even a fully index-warmed, text-enriched LSP—mimicking production LSP-MCP servers by inlining each reference’s line—recovered most of the gap but could not close it, because renames must touch comments and strings that semantic references exclude.

The authors conclude that the implication is not "LSP-always," but an adaptive router that keys retrieval strategy to task class, model capability, and lexical noise. They provide code and data alongside the paper and note that the study maps three pre-stated failure modes onto measurable variables, offering a methodology that can be reused to evaluate retrieval choices in other agentic coding systems.