Researchers propose ASK+ to improve uncertainty-gated LLM assistance for reinforcement learning under partial observability

Researchers identify a context problem, not a capacity problem, as the reason vanilla uncertainty-gated approaches fail to integrate small language model (SLM) guidance in partially observable reinforcement learning (POMDP) settings. Across test environments, these approaches achieved an overwrite rate near zero, meaning the SLM rarely contributed an independent action.

The team proposes ASK+, which supplies the SLM with trajectory-aware context—including a partially revealed map, visited positions, and action history—along with structured chain-of-thought reasoning. This converts the SLM from a passive redundancy check into an informative consultant that occasionally corrects the policy.

The authors further establish that predictive entropy used for selective querying measures action uncertainty rather than state uncertainty and remains informative in POMDPs, making uncertainty-gated assistance viable beyond fully observable settings.

In empirical evaluations, ASK+ delivered substantial gains: on DoorKey, success rose from 89% (vanilla ASK and PPO baseline) to 93%; on FourRooms, from 53% to 70%; and on HigherLower, accuracy reached 73.7%, matching the SLM-only upper bound.

Across all environments, Qwen3.5-2B matched or exceeded Qwen3.5-4B, indicating that prompt design and selective gating can dominate the impact of model scale, enabling effective guidance without larger models.