Paper proposes Li$^{2}$SAO to improve asynchronous RL for post-training LLMs
Li$^{2}$SAO addresses early-stage performance gaps in agentic tasks by bootstrapping with group-relative advantages and returns before switching to GAE.
1 source · cross-referenced
- Li$^{2}$SAO is a new method for asynchronous reinforcement learning (RL) in post-training large language models (LLMs).
- It targets early-stage performance gaps in agentic tasks where prior SAO variants lag due to untrained value models.
- Experiments show Li$^{2}$SAO catches up to or surpasses GRPO in early training and raises the later performance ceiling.
- Code is available at https://github.com/jd-cc-coder/Li2SAO.
A new arXiv preprint introduces Li$^{2}$SAO, a method designed to improve asynchronous reinforcement learning (RL) for post-training large language models (LLMs). The work critiques existing approaches like GRPO and SAO, noting that group-wise sampling in GRPO does not naturally fit asynchronous RL due to long-tailed rollout lengths, while SAO relies on a closed-source pretrained value model that limits reproducibility.
The authors analyze GRPO and LiSAO—a lightweight reproduction of SAO without the pretrained value model—on single-turn reasoning and multi-turn agentic tasks. They report that GRPO outperforms LiSAO on reasoning tasks, whereas LiSAO performs better on agentic tasks. However, LiSAO lags behind GRPO in early training steps because its value model is not yet well trained.
To address this, Li$^{2}$SAO introduces group-relative actor-critic bootstrapping: during early training, a group-relative advantage trains the actor while a group-centered return trains the critic. Once the critic stabilizes, the method switches to single-rollout Generalized Advantage Estimation (GAE).
The authors evaluate Li$^{2}$SAO experimentally and report that it catches up with or surpasses GRPO in early training stages and further raises the later performance ceiling. The code for Li$^{2}$SAO is available at https://github.com/jd-cc-coder/Li2SAO.
- Nov 11, 2026 · The Verge — AI
Google DeepMind unveils AlphaGenome 2 to predict genetic variant effects on cell types
Trust79 - Nov 11, 2026 · Google DeepMind — Blog
Google DeepMind unveils AlphaGenome 2 to predict DNA variant effects in specific cell types
Trust84 - Nov 11, 2026 · arXiv cs.AI
Researchers demonstrate auditable human-agent teams for autonomous synchrotron X-ray experiments
Trust79