Skip to content
Research · Nov 2, 2026

Paper proposes Li$^{2}$SAO to improve asynchronous RL for post-training LLMs

Li$^{2}$SAO addresses early-stage performance gaps in agentic tasks by bootstrapping with group-relative advantages and returns before switching to GAE.

Trust79
HypeLow hype

1 source · cross-referenced

ShareXLinkedInEmail
TL;DR
  • Li$^{2}$SAO is a new method for asynchronous reinforcement learning (RL) in post-training large language models (LLMs).
  • It targets early-stage performance gaps in agentic tasks where prior SAO variants lag due to untrained value models.
  • Experiments show Li$^{2}$SAO catches up to or surpasses GRPO in early training and raises the later performance ceiling.
  • Code is available at https://github.com/jd-cc-coder/Li2SAO.

A new arXiv preprint introduces Li$^{2}$SAO, a method designed to improve asynchronous reinforcement learning (RL) for post-training large language models (LLMs). The work critiques existing approaches like GRPO and SAO, noting that group-wise sampling in GRPO does not naturally fit asynchronous RL due to long-tailed rollout lengths, while SAO relies on a closed-source pretrained value model that limits reproducibility.

The authors analyze GRPO and LiSAO—a lightweight reproduction of SAO without the pretrained value model—on single-turn reasoning and multi-turn agentic tasks. They report that GRPO outperforms LiSAO on reasoning tasks, whereas LiSAO performs better on agentic tasks. However, LiSAO lags behind GRPO in early training steps because its value model is not yet well trained.

To address this, Li$^{2}$SAO introduces group-relative actor-critic bootstrapping: during early training, a group-relative advantage trains the actor while a group-centered return trains the critic. Once the critic stabilizes, the method switches to single-rollout Generalized Advantage Estimation (GAE).

The authors evaluate Li$^{2}$SAO experimentally and report that it catches up with or surpasses GRPO in early training stages and further raises the later performance ceiling. The code for Li$^{2}$SAO is available at https://github.com/jd-cc-coder/Li2SAO.

Sources
  1. 01arXiv cs.CL — Li$^{2}$SAO: Lifting Light Singl-Rollout Asynchronous RL Optimization via Group-Relative Actor-Critic Bootstrapping
Also on Research

Stories may contain errors. Dispatch is assembled with AI assistance and curated by human editors; despite the trust-score filter, mistakes happen. We correct publicly — every article links to its revision history. Nothing here is financial, legal, or medical advice. Verify before relying on any claim.

© 2026 Dispatch. No ads. No sponsorships. No paid placement. Reader-supported via Ko-fi.

Built by a person who cares about honest AI news.