Apple proposes RLTL;DR for self-improving agents that internalize their own feedback

Apple’s Machine Learning Research team describes RLTL;DR, a reinforcement learning approach designed to enable agents to self-improve in domains where tasks are so difficult that initial success rates are effectively zero. The method conditions each new rollout on concise, self-generated insights—formatted as TL;DR statements—derived from verifier feedback after failed attempts. By backpropagating on these in-context insights, the approach internalizes a direct mapping from tasks to actionable insights, allowing the agent to learn even when no external teacher or solution exists.

The authors report that standard GRPO training on a Qwen 3.5 9B Thinking policy on filtered tool-calling and coding datasets (with Pass@128 = 0) yields no progress, maintaining a Pass@1 of 0% to 1%. RLTL;DR breaks this barrier, achieving Pass@1 of 14–31% when insights are included in context during training and 12–13% at evaluation time even without in-context insights. The team attributes this improvement to the internalization of task-to-insight mappings.

To further isolate the mechanism, the researchers introduce SFTL;DR, a supervised fine-tuning variant that trains only on (task, insight) tuples without rollouts or backpropagation. Training on just 4,000 such tuples recovers nearly the full performance of RLTL;DR and classical supervised fine-tuning on full rollouts, suggesting a compact training paradigm where agents learn generalizable rules of the form ‘on this sort of task, keep this sort of thing in mind.’

The paper positions RLTL;DR as a step toward enabling agents to autonomously improve in open-ended, high-stakes environments where external guidance is unavailable, and highlights the potential for more data-efficient training regimes based on internalized feedback.