Apple proposes RISED framework to train generalist LLM agents across diverse environments using rubric-based feedback

Apple’s Machine Learning Research team has proposed RISED (Rubrics for Agentic Multi-Environment Selection and Self-Distillation), a framework designed to train a single large language model (LLM) agent across diverse interactive environments. The work responds to limitations in existing curriculum and data-selection strategies, which often rely on scalar rewards that provide limited information about cross-environment relationships and fail to offer within-group reward contrast when rewards are identical.

The core innovation in RISED is the use of textual rubrics—descriptions of desired and undesired behaviors—to guide both data selection and policy supervision. An LLM judge tags each rollout using a predefined rubric vocabulary shared across environments, producing behavioral profiles that inform online data selection. This process aligns selected data with the overall behavioral composition of mixed-environment batches while limiting overlap with already-selected data.

RISED repurposes rubrics beyond reward signals by using positive rubrics (describing desired behaviors) to provide privileged context for an on-policy self-distillation teacher, supplying additional token-level supervision. Negative rubrics (describing undesired behaviors) guide subsequent rollout generation to avoid recurring failure modes, enabling the agent to learn more robust policies across environments.

The authors report that RISED achieves the highest mean pass rate across environments compared to baselines, and ranks first or second in every individual environment tested. They further analyze behavioral changes using rubric-based metrics to characterize the improvements accompanying these gains.

The work builds on Apple’s ongoing research into rubric-based alignment and reinforcement learning, including prior projects like a rubric-based reward framework for open-domain question answering and rubric-guided reinforcement learning for dense image captioning.