Researchers propose explicit rubric-based method to replace opaque reward models in image generation

Aligning multimodal generative models with human preferences typically relies on reinforcement learning from human feedback (RLHF), in which reward signals guide model training. Conventional approaches reduce multi-dimensional human judgment to scalar scores or pairwise comparisons, collapsing rich preference structure into opaque parametric proxies. This opacity creates vulnerabilities to reward hacking and makes evaluation biases difficult to detect and correct.

The authors propose Auto-Rubric as Reward (ARR), which reframes reward modeling as explicit criteria decomposition. Rather than optimizing implicit weights, ARR externalizes a vision-language model's internalized preference knowledge as prompt-specific rubrics before any pairwise comparison occurs. Each rubric translates holistic intent into independently verifiable quality dimensions—a factorized interface that makes preference structure inspectable and interpretable. This externalization reportedly suppresses evaluation biases including positional bias, and permits zero-shot deployment or few-shot conditioning on minimal supervision.

To integrate structured evaluation into model training, the authors introduce Rubric Policy Optimization (RPO), which distills ARR's multi-dimensional evaluation into a robust binary reward signal. Rather than opaque scalar regression, RPO uses rubric-conditioned preference decisions to stabilize policy gradients during generative training.

On benchmarks covering text-to-image generation and image editing, ARR-RPO reportedly outperforms pairwise reward models and VLM-based judges. The authors conclude that explicit externalization of implicit preference knowledge into structured rubrics achieves more reliable, data-efficient alignment, and that the bottleneck is architectural—the absence of a factorized interface—rather than insufficient preference knowledge.