Researchers propose RIMS, a preference optimization framework for small-scale LLM retrieval-augmented generation

Researchers from undisclosed institutions propose RIMS, a three-stage preference optimization framework for small-scale language models (SLMs) in retrieval-augmented generation (RAG) settings. The method addresses the sensitivity of SLMs to noisy or spurious retrieved evidence by replacing hard selection of preference pairs with a differentiable soft aggregation mechanism. This approach preserves gradient signal from all preference pairs while retaining the discriminative structure of margin-aware selection.

The framework consists of three stages: (1) synthetic chain-of-thought preference data generation via rejection sampling using the target SLM itself, eliminating reliance on proprietary models; (2) a differentiable soft aggregation mechanism that replaces hard argmin/argmax selection with a smooth operator; and (3) preference optimization with the smoothed objective applied to multiple alignment algorithms.

The authors provide theoretical analysis showing that the smoothed approximation admits a controllable error bound and that smooth aggregation yields provably tighter gradient alignment to the oracle objective than hard selection. In experiments on four multi-hop question answering benchmarks, RIMS outperforms state-of-the-art baselines across multiple SLM backbones, achieving consistent gains in Exact Match and F1 under noisy retrieval conditions.

The implementation and associated resources are available on GitHub under the repository https://github.com/tptrix29/RIMS.