ArXiv

RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction

2026-08-06 · 17:24 UTC ·Chenglong Wang, Ziming Zhu, Yifu Huo... ·1 min read
Authors
Chenglong Wang, Ziming Zhu, Yifu Huo...
Categories
cs.LG, cs.CL
arXiv
https://arxiv.org/abs/2608.06310v1
PDF
https://arxiv.org/pdf/2608.06310v1

Brief

The paper tackles the gap preventing generative reward models from powering RL by proposing RRC, which derives scalar learning signals from relative preference rankings. RRC combines self-competitive and anchor-guided ranking to turn ranking strengths into usable rewards. Experiments on open-ended chat and reasoning benchmarks show consistent improvements over existing reward-construction approaches; code is released on GitHub.

Why it matters

Chenglong Wang et al. (arXiv 2026-08-06) introduce RRC, a Ranking-based Reward Construction that converts generative reward models' comparative outputs into scalar RL rewards, addressing the mismatch between comparative generative reward modeling and scalar scoring used by standard RL algorithms.

Key details

  • RRC implements two strategies—self-competitive ranking (comparisons among sampled responses) and anchor-guided ranking (scalable ranking using a small set of reference responses)—and yields substantial, consistent gains on open-ended chat and reasoning benchmarks versus prior reward-construction methods; code: https://github.com/wangclnlp/RRC.
Source evidence

Abstract

Recent advances in reward modeling show a paradigm shift from discriminative reward models to generative reward models. However, despite their strong capabilities in response ranking, generative reward models have not realized their potential in reinforcement learning (RL). Our analysis reveals that this limitation arises from a mismatch between the comparative nature of generative reward modeling and the scalar scoring paradigm adopted by existing RL algorithms. To bridge this gap, we propose a Ranking-based Reward Construction (RRC) approach, which enables generative reward models to provide more effective RL learning signals by deriving rewards from relative preference rankings. RRC introduces two complementary strategies: self-competitive ranking, which exploits comparisons among sampled responses, and anchor-guided ranking, which enables scalable ranking-based reward construction with a small set of reference responses. Experiments across open-ended chat and reasoning benchmarks demonstrate that RRC substantially improves RL training with generative reward models, achieving consistent gains over existing reward construction approaches. Our code can be found at https://github.com/wangclnlp/RRC.