ArXiv

OR Else: A Differentiable Trust Region for Policy Optimization

2026-07-20 · 17:07 UTC ·Chinmay Rane, Kanishka Tyagi, Michael Manry ·1 min read
Authors
Chinmay Rane, Kanishka Tyagi, Michael Manry
Categories
cs.LG, cs.AI
arXiv
https://arxiv.org/abs/2607.18163v1
PDF
https://arxiv.org/pdf/2607.18163v1

Brief

This paper evaluates Output Reset (OR), a smooth one-sided saturation replacement for PPO/GRPO clipping, applied to post-training RL on Llama-3.2-1B-Instruct using Anthropic hh-rlhf. PPO-OR (three seeds) outperforms PPO-clip by +0.305 mean training-time reward-model score under GAE but with increased seed variance. GRPO-OR (G=2) improves stability (smaller spread, near-zero terminal OR residual) without raising mean score. Both group-relative methods produce larger log-ratio displacement; OR alters optimization behavior but yields mixed reward effects. Only the abstract was available.

Why it matters

Under generalized advantage estimation (GAE) with Llama-3.2-1B-Instruct on Anthropic hh-rlhf (one shared reward model, three seeds), PPO-OR achieves a mean final training-time reward-model score 0.305 higher than PPO-clip, but with a larger across-seed spread (reported 2026-07-20).

Key details

  • With group-relative advantages and group size G=2, GRPO-OR does not increase mean reward-model score versus GRPO-clip but yields a smaller observed spread, a near-zero terminal OR residual, and a declining overshoot fraction; however diagnostics did not translate to a score gain at G=2.
  • Both group-relative methods (GRPO and GRPO-OR) show substantially larger rollout-to-current log-ratio displacement than the GAE methods, and Output Reset (OR) does not consistently reduce that displacement; reported metrics are training-time reward-model scores, not held-out human-preference evaluations.
Source evidence

Abstract

PPO and the GRPO baseline studied here use clipped surrogate objectives whose favorable-direction saturation introduces an abrupt change in the scalar objective's derivative. We ask whether Output Reset (OR), a smooth one-sided saturation rule, offers a useful alternative for large language model post-training. PPO-OR and GRPO-OR replace the clipped policy term with an OR squared-margin loss in rollout-relative token log-ratio space; the advantage sign determines the update direction, and a token contributes zero direct OR residual after crossing the favorable margin. We compare PPO-clip with PPO-OR under generalized advantage estimation (GAE), and GRPO with GRPO-OR under group-relative advantages, using \texttt{Llama-3.2-1B-Instruct} on Anthropic \texttt{hh-rlhf} with one shared reward model and three seeds per method. Under GAE, PPO-OR has a mean final training-time reward-model score $0.305$ higher than PPO-clip, with a larger observed across-seed spread. Under group-relative advantages, GRPO-OR does not have a higher mean score, but shows a smaller observed spread, a near-zero terminal OR residual, and a declining overshoot fraction, while the matched GRPO clipped-objective trace remains variable. Both group-relative methods exhibit substantially larger rollout-to-current log-ratio displacement than the GAE methods, and OR does not consistently reduce it. Thus, OR changes optimization behavior in both matched comparisons, but the observed reward effect differs between them. At $G=2$, the GRPO-OR diagnostics do not translate into a reward-score gain. Whether larger groups change this outcome remains open. The reported scores are training-time reward-model measurements, not held-out human-preference performance.

Comment: 22 pages, 5 figures