ArXiv

SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rollouts

2026-08-05 · 15:32 UTC ·Nhat Minh Pham, Duy Tung Doan, Thi Duyen Ngo... ·1 min read
Authors
Nhat Minh Pham, Duy Tung Doan, Thi Duyen Ngo...
Categories
cs.LG, cs.CL
arXiv
https://arxiv.org/abs/2608.04962v1
PDF
https://arxiv.org/pdf/2608.04962v1

Brief

SpecRoll addresses the RL post-training rollout bottleneck by combining fast lightweight future-token heads and a Reflex module that uses delayed verifier feedback for local hidden-state corrections, plus a slow-path parameter update when performance degrades. Evaluated on five models (1.5B–14B) and three math reasoning datasets, it yields up to 2.15x faster generation and outperforms FastGRPO (avg 1.18x end-to-end gain).

Why it matters

SpecRoll attains 1.26–2.15x generation speedup and 1.21–2.04x end-to-end speedup over vanilla GRPO across five models (1.5B–14B) and three mathematical reasoning datasets.

Key details

  • SpecRoll uses two-timescale adaptation: lightweight future-token heads generate parallel proposals while a Reflex module applies delayed verifier feedback to perform bounded, trajectory-local hidden-state corrections without backpropagation; a slow path updates head parameters when sustained degradation is detected.
  • SpecRoll outperforms FastGRPO on both generation and end-to-end time in all 15 matched settings (average pairwise end-to-end gain 1.18x), preserves the target rollout distribution and GRPO objective, and employs concurrency-aware sparse-tree plus exact target verification.
Source evidence

Abstract

Reinforcement learning (RL) post-training improves the reasoning capabilities of large language models, but autoregressive rollout generation remains a major efficiency bottleneck. Speculative decoding can accelerate generation, yet applying it during RL is difficult because the target policy continually evolves: static proposers become stale, while frequent drafter updates add substantial overhead. We introduce SpecRoll, a speculative rollout engine that preserves the target model's sampling distribution while adapting at two timescales. Lightweight future-token heads generate parallel proposals, while our proposed Reflex module uses delayed verifier feedback to perform bounded, trajectory-local hidden-state corrections without backpropagation. A complementary slow path updates the head parameters only when sustained degradation is detected. SpecRoll combines these mechanisms with concurrency-aware sparse-tree verification and exact target verification, leaving the target rollout distribution and GRPO objective unchanged. Across five models ranging from 1.5B to 14B and three mathematical reasoning datasets, SpecRoll achieves 1.26-2.15x generation speedup and 1.21-2.04x end-to-end speedup over vanilla GRPO. It also outperforms FastGRPO in both generation and end-to-end time across all 15 matched settings, with an average pairwise end-to-end gain of 1.18x. Controlled ablations show that the fast and slow adaptation paths provide complementary benefits. Our source code is available at https://anonymous.4open.science/r/SpecRoll-26062006.