FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation

2026-06-24 · 16:23 UTC ·Shuyi Zhang, Yunfan Lou, Hongyang Cheng... ·1 min read

Brief

FORCE (Shuyi Zhang et al., 2026) tackles sample-inefficient RL fine-tuning for Vision-Language-Action models by stabilizing the Q-function and improving data quality. The 3-stage framework uses a Value-Calibrated Warm-Up with on-policy rollouts, then filters policy and expert proposals with the calibrated Q and applies self-distillation. On simulation and real-world benchmarks (abstract only provided), FORCE reports a 79% absolute success-rate gain, +10% vs prior RL, and 32.5% faster training, achieved without human interventions.

Why it matters

FORCE (Zhang et al., published 2026-06-24) proposes a 3-stage Vision-Language-Action (VLA) RL fine-tuning pipeline that addresses (1) catastrophic initial unlearning from an unstable Q-function and (2) low-quality exploration data by combining a Value-Calibrated Warm-Up (on-policy rollouts), online filtering of policy and expert actions via the calibrated Q, and self-distillation.

Key details

On simulation and real-world tasks the method yields large empirical gains: a 79% absolute improvement in success rates, a 10% relative improvement over prior RL baselines, and a 32.5% acceleration in training; importantly, these results were achieved without human intervention and while avoiding the typical success-rate drop.

Abstract

Vision-Language-Action (VLA) models are often constrained by the imitation ceiling imposed by sub-optimal data. While Reinforcement Learning (RL) fine-tuning can surpass this limit, it is notoriously sample inefficient. This challenge arises from two core issues: (1) catastrophic initial unlearning due to an unstable Q-function and (2) inefficient policy updates caused by low-quality exploration data, often forcing a reliance on costly human interventions. We introduce FORCE, a 3-stage framework that stabilizes fine-tuning by tackling both issues. FORCE first incorporates a Value-Calibrated Warm-Up phase, utilizing on-policy rollouts to mitigate the distributional shift of the Q-function. Subsequently, during the online stage, this calibrated Q-function acts as a filter for both the policy's own action proposals and expert data, ensuring only high-value actions are used for the policy update. We evaluate FORCE on various simulation and real-world tasks, and the result shows that FORCE achieves a 79% absolute improvement in success rates and outperform prior RL methods by 10%, while accelerating training by 32.5%. Critically, it mitigates the common success rate drop and achieves this robust performance without human intervention, marking a significant step towards deploying capable and autonomous robotic agents.