ArXiv

dVLA-RL: Reinforcement Learning over Denoising Trajectories for Discrete Diffusion Vision-Language-Action Models

2026-06-22 · 17:19 UTC ·Yuhao Wu, Yitian Liu, Weijie Shen... ·1 min read
Authors
Yuhao Wu, Yitian Liu, Weijie Shen...
Categories
cs.RO
arXiv
https://arxiv.org/abs/2606.23623v1
PDF
https://arxiv.org/pdf/2606.23623v1

Brief

dVLA-RL (Wu et al., 2026) tackles intractable marginal-action probabilities in discrete diffusion VLA models by shifting the objective to the joint probability of denoising trajectories, treating the denoising chain as an MDP and optimizing step-wise transitions. Combined with a task-adaptive denoising-step scheduler, it reports 99.7% on LIBERO and +30.6% vs SFT on RoboTwin 2.0. Full text was not provided, summary based on the abstract.

Why it matters

dVLA-RL reforms discrete diffusion Vision-Language-Action (dVLA) training by optimizing the joint probability of the sampled denoising path (product of step-wise transitions), modeling denoising as an MDP to avoid the intractable marginal final-action probability and support variable denoising steps.

Key details

  • The method adds a unified step-scheduling scheme for multi-task learning (tailoring denoising steps to task complexity) and yields strong empirical gains: 99.7% success on LIBERO and a 30.6% improvement over the supervised fine-tuning (SFT) baseline on RoboTwin 2.0, while remaining competitive with World-Action Model baselines.
Source evidence

Abstract

Vision-Language-Action (VLA) models have established a powerful paradigm for generalist robotic manipulation by grounding control into the semantic reasoning of VLMs. Prevailing architectures typically model actions continuously via diffusion or flow processes, or discretely through either autoregressive generation or parallel decoding. Recently, Discrete Diffusion VLAs (dVLAs) have emerged as a distinct alternative, unifying vision, language, and action into a single discrete token space via masked generative modeling. While combining iterative refinement with unified representations, its training has thus far been restricted to Supervised Fine-Tuning (SFT), leaving the potential of Reinforcement Learning (RL) for further policy refinement largely unexplored. A fundamental challenge in RL for dVLAs is that the marginal probability of the final action generated by dVLAs remains intractable. To solve this problem, we propose \textbf{dVLA-RL}, shifting the learning objective from the marginal action probability to the joint probability of the sampled generation path. Specifically, by modeling the denoising process as a Markov Decision Process (MDP), we mathematically formulate this path probability as a product of step-wise transitions. This trajectory-level objective provides a unified formulation that natively accommodates variable denoising steps. Leveraging this intrinsic fexibility, we introduce a unified step scheduling approach for complex multi-task learning, tailoring denoising steps to specific task complexities to maximize both success rates and computational effciency. Extensive evaluations demonstrate that our approach achieves a success rate of \textbf{99.7\%} on LIBERO. Furthermore, it establishes strong VLA-based results on RoboTwin 2.0 by delivering a \textbf{30.6\%} improvement over the SFT baseline, remaining competitive with strong World-Action Model baselines.