ArXiv

Beyond Monotonic Progress: Retry-Supervised Value Learning for Robot Imitation

2026-06-23 · 14:27 UTC ·Xinyao Qin, Junjie Lu, Kaixin Wang... ·1 min read
Authors
Xinyao Qin, Junjie Lu, Kaixin Wang...
Categories
cs.RO
arXiv
https://arxiv.org/abs/2606.24633v1
PDF
https://arxiv.org/pdf/2606.24633v1

Brief

ReTVL targets robot imitation from demonstrations that include mistakes and retries by using retry events as sparse supervision to learn value functions sensitive to local errors. It fuses global monotonic progress calibration with local pairwise preferences around retry keypoints, then reweights demonstration segments for behavior cloning. Real-robot experiments show finer-grained values and improved imitation versus progress-based baselines.

Why it matters

ReTVL (ReTry-Supervised Value Learning), introduced by Xinyao Qin et al. and published on arXiv 2026-06-23 (arXiv:2606.24633v1), learns mistake-sensitive value functions from mixed-quality robot demonstrations by leveraging sparsely annotated retry events.

Key details

  • The method combines global progress calibration with local pairwise preference learning around annotated retry keypoints to capture local degradation-and-recovery structure; the learned value model reweights demonstration chunks for downstream behavior cloning to downweight harmful errors while preserving corrective behaviors.
  • On real-robot manipulation tasks, ReTVL yields more fine-grained value estimates than progress-based baselines and improves imitation learning from imperfect demonstrations (results reported in the paper; PDF: https://arxiv.org/pdf/2606.24633v1).
Source evidence

Abstract

Human demonstrations for robot imitation learning often contain mistakes and corrective behaviors, such as imprecise grasps, object misalignment, unstable contact, and repeated attempts. While these segments are commonly treated as noisy or suboptimal data, they provide valuable evidence about when execution deviates from a desirable path and how task feasibility can be restored. However, existing reward and value models often rely on monotonic progress assumptions, which capture coarse task advancement but may overlook local execution errors and corrective behaviors in imperfect demonstrations. In this work, we propose ReTVL (ReTry-Supervised Value Learning), a framework for learning mistake-sensitive value functions from mixed-quality robot demonstrations by leveraging retry events as sparse supervision. ReTVL captures the local degradation-and-recovery structure around mistakes by combining global progress calibration with local pairwise preference learning induced by sparsely annotated retry keypoints. The learned value model is then used to reweight demonstration chunks for downstream behavior cloning, reducing the influence of harmful execution errors while preserving useful corrective behaviors. Experiments on real-robot manipulation tasks show that ReTVL produces more fine-grained value estimates than progress-based baselines and improves imitation learning from imperfect demonstrations.