ArXiv

Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data

Authors
Ye Wang, Pei Lin, Xiong-Hui Chen...
Categories
cs.RO
arXiv
https://arxiv.org/abs/2608.02580v1
PDF
https://arxiv.org/pdf/2608.02580v1

Brief

Ego2Robot presents a pipeline to synthesize large-scale robot manipulation training data from egocentric human videos using action retargeting, robot-arm visual synthesis, and multi-level curation. The release contains 18,561 hours across 15 robot morphologies and supports in-the-wild sources. Extended evaluation (RoboTwin2.0 with disentangled perturbations) shows joint pretraining improves OOD generalization and transfers to real-robot tests.

Why it matters

Ego2Robot is a scalable pipeline that converts egocentric human manipulation videos into robot training data via action retargeting, robot-arm visual synthesis, and multi-level quality curation, producing 18,561 hours of robot-format data spanning 15 robot morphologies and supporting both curated datasets and in-the-wild videos.

Key details

  • The authors extended RoboTwin2.0 with disentangled perturbation axes (visual appearance, scene layout, embodiment morphology, task semantics) and show that joint pretraining on Ego2Robot-synthesized plus real robot data consistently improves out-of-distribution generalization across these perturbation types, with benefits validated on real-robot deployment.
  • Ego2Robot constitutes the largest ego-to-robot dataset to date; project materials and examples are available at https://www-ye.github.io/ego2robot_blog/.
Source evidence

Abstract

Learning generalizable robot manipulation policies requires large-scale and diverse demonstration data. Egocentric human manipulation videos offer rich scene and task diversity, and prior work has shown that retargeting and rendering such videos into robot-format data can yield effective per-task policies at small scale. However, whether this approach can provide pretraining benefits for vision-language-action models at scale remains unexplored. We present \textbf{Ego2Robot}, a scalable pipeline that converts egocentric human manipulation videos into robot training data through action retargeting, robot-arm visual synthesis, and multi-level quality curation. Ego2Robot supports both curated datasets and in-the-wild videos, producing 18,561 hours of robot training data spanning 15 robot morphologies, making it the largest ego-to-robot dataset to date. To evaluate generalization, we extend RoboTwin2.0 with disentangled perturbation axes covering visual appearance, scene layout, embodiment morphology, and task semantics. Experiments show that joint pretraining on Ego2Robot-synthesized and robot data consistently improves out-of-distribution generalization across multiple perturbation types, with benefits validated on real-robot deployment. Project page: https://www-ye.github.io/ego2robot_blog/