ArXiv

ArmnetBench v0.1: Parallel Real-World Evaluation of Manipulation Policies on a Low-Cost Arm Farm

2026-07-27 · 14:16 UTC ·Praveen Selvaraj, Lorenzo Uttini, Ville Kuosmanen ·1 min read
Authors
Praveen Selvaraj, Lorenzo Uttini, Ville Kuosmanen
Categories
cs.RO
arXiv
https://arxiv.org/abs/2607.24481v1
PDF
https://arxiv.org/pdf/2607.24481v1

Brief

ArmnetBench v0.1 tackles the real-world evaluation bottleneck by validating an arm farm of low-cost SO-101 cells to benchmark seven manipulation policies on twelve tasks. Policies were trained or fine-tuned with 50 demonstrations per task; the study produced 2,518 policy rollouts plus 600 reference demonstrations (3,118 episodes), all human-scored with three-way labels. The labeled dataset and a leaderboard are publicly released.

Why it matters

ArmnetBench v0.1 runs end-to-end on a fleet of low-cost SO-101 cells under light on-site supervision and compares 7 manipulation policies across 12 tasks; each policy is trained or fine-tuned on 50 demonstrations per task.

Key details

  • The benchmark collected 2,518 policy rollouts and 600 reference demonstrations (3,118 episodes total); every episode is human-scored into three labels (successful, suboptimal, failure), while demonstrations are successful by construction.
  • The authors release the 3,118 core episodes in LeRobot v3.0 and RoboMeter formats, provide an initial leaderboard under a shared budget, and publish the data at https://huggingface.co/collections/armnet/armnetbench-v01.
Source evidence

Abstract

Real-world evaluation is a bottleneck in developing generalist robot manipulation policies. Each rollout requires physical hardware and an operator to set up, reset, and score it. We introduce ArmnetBench v0.1, a benchmark run on a fleet of low-cost SO-101 cells under light on-site supervision. v0.1 validates this arm farm end to end and compares 7 policies across 12 tasks with both single-arm and bimanual configurations. Each policy is trained or fine-tuned on 50 demonstrations per task; the benchmark contains 2,518 policy rollouts and 600 reference demonstrations. All 3,118 episodes carry a three-way label (successful, suboptimal, or failure). Policy rollouts are human-scored, while demonstrations are successful by construction. Beyond evaluation, its quality-labelled trajectories support downstream learning, from reward and predictive world models to policies trained on mixed-quality data. The leaderboard is an initial comparison under this shared budget. We release the 3,118 core episodes in LeRobot v3.0 and RoboMeter formats.

Comment: 11 pages, 6 tables, 3 figures. data available at https://huggingface.co/collections/armnet/armnetbench-v01