A new v0 robotics benchmark by independent researcher that tests robot foundation models on four tasks:
Requiring spatial reasoning, geometric understanding, occlusion handling, and visuospatial planning.
Using a low-cost open-source SO-101 robotic arm.
The tasks progress in difficulty from placing objects in a bin to precise “next to” and “between” placements, and directional movements.
Evaluated on seen objects, partially novel classes, and fully unseen objects after fine-tuning GR00T-N1.6-3B on 4,400 real-world demos.
Real-world results show high success on basic tasks with familiar objects but sharp drops under complex constraints or novelty.
Thanks for sharing, Truman Hickok!
The project includes a matching Isaac Lab simulation suite, released code/datasets.
Btw., Truman is an aspiring full-stack roboticist looking for a job in the Bay Area. Reach out to him!
📍The current code (github.com/5hadytru/so101_be…) and datasets (huggingface.co/5hadytru) are released.
Video
Truman (@h1ckok)
Excited to release v0 of SO-101 Bench, a benchmark with 4 tasks designed to measure various core capabilities of robot foundation models. It includes precise spatial and geometric constraints, novel/unusual objects, temporarily occluded objects, and often requires visuospatial planning. It is presented here as a real-world case study, then a scalable real2sim evaluation suite, all executed on the cheap and open-sourced SO-101 arm!
— https://nitter.net/h1ckok/status/2063327505440821667#m