ArXiv

LabRobFail: A Benchmark for Robotic Failure Analysis in Chemical Self-driving Laboratories

2026-07-26 · 14:56 UTC ·Haobo Wang, Baoli Sun, Anqi Zou... ·1 min read
Authors
Haobo Wang, Baoli Sun, Anqi Zou...
Categories
cs.RO, cs.CV
arXiv
https://arxiv.org/abs/2607.23704v1
PDF
https://arxiv.org/pdf/2607.23704v1

Brief

LabRobFail is a failure-centric benchmark and dataset for robotic failure analysis in chemical self-driving laboratories. Based on the abstract, it uses LabRobFail-Sim to inject controllable control/physics/semantic failures and builds LabRobFail-Data (>20k trajectories, 70+ scenarios, 5 categories, 11 types). LabRobFail-Bench measures six evaluation capabilities, and LabRobFail-VLM produces structured diagnoses, achieving 92.58% detection and 85.58% temporal localization while improving task success by 10–20 percentage points.

Why it matters

LabRobFail provides LabRobFail-Data with over 20,000 trajectories across 70+ task scenarios, organized into five failure categories and 11 fine-grained failure types; failures are injected via LabRobFail-Sim at control, physics, and semantic levels.

Key details

  • LabRobFail-Bench evaluates six capabilities—task understanding, failure detection, temporal localization, severity assessment, failure classification, and actionable correction—enabling fine-grained benchmarking of robotic failure analysis in chemical self-driving labs.
  • The domain-specialized LabRobFail-VLM achieves 92.58% failure-detection accuracy and 85.58% temporal-localization accuracy on seen environments, outperforms general-purpose VLMs, and improves downstream VLA task success rates by 10–20 percentage points; code and data are on GitHub.
Source evidence

Abstract

The deployment of embodied agents in self-driving laboratories could accelerate scientific discovery, yet their reliability is constrained by the irreversible and safety-critical nature of chemical experiments. Progress is further hindered by scarce failure data and the lack of fine-grained evaluation protocols. To address these challenges, we introduce LabRobFail, a failure-centric framework for learning and evaluating robotic failure analysis in chemical laboratories. LabRobFail-Sim injects controllable failures at the control, physics, and semantic levels, enabling the construction of LabRobFail-Data, which contains over 20,000 trajectories across 70+ task scenarios, five failure categories, and 11 fine-grained failure types. LabRobFail-Bench evaluates six capabilities spanning task understanding, failure detection, temporal localization, severity assessment, failure classification, and actionable correction. We further develop LabRobFail-VLM, a domain-specialized vision-language model that generates structured failure diagnoses and recovery instructions. On seen environments, it achieves 92.58% failure-detection accuracy and 85.58% temporal-localization accuracy, substantially outperforming general-purpose VLMs. When integrated as a real-time supervisor, it improves downstream VLA task success rates by 10-20 percentage points, demonstrating the value of fine-grained failure understanding for closed-loop recovery and reliable laboratory autonomy. Our code and data are available at https://github.com/Su-ISE-2001/SciRobo

Comment: Under review. Haobo Wang and Baoli Sun contributed equally. Code and data: https://github.com/Su-ISE-2001/SciRobo