ArXiv

SceneActBench: Can Agents Act on the 3D Scenes They See?

2026-07-24 · 15:16 UTC ·Yifei Zhao, Xiangxin Zhou, Wenhao Yang... ·1 min read
Authors
Yifei Zhao, Xiangxin Zhou, Wenhao Yang...
Categories
cs.AI, cs.CV
arXiv
https://arxiv.org/abs/2607.22393v1
PDF
https://arxiv.org/pdf/2607.22393v1

Brief

SceneActBench introduces a benchmark for visually conditioned action on complete multi-object 3D scenes (five tasks, 210 source instances, 520 task cases) using a single fixed agent-environment loop. Agents receive images/video frames and optional 3D assets; final outputs are scored with task-specific geometric metrics against hidden ground truth. Eleven VLM configurations score 38.6–50.2 overall, revealing inconsistent cross-task performance and detailed failure-mode analysis. Full paper and PDF are available on arXiv.

Why it matters

SceneActBench is a new benchmark for agent actions on multi-object 3D scenes covering five tasks built from 210 source instances and 520 task cases, using PNG images or sampled video frames (and optional 3D assets) in a unified agent-environment loop and task-specific geometric metrics against hidden ground truth.

Key details

  • Evaluation across eleven proprietary vision-language model configurations yields Overall scores from 38.6 to 50.2, with no model performing consistently well across all tasks; the paper includes analyses of where and how failures occur.
Source evidence

Abstract

Vision-language model (VLM) agents increasingly use tools to act on 3D scenes rather than only describe them. Existing 3D benchmarks score textual responses or single-object operations, leaving agent action on complete multi-object 3D scenes under evaluated. We present SceneActBench, a benchmark for visually conditioned action across five 3D tasks under a unified agent-environment loop. Given PNG images or sampled video frames and, where applicable, supplied 3D assets, an agent acts on a 3D environment. We evaluate each final output against hidden ground truth with task-specific geometric metrics. SceneActBench comprises five tasks built from 210 source instances, yielding 520 task cases including paired input conditions. Every task runs through one fixed agent loop to keep the comparison fair. Across eleven proprietary VLM configurations, Overall scores span 38.6-50.2, and none performs consistently well across tasks. We further analyse where and how failures manifest.