ArXiv

The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images

2026-08-06 · 17:01 UTC ·Zhiheng Wang, Bo Peng, Lai Wei... ·1 min read
Authors
Zhiheng Wang, Bo Peng, Lai Wei...
Categories
cs.AI
arXiv
https://arxiv.org/abs/2608.06270v1
PDF
https://arxiv.org/pdf/2608.06270v1

Brief

Thinking-with-images multimodal LLMs are evaluated by formulating visual tool-use as a causal graph and applying interventions at policy, trajectory, and step levels. The step estimand, Visual Evidence Gain, isolates each returned observation's effect. Experiments on six models and five fine-grained perception benchmarks reveal two failure modes (Calling Without Looking; Looking Without Planning) and show aggregate accuracy gains are concentrated in a small calibrated subset, despite higher token costs.

Why it matters

Thinking-with-images was cast as a causal graph and audited with three intervention levels (policy, trajectory, step); the paper introduces a step-level estimand called Visual Evidence Gain to quantify the causal contribution of each returned visual observation (published 2026-08-06).

Key details

  • Across six representative models and five fine-grained perception benchmarks the authors diagnose policy miscalibration with two failure modes: 'Calling Without Looking' (returned observations have no causal effect) and 'Looking Without Planning' (observations are informative but call scheduling is incoherent).
  • They report that visual tool-use often incurs substantially higher token cost but yields only marginal or negative gains over direct inference, with aggregate accuracy improvements concentrated in a calibrated minority—coined the 'illusion of visual tool-use' (code: https://github.com/OpenCausaLab/CauAudit).
Source evidence

Abstract

The "thinking-with-images" paradigm equips multimodal LLMs with active visual operations such as crop-and-zoom. However, models using these operations often achieve only marginal or negative gains over direct inference at substantially higher token cost. They may also repeatedly crop irrelevant regions and fail on questions that direct inference answers correctly. We ask whether the returned visual evidence causally affects the answer. To answer this question, we formulate visual tool-use as a causal graph that separates observation-mediated paths from action-induced shortcuts. We then audit it through interventions at the three levels: policy (comparing tool-use with direct inference), trajectory (corrupting all observations during rollout), and step (counterfactually replacing one individual observation under a fixed prefix). Our step-level estimand, Visual Evidence Gain, isolates the contribution of each returned observation. Across six representative models and five fine-grained perception benchmarks, we uncover policy miscalibration with two failure modes. In Calling Without Looking, returned observations have no causal effect on the answer. In Looking Without Planning, observations are informative but the call schedule is incoherent. A trajectory-level diagnostic decomposes the policy-level accuracy gain and shows that the gain is concentrated in a Calibrated minority. We term this discrepancy the illusion of visual tool-use: despite aggregate accuracy gains, visual tool-use is not causally effective across a broad range of rollouts. The code is available at https://github.com/OpenCausaLab/CauAudit.