ArXiv

When Do Fewer Visual Tokens Accelerate Multimodal Inference? A Break-Even Study Across Decision Locations and Hardware

2026-08-04 · 13:33 UTC ·Hao Dou, Ruiwen Tian ·1 min read
Authors
Hao Dou, Ruiwen Tian
Categories
cs.CV
arXiv
https://arxiv.org/abs/2608.03649v1
PDF
https://arxiv.org/pdf/2608.03649v1

Brief

The paper develops a reproducible break-even protocol and stage-level decomposition to evaluate when reducing visual tokens lowers end-to-end latency, accounting for decision overhead, shared work, and operators avoided. Experiments (Qwen2.5‑VL‑3B‑Instruct) show autoregressive probes can be slower than Full; a lightweight post-vision predictor yields significant speedups on RTX 3090 and A100 after Holm correction, while a pre-vision image-size rule is nominally faster but not Holm-significant. Results are conditional on examples answered correctly by Full.

Why it matters

In a 30-example pilot using Qwen2.5‑VL‑3B‑Instruct, two tested autoregressive probes remained slower than Full inference on RTX 3090 and A100 despite state reuse.

Key details

  • A lightweight post-vision predictor produced paired confidence intervals below zero (speedup) on both RTX 3090 and A100 and remained significant after a conservative all-pairs Holm correction; a pre-vision image-size rule also had intervals below zero on both GPUs but lost significance after Holm correction.
  • Pre-vision routing can avoid preprocessing and vision encoding—on A100 this structural saving outweighed an almost eightfold larger downstream token reduction achieved by the post-vision policy; reported results are conditional on examples answered correctly by Full.
Source evidence

Abstract

Fewer visual tokens do not guarantee lower end-to-end latency. We evaluate break-even with a reproducible protocol that accounts for decision overhead, shared work, and the operators each policy can avoid. A stage-level decomposition reconciles these components with measured end-to-end latency. In a 30-example pilot, the two tested autoregressive probes remain slower than Full despite state reuse. A lightweight post-vision predictor yields paired confidence intervals below zero on RTX 3090 and A100 and remains significant after a conservative all-pairs Holm correction. A pre-vision image-size rule also yields intervals below zero on both GPUs, although neither comparison remains significant after the same correction. Pre-vision routing has a structural opportunity unavailable to post-vision pruning: it can avoid preprocessing and vision encoding. On A100, this opportunity outweighs a nearly eightfold larger downstream token reduction by the post-vision policy. Reported quality is conditional on examples answered correctly by Full and is not benchmark accuracy.

Comment: 16 pages, 3 figures, 13 tables. Experiments use Qwen2.5-VL-3B-Instruct on RTX 3090 and A100 PCIe GPUs