ArXiv

Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models

2026-06-24 · 17:53 UTC ·Akshay Paruchuri, Sanmi Koyejo, Ehsan Adeli ·1 min read
Authors
Akshay Paruchuri, Sanmi Koyejo, Ehsan Adeli
Categories
cs.CL, cs.CV, cs.LG
arXiv
https://arxiv.org/abs/2606.26079v1
PDF
https://arxiv.org/pdf/2606.26079v1

Brief

Facet-Probe is a targeted audit that probes five ordering facets across 18 multimodal LLMs using a Bayesian item-response model and a same-ordering control. The study finds wide ordering sensitivity—panel-mean flip rates of 24–50% and a best-model flip rate of 13.4%—and shows prompt-only mitigations are modality-specific, motivating training or architectural solutions and proposing cross-ordering flip rate as a standard metric.

Why it matters

Facet-Probe: a five-facet audit (option, evidence-chunk, document-rank, image-set, mixed-modality ordering) was applied to 18 frontier and open-weight multimodal LLMs using a Bayesian item-response model to separate ordering noise from per-facet bias.

Key details

  • No model was order-invariant: screened per-facet panel-mean flip rates range 24–50%; even the best model still flips on 13.4% of trials, and a Gemini same-ordering control at temperature 0 shows a substantial ordering excess above decoder-noise.
  • Training-free prompt mitigations in Gemini are modality-conditional and do not transfer from text to visual reasoning, leading the authors to recommend cross-ordering flip rate as a standard reliability metric and to call for training-time or architectural fixes.
Source evidence

Abstract

Standard benchmarks for multimodal large language models (MLLMs) score each item on one canonical ordering and miss whether order-irrelevant shuffling changes the answer, a baseline reliability property called for by emerging AI evaluation guidelines. We introduce Facet-Probe, a five-facet audit (option, evidence-chunk, document-rank, image-set, and mixed-modality ordering) of 18 frontier and open-weight MLLMs. A Bayesian item-response model separates ordering noise from per-facet bias, and a same-ordering control estimates the decoder-stochastic floor for observed flips. We find that none of the 18 MLLMs we audit are order-invariant: screened per-facet panel-mean flip rates span 24-50%. A Gemini same-ordering control at temperature 0 estimates a substantial ordering excess over a same-input decoder-noise floor in verified cells. Capability predicts but does not eliminate flips; the best model still flips on 13.4% of trials. In our Gemini mitigation tests, training-free prompt changes are modality-conditional and do not transfer from text to visual reasoning. These results suggest that prompt-level mitigation alone is unlikely to provide general order robustness, motivating future work on training-time and architectural approaches. We propose cross-ordering flip rate as a standard reporting axis for MLLMs.

Comment: 22 pages, 4 figures, 5 tables