Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models

2026-06-25 · 17:59 UTC ·Shravan Venkatraman, Ritesh Thawkar, Omkar Thawakar... ·1 min read

Brief

VISE introduces an unsupervised self-evolving method to fix visual under-conditioning in large multimodal models by enforcing visual attention through geometric invariance (consistency under spatial transforms) and semantic invariance (detecting absence of evidence when regions are perturbed). Trained on raw unlabeled images without specialist roles or external reward models, VISE substantially improves captioning (±CIDEr gains shown) and reduces hallucination across 18 benchmarks, evaluated with Qwen3-VL-2B and multiple model families (ECCV 2026).

Why it matters

VISE (Visual Invariance Self-Evolution) is a purely unsupervised, single-model self-evolving LMM framework that targets 'visual under-conditioning' by directly regularizing visual conditioning with two invariance-based rewards: a geometric invariance reward (spatial consistency under known transformations) and a semantic invariance reward (penalizes evidence-agnostic generation when predicted regions are perturbed).

Key details

On 18 benchmarks using Qwen3-VL-2B as the base model, VISE yields large gains: +16.85 CIDEr on COCO and +19.66 CIDEr on TextCaps, reduces object hallucination by 5.0 Chair-I points, and generalizes across four model families and scales; code and models are available at https://mbzuai-oryx.github.io/VISE (ECCV 2026).

Abstract

Recently, self-evolving large multimodal models (LMMs) have received attention for improving visual reasoning in a purely unsupervised setting. However, multi-role self-play and self-consistency reward schemes in existing self-evolving LMMs optimize answer agreement without ensuring the decoder attends to visual content, relying instead on statistical language priors to produce self consistent outputs. This leads to a persistent failure mode we term visual under-conditioning, where the decoder relies on language priors rather than the image during generation, manifesting as insufficient attention to visual tokens. As a result, current self-evolving LMMs struggle on vision--language understanding tasks such as image captioning and visual question answering. To address this, we propose VISE (Visual Invariance Self-Evolution), a purely unsupervised self-evolving framework that directly regularizes the model's visual conditioning policy through two complementary invariance-based rewards: a geometric invariance reward that enforces spatial consistency under known transformations, and a semantic invariance reward that penalizes evidence-agnostic generation by requiring the model to recognize the absence of evidence when predicted regions are perturbed. VISE operates within a single model without specialist roles, external reward models, or annotations, and is trained on raw unlabeled images. Experiments on 18 benchmarks demonstrate the efficacy of our approach. Using Qwen3-VL-2B as the base model, VISE achieves gains of $+16.85$ CIDEr on COCO and $+19.66$ CIDEr on TextCaps, reduces object hallucination by $5.0$ Chair-I points, and generalizes across four model families and scales. Our code and models are available at https://mbzuai-oryx.github.io/VISE

Comment: ECCV 2026