ArXiv

Should Missing Modalities Always Be Necessary to Repair for Multi-modal Sentiment Analysis?

Authors
Yubo Gao, Haotian Wu, Xiaoyu Xu...
Categories
cs.CL
arXiv
https://arxiv.org/abs/2607.17262v1
PDF
https://arxiv.org/pdf/2607.17262v1

Brief

Multimodal sentiment analysis under missing modalities: the authors show via per-sample oracle analysis that repairing every missing modality is not universally beneficial. They introduce SIEVE, a repair-agnostic dual-branch method that routes each input through either a direct-prediction or repair branch using a loss-gap sufficiency signal and an evidential gate for uncertainty. Experiments on CMU-MOSI and IEMOCAP report consistent improvements across missing rates; only the abstract was available.

Why it matters

Per-sample oracle analysis shows full-modality input is optimal for only a small fraction of samples and every modality subset is preferred by some samples, contradicting the common "repair-first" assumption in multimodal sentiment analysis.

Key details

  • SIEVE (Sufficiency-Informed Evidential ValvE) makes a learnable, per-sample decision to repair by comparing a direct-prediction branch with a repair branch, using the per-sample loss gap as an empirical sufficiency signal and an evidential gate that models sufficiency plus epistemic uncertainty; on CMU-MOSI and IEMOCAP (paper posted 2026-07-19) it consistently improves representative repair backbones and approaches the per-sample dual-branch achievable optimum.
Source evidence

Abstract

Existing methods for multimodal sentiment analysis (MSA) under missing modalities usually follow a repair-first paradigm. We revisit this assumption and ask: \emph{should every missing modality be repaired?} A per-sample oracle analysis shows the answer is not always: full-modality input is optimal for only a small fraction of samples, and every modality subset is preferred by some samples. These results suggest that adding or repairing modalities may not always improve prediction, and that the utility of each modality is sample-dependent. Building on this finding, we propose \textbf{S}ufficiency-\textbf{I}nformed \textbf{E}vidential \textbf{V}al\textbf{vE} (\textbf{SIEVE}) that turns ``whether to repair'' into an explicit, learnable decision at the sample level. SIEVE compares a direct prediction branch with a repair branch, derives an empirical sufficiency signal from their per-sample loss gap, and routes each input through an evidential gate that jointly models sufficiency and its epistemic uncertainty. SIEVE is repair-agnostic: it operates as a plug-and-play decision on top of any explicit or implicit repair module, without modifying its internal design. Experiments on CMU-MOSI and IEMOCAP show that SIEVE consistently improves representative repair backbones across evaluated missing rates, and approaches the per-sample dual-branch achievable optimum.