ArXiv

Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

2026-08-05 · 16:09 UTC ·Junlin Han, Shengbang Tong, David Fan... ·1 min read
Authors
Junlin Han, Shengbang Tong, David Fan...
Categories
cs.CV, cs.LG, cs.MM
arXiv
https://arxiv.org/abs/2608.05000v1
PDF
https://arxiv.org/pdf/2608.05000v1

Brief

Multimodal pretraining is examined through controlled experiments on synthetic and large-scale real-world data to map how modalities interact. The paper identifies four principles—Knowledge Flow, Synergy vs. Competition, Early Unification, and practical Recipes—demonstrating that shared attention/normalization with modality-specific FF layers yields synergy, early joint training avoids a 'vision laziness' bias, and that efficient recipes can cut compute to ~5%. Results are validated by training multiple 13.5B MoE models on 2T tokens.

Why it matters

The authors isolate four core phenomena in unified multimodal pretraining—Knowledge Flow, Synergy vs. Competition, Early Unification, and Recipes—showing distinct, asymmetric knowledge transfer between language, visual understanding, and visual generation.

Key details

  • Architectural choices that promote modality synergy include shared attention and shared normalization paired with modality-specific feed-forward layers; early joint training outperforms late alignment and uncovers a 'vision laziness' effect where delayed visual integration makes models over-rely on language priors.
  • They derive efficient pretraining recipes that achieve strong generative performance using only ~5% of typical compute, and validate findings at scale by training multiple 13.5B Mixture-of-Experts models on 2T tokens (paper published 2026-08-05 by Junlin Han et al.).
Source evidence

Abstract

Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, the design space and the fundamental mechanisms of how modalities interact during unified training remain underexplored. We provide empirical clarity through a systematic exploration of multimodal pretraining. Our controlled experiments on both synthetic and large-scale real-world datasets yield four key insights into the physics of multimodal pretraining: (i) Knowledge Flow: We disentangle how language, visual understanding, and visual generation transfer knowledge across modalities, revealing distinct patterns of influence and asymmetry; (ii) Synergy vs. Competition: We show that data "complexity" largely determines whether modalities are synergistic, identify architectural choices that promote synergy: such as shared attention and normalization with modality-specific feed-forward layers, and find that these behaviors generalize across different visual tokenizer designs; (iii) Early Unification: Unifying modalities from the very early stages and training them jointly is shown to be more effective than late alignment or sequential training. This process uncovers a vision laziness phenomenon, where delayed integration leads models to rely on language priors; (iv) Recipes: We derive efficient pretraining recipes that achieve strong generative performance using only 5% of the compute budget. These core findings are subsequently validated at scale by training multiple 13.5B MoE models on 2T tokens. We hope this study provides a principled foundation for understanding and scaling multimodal pretraining.

Comment: Project page: https://junlinhan.github.io/projects/physicsofmm_pretrain/