ParVL: Parallel Scaling and Expandable Compute Allocation for Multimodal LLMs

2026-08-04 · 17:59 UTC ·Yang Yang, Qinyu Zhao, Mouxiang Chen... ·1 min read

Brief

ParVL addresses inefficient scaling and rigid vision–language compute splits in multimodal LLMs by scaling parallel computation: it reuses a single ViT and LLM backbone across multiple branches and adds branch-specific prefix parameters. The authors train the model end-to-end (supervised fine-tuning on ~13B tokens) and systematically study how to allocate extra shared-backbone computation between ViT encoder and LLM decoder, reporting improved multimodal performance over single-branch baselines and task-dependent optimal allocations.

Why it matters

ParVL (Parallel Vision-Language) reuses existing ViT and LLM backbone parameters across multiple vision and language branches, instantiating each parallel stream with branch-specific prefix parameters and training end-to-end via full-parameter supervised fine-tuning on roughly 13B tokens (Yang Yang et al., arXiv 2026-08-04).

Key details

ParVL yields higher overall multimodal performance than same‑recipe single‑branch baselines, and the optimal allocation of additional shared-backbone computation between the ViT encoder and LLM decoder differs by task; code is available at https://github.com/YangYangGirl/ParVL.

Abstract

Existing scaling strategies for Multimodal Large Language Models (MLLMs) typically expand either model parameters or sequential inference computation, incurring substantial memory or latency overhead. More importantly, most existing methods fail to alter the rigid, fixed computation allocation between the Vision Transformer and the Large Language Model components, limiting task-specific optimization. To address this, we introduce the Parallel Vision-Language (ParVL) scaling framework for MLLMs, which scales parallel computation by reusing the existing ViT and LLM backbone parameters across multiple vision and language branches. This framework raises a central question: given a fixed backbone parameter budget, how should additional shared-backbone computation be allocated between the vision and language modalities? We instantiate each parallel computational stream with branch-specific prefix parameters over a shared backbone, and train the entire model end-to-end via full-parameter supervised fine-tuning on roughly 13B tokens. We systematically study the computation-allocation trade-off between the ViT encoder and LLM decoder. ParVL improves overall multimodal performance over same-recipe single-branch baselines, and the best evaluated vision--language allocation varies across tasks. Code is available at https://github.com/YangYangGirl/ParVL.

Comment: 14 pages, 4 figures