ArXiv

LangMAP: A Language-Adaptive Approach to Tokenization

2026-06-22 · 16:32 UTC ·Clara Meister, Suchir Salhan, Andrzej Szablewski... ·1 min read
Authors
Clara Meister, Suchir Salhan, Andrzej Szablewski...
Categories
cs.CL
arXiv
https://arxiv.org/abs/2606.23566v1
PDF
https://arxiv.org/pdf/2606.23566v1

Brief

LangMAP adapts UnigramLM to the multilingual setting to produce language-specific tokenizations from a single shared vocabulary, addressing the cost of training new models or changing vocabularies. Evaluated on 14 tokenizers (9 natural, 9 programming languages), it improves morphological and AST-leaf alignment; fine-tuning gains are strong for grammatical acceptability (MultiBLiMP) but mixed on knowledge tasks. (Abstract only; full text not provided.)

Why it matters

LangMAP (Language-adaptive MAP Tokenization) extends the UnigramLM algorithm to produce language-specific tokenizations from a single shared vocabulary, allowing adaptation of a pretrained model's tokenizer without changing its vocabulary; it requires language labels during training but performs language-specific tokenization at inference without input-language metadata.

Key details

  • Empirical evaluation used 14 open-source tokenizers across 9 natural languages and 9 programming languages: LangMAP improved morphological boundary alignment for natural languages and—for all 9 coding languages tested—improved alignment with AST leaf boundaries.
  • In fine-tuning experiments LangMAP improved target-language grammatical acceptability measured by MultiBLiMP on the languages tested, but yielded less consistent gains on knowledge-related tasks such as Global-PIQA and Belebele.
Source evidence

Abstract

Language-specific tokenizers improve tokenization quality and the downstream performance of models on those languages. However, using such a tokenizer comes at a cost: either a new model must be trained from scratch, or the vocabulary of an existing pretrained model must be adapted. We propose Language-adaptive Maximum a Posteriori (LangMAP) Tokenization, a tokenization scheme that extends the UnigramLM algorithm to the multilingual setting, producing language-specific tokenization from a single shared vocabulary. Notably, LangMAP can be used when training a multilingual language model from scratch or to adapt a pretrained model's tokenizer to individual languages without changing its vocabulary. While language labels are required at training time, a key feature of the algorithm is that it then performs language-specific tokenization at inference without knowledge of the input's language. Across 14 open-source tokenizers, 9 natural languages, and 9 programming languages, LangMAP improves morphological boundary alignment and, for all coding languages tested, alignment with abstract syntax tree (AST) leaf boundaries. In fine-tuning experiments, results are mixed: LangMAP improves target-language grammatical acceptability (MultiBLiMP) on the languages tested; its benefits are less consistent on knowledge-related tasks (Global-PIQA, Belebele).