PleIAs/CommonLingua · Hugging Face

CommonLingua is a compact (2.35M-parameter) byte-level language identification model released 2026-04-28 and trained on 2,482,568 paragraphs drawn from Structured Wikipedia and the PleIAs Common Corpus, with explicit focus on OCR-noisy realistic documents and long-tail coverage including 61 African languages. The model operates on raw UTF-8 bytes (512-byte padded inputs), uses a 4096-bucket polynomial trigram-hash embedding, three causal Conv1D layers and a single bidirectional attention layer with RoPE; ablations report the trigram signal improved macro F1 by +1.2 points. On the CommonLID benchmark (376k held-out paragraphs, 200+ languages) CommonLingua achieves strict accuracy 77.63%, equivalent accuracy 82.92% and macro F1 0.7879, a 11.5-point macro F1 gain over the next-best baseline. The authors publish the training split under open licenses and provide a predict.py example and throughput numbers (H100 bf16 ≈ 26.2k texts/sec at bs=4096).