ArXiv

LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs

2026-08-06 · 13:52 UTC ·Lukas Twist, Twm Stone, Helen Yannakoudakis... ·1 min read
Authors
Lukas Twist, Twm Stone, Helen Yannakoudakis...
Categories
cs.SE, cs.CL
arXiv
https://arxiv.org/abs/2608.06041v1
PDF
https://arxiv.org/pdf/2608.06041v1

Brief

LangChoiceBench introduces a 28-project, seven-area benchmark to quantify LLM programming-language choice. Evaluating 25 models, the paper finds heavy Python over-selection, poor recommendation–implementation consistency, and stronger Python bias in smaller open-weight models. Analysis of 9,826 reasoning traces reveals mostly automatic/ease-driven choices and instances of "phantom evidence" and self-contradictory outputs.

Why it matters

LangChoiceBench is a project-level benchmark covering 28 projects across seven software areas (chosen where Python is often a poor default) for measuring Python preference, recommendation-implementation consistency, and language diversity.

Key details

  • The authors evaluated 25 diverse LLMs and found pervasive Python over-selection, low recommendation-to-implementation consistency, and that smaller open-weight models generally show stronger Python preference and lower language diversity.
  • Analysis of 9,826 reasoning traces shows most Python choices are automatic or ease-driven; the paper documents a failure mode called "phantom evidence" (fabricated contextual support) and cases where produced code contradicts the model's stated language choice.
Source evidence

Abstract

Large language models (LLMs) have been shown to exhibit strong Python preferences when generating project-level code, but there is currently no systematic way to measure this behaviour across new models. To bridge this gap, we introduce LangChoiceBench, a project-level code-generation benchmark for measuring Python preference, recommendation-implementation consistency, and language diversity. LangChoiceBench covers 28 projects across seven software areas where Python is often a poor default. We evaluate 25 diverse LLMs and find that Python remains heavily over-selected, recommendation-implementation consistency is low, and smaller open-weight models generally show stronger Python preference and lower language diversity. We further analyse 9,826 reasoning traces and find that most Python choices are automatic or driven primarily by ease, rather than explicit consideration of project requirements. In a smaller but important set of cases, models fabricate contextual support for choosing Python - a failure mode we call phantom evidence - or produce code that contradicts the language selected in their own reasoning.

Comment: 19 pages, 9 tables, 2 figures