ArXiv

Any system that outputs one member model's answer is upper-bounded by accuracy ≤…

2026-06-25 · 17:06 UTC ·Josef Chen ·1 min read
Authors
Josef Chen
Categories
cs.AI, cs.LG
arXiv
https://arxiv.org/abs/2606.27288v1
PDF
https://arxiv.org/pdf/2606.27288v1

Brief

Chen shows a concrete "co-failure" ceiling for multi-model LLM systems: any policy that returns a single model's answer cannot exceed accuracy 1−β, where β is the rate all models fail on the same query. Using Clopper–Pearson finite-sample bounds and a 67-model empirical pool, the paper finds higher all-wrong tails than Gaussian-copula models predict (e.g., β=0.052 vs 0.023 on open math), and demonstrates that ensemble gains require heterogeneous failure patterns and query-level routing signals; full text not available, summary based on the abstract.

Why it matters

Any system that outputs one member model's answer is upper-bounded by accuracy ≤ 1 − β, where β is the rate that every model is wrong on the same query; a Clopper–Pearson bound on β gives a finite-sample certificate on the maximum possible gain from routing, voting, or cascades.

Key details

  • Empirical evaluation across 67 models from 21 providers: on open-ended mathematics observed β = 0.052 versus β = 0.023 under a 67-model Gaussian copula (≈2.5× underpricing; 90% CI 1.7–3.4, k = 17); execution-graded code showed β = 0.079.
  • Re-asking GPQA-Diamond questions in free-response increased co-failure (β = 0.127) with a five-judge panel κ = 0.73–0.92; combining models rarely outperforms the single best model on checkable tasks without strong query-level routing, and gains come from models failing on different questions (heterogeneity), not just adding more models.
Source evidence

Abstract

Multi-model LLM systems such as routing, voting, cascades, fusion, and mixture-of-agents are used to beat single-model accuracy. We show that their gain is capped by a quantity the field rarely reports. For any policy whose output is one member model answer, accuracy cannot exceed one minus beta, where beta is the rate at which every model is wrong on the same query. In contrast, the usual diagnostic, average pairwise error correlation rho, cannot identify beta: error laws with identical marginals and pairwise correlations can have different all-wrong rates. A Clopper-Pearson bound on beta gives a finite-sample certificate on the largest gain any router, vote, or cascade could deliver before training a router. Across 67 models from 21 providers, a tetrachoric-calibrated single-factor model still underprices the all-wrong tail: on open-ended mathematics, observed beta is 0.052 versus 0.023 under the full 67-model Gaussian copula, about 2.5 times underpricing, with 90 percent CI 1.7 to 3.4 and k equals 17. The effect recurs on execution-graded code, where beta is 0.079. Re-asking the same GPQA-Diamond questions in free-response rather than multiple-choice form reopens the tail, with beta 0.127 and a five-judge panel with kappa 0.73 to 0.92, locating co-failure in answer format rather than subject. At matched quality, low-rho heterogeneous ensembles beat high-rho Self-MoA, but on checkable tasks in our pool, combining models rarely beats the single best model without a strong query-level routing signal. Gains come from models failing on different questions, not from adding more models.