ArXiv

Poli-Bias: Understanding and Measuring Large Language Model Biases in International Political Conflicts

2026-08-06 · 14:54 UTC ·Massi-Nissa Abboud, Aladin Djuhera, Elena Cabrio... ·1 min read
Authors
Massi-Nissa Abboud, Aladin Djuhera, Elena Cabrio...
Categories
cs.AI, cs.CL
arXiv
https://arxiv.org/abs/2608.06123v1
PDF
https://arxiv.org/pdf/2608.06123v1

Brief

Poli-Bias presents a counterfactual auditing framework that detects political bias in LLMs by swapping country identities in legally equivalent conflict prompts. It decomposes response differences into five interpretable dimensions and, across 13 models, shows systematic effects of target country and user affiliation on framing, legal evaluation, and defenses—exposing sycophancy and unevenness. (Only abstract was available.)

Why it matters

Poli-Bias introduces a counterfactual framework that compares LLM responses to paired prompts where country identities are systematically swapped across legally equivalent international-conflict scenarios, and decomposes response disparities into five interpretable dimensions.

Key details

  • Evaluated on 13 contemporary LLMs spanning diverse families and sizes, the study finds that country identities and user affiliations systematically change how equivalent actions are described, evaluated, and defended under international law, revealing sycophancy and uneven treatment.
  • Paper by Massi‑Nissa Abboud, Aladin Djuhera, Elena Cabrio, and Holger Boche; arXiv:2608.06123v1 (published 2026-08-06). Only the abstract was available for this summary; full paper PDF is linked on arXiv.
Source evidence

Abstract

Measuring political bias in large language models (LLMs) remains challenging as it can manifest through subtle differences in framing, argumentation, and legal reasoning that are difficult to capture with a single metric. In this work, we introduce Poli-Bias, a counterfactual framework for measuring whether LLMs treat legally equivalent conflict scenarios differently depending on the countries involved. Poli-Bias compares responses to paired prompts in which country identities are systematically swapped across diverse geopolitical relationships, legal violations, and reasoning tasks. Rather than reducing bias to a single judgment, our framework decomposes response disparities into five interpretable dimensions, revealing how and where unequal treatment manifests. Across 13 contemporary LLMs spanning diverse model families and sizes, we find that country identities and user affiliations can systematically affect how equivalent actions are described, evaluated, and defended under international law. Our results thus establish Poli-Bias as a fine-grained framework for auditing political even-handedness and sycophancy in LLMs.