ArXiv

The paper adapts the QQ (quantum question) equality as an audit criterion for…

Authors
Pilsung Kang
Categories
cs.CL, cs.AI, quant-ph, stat.ME
arXiv
https://arxiv.org/abs/2607.17219v1
PDF
https://arxiv.org/pdf/2607.17219v1

Brief

Auditing question-order effects in LLMs using the QQ equality, Kang provides theory (mechanism characterizations, closed-form violation conditions, and the audit inequality |qQQ| ≤ OSS) and a pre-specified pipeline combining robustness envelopes, consistency checks, counterbalancing, and a saturation diagnostic. A pilot on an open-weight instruction-tuned model passed health gates but found pervasive saturation (17/18 and 7/8 pairs) and no residual contextuality, implying forced-binary next-token logits are insufficient for QQ audits.

Why it matters

The paper adapts the QQ (quantum question) equality as an audit criterion for sequential judgments by autoregressive LLMs and characterizes mechanisms that satisfy it: marginal-independent kernels satisfy QQ iff all four mismatch transition rates coincide (including a 2D rank-1 projective model), polarity- and position-dependent repetition families obey an exact cross-symmetry condition with closed-form violations, QQ behaviors are closed under order-matched mixing, and the rank-2 Contextuality-by-Default test translates to the audit bound |qQQ| ≤ OSS (order-sensitivity score).

Key details

  • The author develops a pre-specified, audit-logged pipeline (worst-case robustness envelopes, sampling-consistency spot checks, full label counterbalancing, saturation diagnostic) and pilots it on an open-weight instruction-tuned model: all health gates passed, but 17/18 and 7/8 item pairs (under two framings) were saturated (near-deterministic) and no item was certified residually contextual, so forced-binary next-token log-probabilities were inadequate for distribution-level QQ audits.
Source evidence

Abstract

Human survey respondents exhibit question-order effects that satisfy the QQ (quantum question) equality, an a priori, parameter-free prediction of the projective quantum question-order model. We develop the QQ equality into an audit criterion for sequential judgments of autoregressive large language models (LLMs). Theoretically, we characterize which mechanism classes satisfy it robustly: marginal-independent kernels satisfy QQ iff all four mismatch transition rates coincide (a class containing the 2D rank-1 projective model with a fixed measurement pair under state variation); a polarity- and position-dependent repetition family is characterized by an exact cross-symmetry condition with closed-form violations; QQ-satisfying behaviors are closed under order-matched mixing; and the rank-2 Contextuality-by-Default criterion translates into audit coordinates as $|\qQQ|\le\OSS$, where $\OSS$ (the order-sensitivity score) totals the order sensitivity of the two marginals. Methodologically, we develop a pre-specified, audit-logged pipeline applicable to any model exposing next-token log-probabilities; it combines worst-case robustness envelopes, sampling-consistency spot checks, full label counterbalancing, and a saturation diagnostic. Empirically, in a first-signal pilot on an open-weight instruction-tuned model under two framings, all pre-specified health gates passed, yet 17/18 and 7/8 item pairs, respectively, were saturated (near-deterministic), and no item was certified residually contextual. Forced-binary next-token log-probabilities were thus inadequate for distribution-level QQ audits under the tested model and prompting conditions; we recommend pre-specified saturation diagnostics whenever next-token distributions are treated as survey-response distributions.