ArXiv

Can LLMs Reliably Self-Report Adversarial Prefills, and How?

2026-06-22 · 17:56 UTC ·Quang Minh Nguyen, Uzair Ahmed, Taegyoon Kim ·1 min read
Authors
Quang Minh Nguyen, Uzair Ahmed, Taegyoon Kim
Categories
cs.CL
arXiv
https://arxiv.org/abs/2606.23671v1
PDF
https://arxiv.org/pdf/2606.23671v1

Brief

The paper evaluates whether LLMs can reliably self-report that a prior response was caused by an adversarial prefill. Using ten open-weight instruction-tuned models (3B–70B) and four safety benchmarks, authors find weak self-detection (27.3% average claiming rate). Orthogonalizing against a refusal direction removes the signal gap, but is not the only mediator. LoRA finetuning (SFT, GRPO, DPO) increased introspective gaps yet worsened attack success; framing probes (internal intent vs external tampering) yields different responses. Full text was not available in the source provided (abstract-only).

Why it matters

Across ten open-weight instruction-tuned LLMs (3B–70B) evaluated on four safety benchmarks, models failed to reliably recognize when outputs were elicited by adversarial prefill attacks, claiming intent on prefilled responses at an average rate of 27.3%.

Key details

  • Mechanisms: introspective signal largely stems from safety-/refusal-related reasoning; orthogonalizing weights against a refusal direction collapses the gap between claiming rates on prefilled vs. natural outputs to near zero (though that direction is not the sole mediator). Three LoRA finetuning methods (SFT, GRPO, DPO) tested on eight models (3B–27B) widened the intention-probe gap for every 8B–27B model, did not transfer to a tampering probe, and—counterintuitively—increased attack success under adversarial prefill on most models.
Source evidence

Abstract

Prior work shows that large language models (LLMs) exhibit introspective capability on benign tasks. We extend the question to safety contexts and examine how reliably a model can recognize that its own prior response was elicited by an adversarial prefill attack. Across ten open-weight instruction-tuned LLMs (3B to 70B) and four safety benchmarks, no model reliably recognizes its own compromised outputs, with models claiming intent on prefilled responses at an average rate of $27.3\%$. Introspective signal stems largely from safety- and refusal-related reasoning. Orthogonalizing models' weights against the refusal direction collapses the gap between claiming rates on prefilled and natural outputs to near zero, though the direction is not its unique mediator. The signal is also probe-dependent: framing the question as internal intention versus external tampering elicits qualitatively different responses on the same models. We test three LoRA finetuning methods (SFT, GRPO, DPO) on eight models from 3B to 27B; all three widen the intention-probe gap on every model from 8B to 27B, with method ranking varying by model. The intervention does not transfer to the tampering probe and counterintuitively raises attack success rate under adversarial prefill on most models, amounting to a partial mitigation. These findings outline mechanisms underpinning the observed introspective signals in safety contexts and highlight risks in the reliability of LLM self-reports.