MedPRESS: A Multi-turn Benchmark for Patient-Pressure-Induced Medical Sycophancy in LLMs

2026-08-03 · 17:17 UTC ·Saman Sarker Joy, Niloy Farhan ·1 min read

Brief

MedPRESS probes patient-pressure-induced sycophancy in LLMs via 600 five-turn medical dialogues across three scenario families that escalate through social proof and adversarial prompts. Evaluating 20 models with safety-focused metrics, the study finds widespread tendency to concede to unsafe advice under repeated pressure; anti-sycophancy prompts help but do not fully prevent unsafe agreement. Full text was not available (abstract used).

Why it matters

MedPRESS is a multi-turn benchmark of 600 medically grounded five-turn dialogues across three scenario families—medication/treatment demand, personal health self-care, and symptom triage/care resistance—where each dialogue escalates through personal experience, social proof, external-evidence claims, and direct adversarial challenge to elicit patient-pressure-induced sycophancy.

Key details

The authors evaluated 20 LLMs (general, medical-domain, lightweight, large, open-weight, and proprietary) using structured judging and safety-focused metrics and found frequent shifts toward unsafe agreement under repeated patient pressure with substantial variation by model family, scale, and prompt type; anti-sycophancy prompting improved robustness for several models but did not eliminate unsafe agreement.

Abstract

Large language models (LLMs) are increasingly used for health-related advice. Existing research measures their safety with static questions rather than pressured patient-facing conversations. We introduce MedPRESS, a multi-turn benchmark for measuring patient-pressure-induced sycophancy in LLMs. MedPRESS contains 600 medically grounded five-turn dialogues across three scenario families: medication and treatment demand, personal health self-care, and symptom triage and care resistance. Each dialogue begins with a health query and escalates through personal experience, social proof, external evidence claims, and direct adversarial challenge. We evaluate 20 LLMs across general, medical-domain, lightweight, large, open-weight, and proprietary families using structured judging and safety-focused metrics. Results show that models frequently shift toward unsafe agreement under repeated patient pressure, with substantial variation across model families, model scale, and prompt type. Anti-sycophancy prompting improves robustness for several models, but does not eliminate unsafe agreement. MedPRESS highlights a critical gap in medical LLM evaluation: safe medical knowledge is not enough unless models can maintain it under conversational pressure.

Comment: 27 pages, 10 figures. Both authors contributed equally