Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents

2026-08-06 · 17:39 UTC ·Noam Koren, Roy Bar-Haim, Abigail Goldsteen ·1 min read

Brief

The paper presents a reference-free evaluation framework that leverages LLM judges to quantify benchmark consistency, scenario complexity, and policy coverage for task-oriented conversational agents, filling a gap where benchmark quality is rarely assessed. The authors validate the approach via agreement with human annotations and experiments on LLM-generated and perturbed benchmarks, showing metrics reliably separate quality levels across domains and judge models.

Why it matters

Introduces a reference-free framework that uses LLM judges to assess benchmark quality for task-oriented conversational agents along dimensions of consistency, complexity, and policy coverage, and produces actionable diagnostics (authors: Noam Koren, Roy Bar-Haim, Abigail Goldsteen; arXiv:2608.06329v1; published 2026-08-06).

Key details

Validated by agreement with independent human annotations and by testing on benchmarks generated by LLMs of varying capabilities and on benchmarks with controlled quality-degrading perturbations; reported metrics consistently distinguish benchmark quality levels across domains and judge models and apply to both synthetic and manually curated benchmarks.

Abstract

Task-oriented conversational agents are evaluated using curated or automatically generated benchmarks, yet benchmark quality is rarely assessed. Poor benchmarks may contain inconsistent tasks, simplistic scenarios, or limited policy coverage, leading to unreliable evaluations. We introduce a reference-free framework that uses LLM judges to assess benchmark consistency, complexity, and policy coverage, while providing actionable diagnostics of weaknesses. We validate the framework by demonstrating agreement with independent human annotations and by evaluating benchmarks generated by LLMs of varying capabilities, as well as benchmarks subjected to controlled quality-degrading perturbations. Across domains and judge models, the proposed metrics consistently distinguish between benchmark quality levels. We further demonstrate the framework's applicability to manually curated benchmarks. Our framework offers a practical approach for evaluating synthetic and manually curated conversational-agent benchmarks.

Comment: 15 pages