ArXiv

Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidence

2026-07-16 · 15:01 UTC ·Haocheng Yang, Licheng Pan, Xiaoxi Li... ·1 min read
Authors
Haocheng Yang, Licheng Pan, Xiaoxi Li...
Categories
cs.CL
arXiv
https://arxiv.org/abs/2607.15092v1
PDF
https://arxiv.org/pdf/2607.15092v1

Brief

Rubrics on Trial introduces a procedure that iteratively grows and validates query-specific rubrics from scratch by generating and checking synthetic rubric-conditioned response pairs, avoiding external annotations or model retraining. The framework screens out rubrics that fail to discriminate quality or that merely encode style, and empirically attains top average accuracy, winning six of seven evaluation sets across five preference benchmark suites (Yang et al., 2026).

Why it matters

Rubrics on Trial is a query-only framework that evolves a rubric set from an empty seed using only synthetic rubric-conditioned response pairs (no human-written rubrics, preference data, or model training); it validates each proposed rubric and filters out non-discriminative, over-specific, and style-only candidates.

Key details

  • In experiments across five preference benchmark suites, the method achieves the best average accuracy and leads on six of seven evaluation sets (ArXiv preprint 2607.15092v1, published 2026-07-16; authors: Haocheng Yang et al.).
Source evidence

Abstract

Rubrics provide structured, fine-grained signals for training and evaluating large language models (LLMs). Yet reliable query-specific rubrics are difficult to construct. Existing approaches often derive supervision from human-written rubrics, preference data, or sampled responses. Direct query-to-rubric generation avoids these resources, but provides no explicit check that a plausible rubric is useful. Such a rubric may fail to distinguish answer quality, reward an optional style, or penalize a valid alternative strategy. We introduce Rubrics on Trial, a query-only framework that evolves a rubric set from an empty set without external annotations or model training. It derives supervision solely from synthetic rubric-conditioned response pairs and validates each proposed rubric before adding it, screening out non-discriminative, over-specific, and style-only candidate rubrics. Experiments across five preference benchmark suites demonstrate the effectiveness of Rubrics on Trial, which achieves the best average accuracy and leads on six of seven evaluation sets.