ArXiv

Benchmarking and Enhancing LLMs for Rule-Intensive Review of National Standard Documents

2026-08-06 · 17:27 UTC ·Tao Wang, Qihao Yang, Rongjiao Liang... ·1 min read
Authors
Tao Wang, Qihao Yang, Rongjiao Liang...
Categories
cs.CL
arXiv
https://arxiv.org/abs/2608.06312v1
PDF
https://arxiv.org/pdf/2608.06312v1

Brief

GB/T-Bench targets rule-intensive review of China GB/T national standards by generating 7,306 traceable error instances from 488 documents and a 25-type hierarchical taxonomy. The authors design a diagnosis-oriented evaluation (exact match on location/dimension/type) and propose GB/T-Reviewer, a multi-agent framework; across 14 LLMs best CMCS=0.3280 (experts 0.6640), GB/T-Reviewer improves to 0.5094. Full text not available.

Why it matters

GB/T-Bench converts 488 China GB/T standard documents into 7,306 traceable review-error instances via a controllable counterexample generation combining deterministic rules and constrained LLM rewriting; it defines a hierarchical GB/T Review Taxonomy with 25 diagnosable error types.

Key details

  • Evaluation over 14 mainstream LLMs shows a large human–LLM gap: the strongest model achieves CMCS=0.3280 versus expert CMCS=0.6640; the proposed multi-agent GB/T-Reviewer raises the best CMCS to 0.5094, improving but not closing the gap.
  • The work introduces a diagnosis-oriented evaluation protocol requiring exact matches on error location, review dimension, and error type, plus document-level coverage metrics to assess rule-intensive review performance.
Source evidence

Abstract

Large language models (LLMs) increasingly support complex professional tasks, yet their capabilities in rule-intensive document review remain insufficiently evaluated. National standard documents, such as China GB/T standards, offer a representative testbed: they are lengthy, highly structured, and governed by explicit rules for scope, terminology, normative wording, and cross-section consistency. Existing benchmarks focus on domain knowledge and question answering, largely overlooking intrinsic quality review for professional documents. Such reviews rely heavily on human experts, making them costly and difficult to scale. To bridge this gap, we introduce GB/T-Bench, the first benchmark for the structured review of national standard documents. Its GB/T Review Taxonomy is a hierarchical schema covering document structure, scope alignment, normative modality, terminology consistency, and normative references, with 25 diagnosable error types. A controllable counterexample generation mechanism combines deterministic rules and constrained LLM rewriting to process 488 documents into 7,306 traceable review error instances for evaluation. We also develop a diagnosis-oriented evaluation protocol requiring exact matches on error location, review dimension, and error type, plus document-level coverage metrics. We further propose GB/T-Reviewer, a multi-agent framework that converts review knowledge into specialized skills and coordinates global inspection, targeted diagnosis, rule scanning, and result verification. Experiments with 14 mainstream LLMs reveal a substantial human-LLM gap: the strongest model achieves only 0.3280 CMCS versus 0.6640 for experts. GB/T-Reviewer raises the best CMCS to 0.5094, showing the value of structured skill coordination for rule-intensive document review. This work paves the way for trustworthy AI in standardization and other high-stakes document domains.