ArXiv

ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

2026-07-31 · 17:55 UTC ·Boyang Zhang, Adrian Lyjak, Eli Stewart... ·1 min read
Authors
Boyang Zhang, Adrian Lyjak, Eli Stewart...
Categories
cs.AI
arXiv
https://arxiv.org/abs/2607.29677v1
PDF
https://arxiv.org/pdf/2607.29677v1

Brief

ExtractBench is a schema-guided extraction benchmark built from 4,869 pages (370 enterprise documents, 8 domains, 67 types) that scores value accuracy (order-insensitive F1), record completeness, grounding (word- and page-level F1), and measured cost. The curation pipeline mixes independent-system agreement, known synthetic values, and human verification for forms. Results show VLMs struggle on long lists, coding agents are more accurate but costly, and LlamaExtract Agentic Plus achieves top metrics cost-effectively. (Abstract only; full text not provided.)

Why it matters

ExtractBench comprises 4,869 pages across 370 enterprise documents, spanning 8 business domains and 67 document types, and is presented as the first benchmark to jointly evaluate value accuracy, record completeness at scale, grounding, and measured cost.

Key details

  • Evaluation uses an order-insensitive value F1 plus two grounding metrics (word- and page-level F1); commercial VLMs perform well on short documents but often truncate long record lists, coding agents achieve higher accuracy at much higher cost, and LlamaExtract Agentic Plus ranks first on all three metrics with accuracy comparable to coding agents at a fraction of the cost.
Source evidence

Abstract

Enterprise workflows increasingly rely on agents for \emph{schema-guided extraction}: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a benchmark for schema-guided extraction and, to our knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together. The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with clear tags differentiating their challenge scenarios. The scalable schema and ground-truth curation pipeline combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. We report order-insensitive value F1 for value accuracy, plus two grounding metrics for source traceability: word- and page-level F1. Commercial VLMs perform well on short documents but often truncate record lists on long ones, while coding agents retain higher accuracy at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost. Dataset and evaluation code are available on \href{https://huggingface.co/datasets/llamaindex/ExtractBench}{HuggingFace} and \href{https://github.com/run-llama/ExtractBench}{GitHub}.