ArXiv

How Robust is OCR-Reasoning? Evaluating OCR-Reasoning Robustness of Vision-Language Models under Visual Perturbations

2026-06-24 · 17:15 UTC ·Yuxing Cheng, Yuan Wu, Yi Chang ·1 min read
Authors
Yuxing Cheng, Yuan Wu, Yi Chang
Categories
cs.CV, cs.CL
arXiv
https://arxiv.org/abs/2606.26041v1
PDF
https://arxiv.org/pdf/2606.26041v1

Brief

OCR-Robust evaluates VLM OCR-reasoning under controlled visual perturbations using 812 samples split into OCR1.0 (documents, scene text, receipts, handwriting, math) and OCR2.0 (charts, geometry diagrams, tables). The authors narrowed 18 candidate corruptions to five types at three severity levels and benchmarked 18 models with metrics (clean accuracy, RCR, WCR, CRI), finding clean accuracy is a poor proxy for corruption robustness and that charts/tables are especially vulnerable.

Why it matters

Introduces OCR-Robust (published 2026-06-24 by Yuxing Cheng, Yuan Wu, Yi Chang): 812 samples across two subsets — OCR1.0 (documents, scene text, receipts, handwriting, mathematical content) and OCR2.0 (charts, geometry diagrams, tables).

Key details

  • Method: pilot-tested 18 candidate visual perturbations, selected 5 representative types at 3 severity levels; evaluated 18 models (proprietary systems, open-source VLMs, and OCR+LLM pipelines) using clean accuracy, Relative Corruption Retention (RCR), Worst-Case Retention (WCR), and Corruption Robustness Index (CRI).
  • Key results: higher clean accuracy often fails to predict robustness; models exhibit pronounced worst-case degradation on structure-sensitive OCR tasks, with charts and tables substantially more fragile than document-like inputs under perturbations.
Source evidence

Abstract

Vision-language models (VLMs) have achieved strong performance on OCR-based benchmarks and increasingly focused on text-rich understanding, but their robustness under controlled visual degradation remains insufficiently understood. This gap is critical for OCR reasoning, where visual corruption can induce OCR errors and structural distortions, thereby introducing uncertainty into the reasoning task. To systematically study this problem, we introduce OCR-Robust, a benchmark designed for evaluating OCR reasoning robustness under visual perturbations. It contains 812 samples across two complementary subsets: OCR1.0, covering documents, scene text, receipts, handwriting, and mathematical content, and OCR2.0, focusing on charts, geometry diagrams, and tables. To enable efficient yet informative evaluation, we conduct a pilot study over 18 candidate perturbations and select 5 representative types at 3 severity levels each based on their impact and cross-model discriminability. We evaluate robustness using clean accuracy, Relative Corruption Retention (RCR), Worst-Case Retention (WCR), and a composite Corruption Robustness Index (CRI), and benchmark 18 models spanning proprietary systems, open-source VLMs, and OCR+LLM pipelines. Our results show that higher clean accuracy does not necessarily imply stronger robustness, and that models can suffer pronounced degradation in the worst case on OCR tasks that are sensitive to structure, and charts and tables are substantially more fragile than document-like inputs under perturbation.