ArXiv

ERUnderstand: Evaluating Vision-Language Models on Structured ER Diagrams

2026-07-27 · 17:46 UTC ·Ali Ansari, Yasmin Mohammadi, Farnoush Nili... ·1 min read
Authors
Ali Ansari, Yasmin Mohammadi, Farnoush Nili...
Categories
cs.AI, cs.CV, cs.DB
arXiv
https://arxiv.org/abs/2607.24707v1
PDF
https://arxiv.org/pdf/2607.24707v1

Brief

ERUnderstand presents the first large-scale benchmark (2,960 diagrams) for structured understanding of ER/EER diagrams by pairing rendered images with standardized machine-readable schemas. The authors evaluate state-of-the-art vision-language models, report high F1 on common elements but major failures on weak entities, multivalued attributes and N-ary relations, and show 15–25% gains from reasoning-augmented models. Full materials are publicly released.

Why it matters

ERUnderstand is a new benchmark of 2,960 ER/EER diagrams drawn from educational sources, real-world schemas, and synthetic generation, each paired with a standardized machine-readable schema; dataset, evaluation toolkit, and generation code are publicly released at https://github.com/salinaria/ERUnderstand (ArXiv: 2607.24707v1, 2026-07-27).

Key details

  • Evaluation of state-of-the-art vision-language models shows reliable recovery of common ERD elements (F1 > 0.74) but sharp drops on weak entities (0.28 F1), multivalued attributes (0.14 F1), and N-ary relationships (0.07 F1); reasoning-augmented models raise overall performance by 15–25% yet remain sensitive to linguistic priors and diagram complexity.
Source evidence

Abstract

Entity-Relationship Diagrams (ERDs) are central to conceptual database design, yet they are typically available only as rendered images rather than machine-readable schemas, limiting AI-assisted database engineering. We introduce ERUnderstand, the first large-scale benchmark for structured understanding of ER diagrams, comprising 2,960 diagrams collected from curated educational sources, real-world schemas, and synthetically generated examples spanning diverse domains, notations, complexity levels, and Extended Entity-Relationship (EER) constructs. Each diagram is paired with a standardized machine-readable representation for fine-grained evaluation of schema elements. Evaluating state-of-the-art Vision-Language Models (VLMs), we find that while common ERD elements are recovered reliably (F1 > 0.74), performance drops sharply on weak entities (as low as 0.28 F1), multivalued attributes (0.14 F1), and N-ary relationships (0.07 F1). Reasoning-augmented models improve overall performance by 15-25% but remain sensitive to linguistic priors and increasing diagram complexity. ERUnderstand provides a standardized benchmark for evaluating multimodal understanding of conceptual database schemas. The benchmark, dataset, evaluation toolkit, and generation code are publicly available at https://github.com/salinaria/ERUnderstand.