ArXiv

DBA-Bench: A Production-Fidelity Benchmark for LLM-Based Database Operations Agents

2026-07-24 · 10:11 UTC ·Junming Chen, Junyang Jiang, Xu Chen... ·1 min read
Authors
Junming Chen, Junyang Jiang, Xu Chen...
Categories
cs.DB, cs.AI, cs.CL, cs.LG
arXiv
https://arxiv.org/abs/2607.22165v1
PDF
https://arxiv.org/pdf/2607.22165v1

Brief

DBA-Bench provides a reproducible, outcome-first benchmark for LLM-driven database operations by running 106 instrumented PostgreSQL scenarios (seven task domains) with active workloads and safety-constrained success criteria. The paper measures end-to-end diagnosis and remediation across 848 runs, revealing low automated Safe Pass rates (12.4% overall, 17.9% best baseline) compared with 93.4% for a human DBA, and markedly worse performance on harder scenarios.

Why it matters

DBA-Bench introduces a production-fidelity benchmark for LLM-based database agents using instrumented PostgreSQL with active workloads, persistent state, multi-source observations, and snapshot restoration; it targets four evaluation gaps: live-environment fidelity, observation-space scale, solution-space openness, and scenario complexity.

Key details

  • The benchmark includes 106 scenarios across seven task domains with two public difficulty labels; across 848 automated runs the reported Diagnosis, Outcome, and Safe Pass rates are 32.7%, 19.6%, and 12.4% respectively.
  • Nine baselines were evaluated (six foundation-model systems, two GPT-5.5-backed agents, and a Human DBA): the best automated baseline achieved 17.9% Safe Pass versus 93.4% for the Human DBA; automated Safe Pass drops from 19.6% on Easy scenarios to 7.6% on Hard scenarios.
Source evidence

Abstract

LLM-based database agents show promise, but differing task scopes, testbeds, and metrics hinder comparison. We identify four gaps between evaluation and production operations: live-environment fidelity (multi-turn read-write interaction with a running database); observation-space scale and complexity (causal diagnosis across thousands of time series, business logs, and concurrent activity); solution-space openness (multiple remediations with different operational trade-offs); and scenario complexity and coverage (faults cascading across internal mechanisms and operational domains). We present DBA-Bench, a benchmark addressing these gaps through production fidelity, outcome-first evaluation, and controlled scenario reproducibility. It uses instrumented PostgreSQL environments with active workloads, persistent state, and multi-source observations; defines success by measurable recovery or fault elimination under safety constraints; and restores snapshots with scenario-specific checks before each run. The benchmark contains 106 scenarios across seven task domains, with two public difficulty labels based on reference-path diagnostic depth and environmental complexity. We evaluate nine baseline groups, including six foundation-model systems, two GPT-5.5-backed database agents, and a Human DBA reference. Across 848 automated runs, Diagnosis, Outcome, and Safe Pass rates are 32.7%, 19.6%, and 12.4%; the best automated baseline reaches 17.9% Safe Pass versus 93.4% for the Human DBA reference. Automated Safe Pass falls from 19.6% on Easy scenarios to 7.6% on Hard scenarios, underscoring the difficulty of safe end-to-end remediation.

Comment: 14 pages, 6 figures, 2 tables