ArXiv

OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding

2026-07-29 · 17:33 UTC ·Jingbo Zhou, Yusai Zhao, Qi Bao... ·1 min read
Authors
Jingbo Zhou, Yusai Zhao, Qi Bao...
Categories
cs.AI, cs.CL, cs.HC
arXiv
https://arxiv.org/abs/2607.27155v1
PDF
https://arxiv.org/pdf/2607.27155v1

Brief

OmegaUse-OfficeVal is a benchmark for long-horizon office-suite workflows that adds economic grounding: 100 practitioner-derived tasks (avg. 2.32 hours human labor) each paired with human labor time and a task price proxy. The authors supply code-based verifiers from fine-grained rubrics and evaluate multiple frontier LLMs plus a human baseline, finding LLMs far cheaper and faster but below human deliverable quality; data and code are open-source.

Why it matters

OmegaUse-OfficeVal is a new benchmark of 100 long-horizon office-suite tasks (sourced from practitioner requests via a privacy-preserving process) with an average human completion time of 2.32 hours per task.

Key details

  • Each task is paired with two economic signals—human labor time and a task price proxy—and the authors provide code-based verifiers built from fine-grained rubrics to enable value-weighted and cost-comparative evaluation.
  • Evaluations on several frontier LLMs vs. a human baseline show LLM agents are substantially cheaper and faster than humans but have not reached human-level deliverable quality; the dataset and code are fully open-sourced (https://omegause-officeval.github.io).
Source evidence

Abstract

Large language model (LLM) agents are increasingly expected to assist users in completing tasks. However, existing benchmarks provide limited support for evaluating whether agents can carry out office-suite workflows at a reasonable cost. We introduce OmegaUse-OfficeVal, a benchmark for evaluating LLM agents on long-horizon office-suite tasks with task-level economic grounding. The benchmark comprises 100 tasks derived from office-suite requests proposed by practitioners and adapted through a privacy-preserving process. On average, these tasks require 2.32 hours of human labor to complete. An important feature of the benchmark is that each task is paired with two economic signals: human labor time and task price proxy. These signals enable direct comparisons between human costs and LLM inference costs, as well as value-weighted evaluation. To support stable evaluation, we develop code-based verifiers from fine-grained rubrics. We evaluate several frontier LLMs together with a human baseline. Although all evaluated LLMs are substantially cheaper and faster than human workers, they have not yet approached human-level deliverable quality. The code and dataset are fully open-sourced, and more information is available on our project website: https://omegause-officeval.github.io.