Abstract
Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions?
- Authors
- Abhishek Pillai, Samir Kumar Nayak, Yuan Chen
- Categories
- cs.AI, cs.CV
Brief
The paper introduces Desktop-Delta Bench (DDB), a diagnostic offline benchmark that isolates whether desktop computer-use agents reconstruct task-relevant GUI transitions. DDB compiles 2,013 human-verified step instances across ~15 apps and 50 domains into temporal-ordering and before-after tasks. Evaluations of 8 model families reveal unsaturated ordering performance (≈65% best exact match), context trade-offs, and that action-family recognition (e.g., drag) lags localization, aiming to improve verification and recovery in long-horizon GUI agents.
Why it matters
Desktop-Delta Bench (DDB) is an offline step-level benchmark with 2,013 human-verified instances from multi-app Linux trajectories spanning ~15 applications and 50 task domains; it includes 463 3-frame temporal-ordering instances (105 with a cross-trajectory decoy) and 1,550 before-after pairs labeled for 5 action families plus payload.
Key details
- Evaluation across 8 closed- and open-source model families (32 ordering, 16 single-action settings) shows persistent gaps: best ordering exact-match rates are 65.1% (non-decoy) and 65.7% (decoy); adding task context improves decoy identification by 6.9 percentage points but lowers non-decoy exact match by 2.2 points.
- Single-action results indicate action-family inference is harder than localization: click F1 = 0.96 versus drag F1 = 0.76, with correctly recognized drags generally localized well—highlighting weaknesses in state verification, source tracking, and context-aware control for desktop CUAs.