The author reports a substantial improvement on OSWorld v2—55.7% to 70.6%—and is seeking more difficult computer-use evaluations. They propose sponsoring a benchmark that would push Opus 5 below 50% and invite contributors with matching evals to contact them. The post also cites Claude’s claim that Opus 5 now leads on multiple coding and knowledge-work tests.
Performance jump from 55.7% to 70.6% on OSWorld v2 (a benchmark released weeks earlier).
55.7% -> 70.6% on OSWorld v2 (released just a few weeks ago)
Genuinely would love to have a harder computer use evals. Happy to sponsor a computer use benchmark that Opus 5 gets below 50% on!
If you have an eval that matches this criteria please reach out! Happy to chat.
Claude (@claudeai)
On several coding and knowledge work evaluations, Opus 5 is the new state-of-the-art:
— https://nitter.net/claudeai/status/2080699497064083942#m