Twitter/X

SWE-Together converts 11,260 recorded user–agent coding sessions into 109…

2026-06-30 · 15:17 UTC ·@yifannnwu ·1 min read

Brief

SWE-Together is a multi-turn benchmark that turns 11,260 recorded user–agent coding sessions into 109 repo-level tasks replayed by a reactive LLM user simulator that preserves original users’ intent. It measures agents as collaborators via final pass rate and user interventions; in a 2026-06-30 snapshot, claude-opus-4.8 led among seven agents.

Why it matters

SWE-Together converts 11,260 recorded user–agent coding sessions into 109 repo-level tasks, replayed with a reactive LLM user simulator that preserves the original user’s intent.

Key details

  • The benchmark evaluates agents as collaborators using two collaboration metrics: final pass rate and the number of user interventions required to reach a passing solution (reproducible and verifiable).
  • In a snapshot evaluation announced 2026-06-30 by @yifannnwu, claude-opus-4.8 led among seven agents tested, achieving the highest pass rate while requiring the fewest user interventions.
Source evidence

Introducing SWE-Together: a multi-turn benchmark built from real user–agent coding sessions.

Coding agents are often benchmarked like exam-takers: given the full spec up front, then graded on the final code. But real coding help is a conversation — users clarify goals, add constraints, and correct course along the way.

SWE-Together turns real coding work into a reproducible, verifiable benchmark: 109 repo-level tasks curated from 11,260 recorded sessions, replayed with a reactive LLM user simulator that preserves the original user’s intent.

We evaluate agents as collaborators, not just patch generators: final pass rate and how many user interventions were needed to get there.

In this evaluation snapshot, claude-opus-4.8 currently leads among the 7 agents we tested — achieving the highest pass rate while requiring the fewest user interventions.

📄 Paper: arxiv.org/abs/2606.29957
💻 Code: github.com/Togetherbench/SWE…
🌐 Website: togetherbench.com