Twitter/X

Probe-and-refine tuning of repository guidance using synthetic bug-fix probes…

2026-06-23 · 13:33 UTC ·@boyuan_chen ·0 min read

Brief

Probe-and-Refine tuning of repository guidance improves LLM coding-agent performance by generating synthetic bug-fix probes that refine AGENTS.md-like knowledge. On SWE-bench it increases Verified fixes to 33.0% versus 28.3% for a static knowledge base, boosting evaluable-patch coverage without reducing precision and focusing on repo operational details.

Why it matters

Probe-and-refine tuning of repository guidance using synthetic bug-fix probes raises SWE-bench Verified from 28.3% (static KB) to 33.0%.

Key details

  • The reported gain comes from increased coverage—more evaluable patches are produced—while patch precision remains the same.
  • The method targets AGENTS.md-style repo guidance for LLM coding agents, encoding operational knowledge like file→subsystem mappings, test commands, and historical workflows.
Source evidence

Your AGENTS.md may be silently wasting agent steps. Probe-and-refine tunes repo guidance with synthetic bug-fix probes: 33.0% SWE-bench Verified vs 28.3% for a static KB. The gain is coverage: more evaluable patches, same precision. arxiv.org/abs/2606.20512

Link

Probe-and-Refine Tuning of Repository Guidance for Coding Agents

LLM-based coding agents need higher-level operational knowledge about a repository (which files house which subsystems, how to run the test suite, which workflows have historically led to wrong...
arxiv.org