Twitter/X

Frontier LLMs solve <1% of ProgramBench tasks when asked to build complete…

2026-07-30 · 01:56 UTC ·@boyuan_chen ·1 min read

Brief

MindForge is a pipeline that trains smaller models to construct complete software from scratch using a software-engineering lens. Applied via SFT to Qwen 3.6 27B, it boosted ProgramBench from 37.98% to 49.51% and produced large, statistically significant generalization gains across seven unseen benchmarks, including C→Rust and DeepSWE.

Why it matters

Frontier LLMs solve <1% of ProgramBench tasks when asked to build complete programs from scratch; the author identifies a key bottleneck as the absence of scalable training environments that span the full software development lifecycle.

Key details

  • MindForge SFT on Qwen 3.6 27B raised ProgramBench performance from 37.98% to 49.51% (+11.5 points, +30% relative), outperforming DeepSeek V4 Pro (47.8%) and approaching GLM-5.1 (50.9%) and Claude Opus 4.7 (51.4%) despite being dozens of times smaller.
  • The improvement generalized to seven unseen benchmarks: +31 points on C→Rust repository translation, a 9× improvement on DeepSWE, +10.7 on NL2Repo, +5 on SWE-bench Verified, and additional statistically significant gains on SWE-bench Pro, Multilingual, and FeatBench.
Source evidence

(1/n) 🧵

Coding agents have become good at editing code they can already see, e.g., bug fixes and feature patches. But ask them to build an entire program from nothing, and even frontier models fully solve <1% of ProgramBench tasks. A key bottleneck exists: there are no scalable training environments for this setting. Existing frameworks only cover single phases of development.

We built a pipeline, MindForge, that teaches small models to do what even frontier LLMs struggle with: constructing complete software from scratch, through a software engineering lens. The results are surprisingly generalizable across the whole software engineering cycle.

SFT on Qwen 3.6 27B, 📈 ProgramBench: 37.98% → 49.51% (+11.5 pts, +30% relative) — above DeepSeek V4 Pro (47.8%) and right beside GLM-5.1 (50.9%) and Claude Opus 4.7 (51.4%), models dozens of times its size. Moreover, the improvement generalizes to 7 benchmarks it never trained on: +31 pts on C→Rust repository translation, a 9× improvement on DeepSWE, +10.7 on NL2Repo, +5 on SWE-bench Verified, plus gains on SWE-bench Pro, Multilingual, and FeatBench — all statistically significant.

How we did it?