ArXiv

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

2026-08-06 · 17:53 UTC ·Fanzhe Meng, Guoxin Chen, Jiale Zhao... ·1 min read
Authors
Fanzhe Meng, Guoxin Chen, Jiale Zhao...
Categories
cs.LG, cs.CL
arXiv
https://arxiv.org/abs/2608.06352v1
PDF
https://arxiv.org/pdf/2608.06352v1

Brief

CalibForge is an autonomous system for synthesizing terminal tasks by adversarially calibrating candidate problems against solver behavior, operationalizing a solver-relative learnable zone via multi-solver and contrastive calibration. The authors produce 5,431 calibrated tasks and show that training on them yields substantial transfer gains—up to ~30 percentage points on downstream benchmarks—supporting solver-relative learnability as a practical target.

Why it matters

CalibForge synthesizes 5,431 calibrated terminal tasks via adversarial solver calibration, using two strategies: multi-solver calibration (targets disagreement in heterogeneous solver pools) and contrastive solver calibration (targets a strong-pass/weak-fail relation).

Key details

  • Models trained on the CalibForge collection achieve 32.58% and 47.57% on Terminal-Bench 2.0; the largest reported improvements over base models are +24.71 percentage points (Terminal-Bench 2.0), +27.68 points (SWE-bench Pro), and +30.04 points (Doc2Repo).
Source evidence

Abstract

Training terminal agents requires executable and verifiable tasks that are not merely solvable, but appropriately challenging for learning. Executable validation establishes feasibility, yet does not reveal how a task behaves relative to a given solver setting. In this paper, we present CalibForge, an autonomous terminal-task synthesis system that uses verified solver behavior to revise candidate tasks through adversarial solver calibration. Multi-solver calibration targets disagreement within a heterogeneous solver pool, whereas contrastive solver calibration targets a designated strong-pass/weak-fail relation; both operationalize a solver-relative learnable zone anchored in demonstrated solvability. Using CalibForge, we construct 5,431 calibrated terminal tasks. Our ablations show that both strategies yield more effective supervision than authoring and validation alone or ordinary single-solver feedback. Models trained on the full collection achieve 32.58% and 47.57% on Terminal-Bench 2.0. The largest improvements over the corresponding base model reach 24.71 percentage points on Terminal-Bench 2.0, 27.68 points on SWE-bench Pro, and 30.04 points on Doc2Repo. Together, these results support solver-relative learnability as a practical target for constructing effective and transferable agent training data.

Comment: Dataset: https://huggingface.co/datasets/AweAI-Team/CalibForge. Repository: https://github.com/AweAI-Team/CalibForge