OpenAI (via Future Tools)

How enabling two settings tripled our scores on the ARC-AGI-3 benchmark

2026-07-29 · 00:00 UTC ·4 min read

Brief

Enabling retained reasoning and compaction on the Responses API substantially changed ARC‑AGI‑3 results: when GPT‑5.6 Sol was allowed to keep private reasoning messages and the harness used compaction instead of rolling truncation, its score rose to roughly three times the previous result while using about six times fewer output tokens. The team traced poor baseline performance (GPT‑5.6 Sol 7.8%, GPT‑5.5 0.4%) to two harness features: discarding internal chain‑of‑thought after each action and truncating history once the context exceeded ~175,000 characters. Because the models are trained to produce private reasoning that is retained across turns, preserving that reasoning and compacting older content let the agent learn strategies over many steps. The authors therefore recommend the Responses API with retained reasoning and compaction for fairer, production‑relevant evals.

Why it matters

Enabling two API settings—retained reasoning and compaction—on the Responses API produced roughly a 3× improvement in GPT‑5.6 Sol’s ARC‑AGI‑3 score while reducing output tokens by about 6×.

Key details

  • Baseline performance was very low: GPT‑5.6 Sol scored 7.8% on ARC‑AGI‑3 and GPT‑5.5 scored 0.4% on the same 2D puzzle-game benchmark (ARC‑AGI‑3 exposes 25 demo games).
  • The original ARC harness discarded private reasoning after every action and used rolling truncation that dropped oldest messages once context exceeded 175,000 characters, which prevented the model from retaining plans and learning across turns.
  • Practical recommendations: use the Responses API (not Chat Completions), pass the previous response ID to retain private reasoning, and enable compaction to preserve long-term context and better match production deployments.
Source evidence

How enabling two settings tripled our scores on the ARC-AGI-3 benchmark

When we first saw GPT‑5.6 Sol’s low scores on the ARC-AGI-3(opens in a new window) benchmark, we were puzzled.

GPT‑5.6 Sol has solved longstanding open problems in mathematics like the cycle double cover conjecture(opens in a new window) and beaten games like Pokémon FireRed. But on ARC-AGI-3, a benchmark of 2D puzzle games, GPT‑5.6 Sol scored just 7.8%, and GPT‑5.5 could barely play the games at all, scoring a paltry 0.4%.

Were 2D puzzle games unusually difficult for our models? Or was something else going on?

Benchmarks rarely measure AI models in isolation. They also measure less visible choices about API settings, harness design, and prompting. In the case of ARC-AGI-3, we discovered that turning on two API settings we use in ChatGPT and Codex—retained reasoning and compaction—tripled scores and cut output tokens by 6x on the public task set.

ARC-AGI-3 is a benchmark designed to measure how well AI agents learn and reason. Agents explore unfamiliar 2D games and infer how they work without explicit instructions. You can play 25 demo games at arcprize.org/tasks(opens in a new window).

ARC-AGI-3 uses an intentionally generic harness, without tools or special features. ARC’s reasoning was that a simple harness makes model shortcomings more visible and makes model comparisons more fair. Commercial developers, by contrast, optimize harnesses for each model’s features and quirks.

In gaming, GPT‑5.6 Sol has beaten Pokémon FireRed with a vision-only harness (as streamed by GPTPlaysPokemon(opens in a new window)), Slay the Spire with Codex computer use (as streamed by EpochAI(opens in a new window)), and the first stages of Baba Is You (as shared by Piotr Migdał & Piotr Grabowski(opens in a new window)). What was so different about ARC-AGI-3?

Inspired by ARC’s analysis of GPT‑5.5’s shortcomings(opens in a new window), we examined some of the GPT‑5.6 Sol’s attempts. Like ARC, we saw that the model didn’t appear too bright. It dwelled a long time on each action and struggled to make progress.

But as we looked deeper, we discovered much of the model’s confusion was not inherent to the model itself, but due to settings in the harness.

First, we noticed that after each game action, all private reasoning was discarded. This meant that with each action, GPT‑5.6 Sol was asked to figure out the game anew, unable to remember its past thinking. The model could still see a record of past moves and brief accompanying notes, but it could not see the plans, insights, or thoughts that led to them.

Second, we saw that the harness used a rolling truncation window, causing older actions to become invisible as the history grew. So not only was GPT‑5.6 Sol unable to remember its past thinking, it was losing memory of its past actions too.

Together, these two features of the harness—discarding reasoning and rolling truncation—helped explain why GPT‑5.6 Sol was struggling to learn over time.

Our models are trained to think with private reasoning messages before they output replies or tool calls. These private thinking messages are retained as part of the conversation history. If a conversation grows too long, we summarize it and continue.

This is how our models are trained, and also how they are deployed in ChatGPT and Codex. To better match our production setup, we implemented the ARC-AGI-3 harness with our Responses API(opens in a new window). Our API makes it easy to manage context: for GPT‑5.6, passing the previous response ID automatically retains reasoning across tool calls and turns.

With reasoning retained, we noticed two big changes. First, GPT‑5.6 Sol spent less time thinking before each action, because it no longer had to interpret the game from scratch every turn. Second, when it was able to remember its past thoughts, GPT‑5.6 Sol was much better at learning over time and employing coherent strategies.

The next improvement came from replacing rolling truncation with compaction(opens in a new window), another setting in the Responses API.

The ARC-AGI-3 harness addresses context limits with rolling truncation. When the conversation context exceeds 175,000 characters, the oldest messages are discarded.

Rolling truncation has two drawbacks. First, the model loses earlier observations and actions. Second, it spends much of the tasks operating with a fuller context window, which can slightly impair performance.

When we enabled compaction on ARC-AGI-3, GPT‑5.6 Sol was better able to preserve what it had learned about each game across longer runs, and achieved a higher score with fewer output tokens.

To illustrate the effect of retaining reasoning and enabling compaction, here’s an animation showing GPT‑5.6 Sol’s 175K context window as it solves a series of ARC-AGI-3 puzzles with each harness.

Together, retaining reasoning and compaction allow GPT‑5.6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens.

We hope these experiments serve as a reminder that evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting. This isn’t the first time we’ve been surprised by low scores on a public benchmark and then discovered that the eval runner was using a generic harness that dropped reasoning messages.

If you’re an API developer trying to maximize performance, we recommend using the same settings that we deploy in our own products:

  • Use our Responses API, not our legacy Chat Completions API
  • Retain reasoning
  • Use compaction

And if you’re comparing models, we recommend relying on evals that use the settings above, which best match real-world use in ChatGPT and Codex.

We are grateful to ARC for their years of creative work on AGI evaluation, and for their analysis that inspired us to take a closer look here.

If you want to test your own mettle against frontier models, try the public games yourself at arcprize.org/tasks(opens in a new window).

Author

Keep reading

View all