GPT-5.6's 3x benchmark jump was a harness story

OpenAI has published one of the clearest demonstrations yet that agent performance is not only a model property.

On the ARC-AGI-3 public task set, OpenAI says GPT-5.6 Sol scored 13.3% with the official harness. When its team retained reasoning between turns and replaced rolling truncation with compaction, the score rose to 38.3%.

That is roughly three times the score with six times fewer output tokens.

The model did not change. The surrounding system did.

The benchmark was also measuring the harness

ARC-AGI-3 asks agents to explore unfamiliar 2D games and infer how they work without explicit instructions. It is meant to test learning and reasoning, not simple recall.

The official harness was intentionally generic. That makes comparisons easier, but OpenAI found two design choices that worked against the way GPT-5.6 Sol had been trained and deployed.

First, the harness discarded reasoning state after every game action. The model could still see past moves and short notes, but not the internal state that produced its current plan. Each new action forced it to reconstruct more of the problem.

Second, the harness used a rolling truncation window. As the interaction grew, older observations and actions disappeared. The result was an agent that could act, but struggled to learn coherently over time.

This is not just a benchmark issue. It is a familiar failure mode in production agent systems.

Retained reasoning changed the loop

OpenAI rebuilt the evaluation using its Responses API. For GPT-5.6, chaining the previous response can retain supported reasoning state across tool calls and turns.

OpenAI reported that once reasoning was retained, the model spent less time reinterpreting the game and became better at carrying a coherent strategy forward.

This does not mean exposing private chain-of-thought to the user. It means preserving the model’s supported reasoning state inside the API workflow so the next step does not begin cold.

The useful design question is not whether a system can store more chat history. It is whether it preserves the state the model needs to continue the task correctly.

Long-running agents eventually approach a context limit. Something has to happen to older state.

The official ARC harness dropped the oldest messages when its rolling window filled. OpenAI instead enabled compaction.

OpenAI’s API documentation describes compaction as a way to reduce context size while carrying forward key prior state and reasoning in fewer tokens. Developers can enable server-side compaction with a threshold or explicitly call the compact endpoint.

OpenAI says this allowed GPT-5.6 Sol to preserve what it had learned across longer runs while using fewer output tokens.

That is a stronger pattern than retaining everything forever. Good agent memory is not unlimited accumulation. It is selective continuity: preserve the right state, remove redundant weight and keep the next action grounded in what has already happened.

The claim needs the right boundary

This was an OpenAI-run experiment using OpenAI’s model, API and preferred settings. It is publisher-reported benchmark evidence, not independent validation by ChaseInTech.

The result also does not prove that every agent workflow will triple its performance. ARC-AGI-3 is a specific environment. The model, task, scoring method, tool loop and context pattern all matter.

But the experiment establishes a testable engineering point:

Before blaming or replacing the model, measure what your harness is making it forget.

What I would test in a real agent stack

Run the same operational task twice with the same model, prompt, tools and success criteria.

In the first run, use the current default state and truncation behaviour. In the second, preserve supported reasoning state and compact context deliberately.

Compare completed outcomes rather than message quality:

Output tokens and total cost
Latency per successful task
Retries and recovery events
Consistency across long tool loops
Final-artifact quality
Evidence available for human review

If the second workflow performs materially better, the bottleneck was not only model intelligence. It was orchestration.

Why this matters for ChaseOS

I am building ChaseOS as an agent harness and operating layer rather than a wrapper around one model provider.

The model should remain replaceable. The system around it must preserve authority, state, evidence, approvals and recovery.

That means treating context management as infrastructure, not a prompt-engineering detail. A stronger model can still underperform inside a weak harness. A well-designed harness can unlock capability that a generic evaluation or stateless workflow leaves behind.

The wider operator lesson

The next generation of agent systems will not be differentiated only by which frontier model they call.

They will be differentiated by what they preserve, what they compact, what they verify and what they can recover when a long-running task goes wrong.

GPT-5.6 supplied the headline number. The harness supplied the builder lesson.