Four months. That is how long Claude Opus 4.8 held the top spot in our production agent. We tested every frontier model release against it. Nothing beat it. Until GPT-5.6 Sol showed up.
Here is the full story of what happened when we actually ran the migration, not just on benchmarks but in the real workflow that powers Ploy workspaces: building and editing actual marketing websites, reading codebases, generating imagery, and deciding when the work is done.
The Numbers That Started the Conversation
After cleaning up our eval harness, we ran head-to-head tests on our redesign suite, where the agent rebuilds a brand’s homepage against a reference design.
| Metric | Claude Opus 4.8 | GPT-5.6 Sol |
|---|---|---|
| Wall-clock time | 8m 00s | 3m 42s |
| Cost per build | $3.06 | $2.22 |
| Output tokens | 33.0K | 17.1K |
| Visual quality score | 0.936 | 0.970 |
2.2 times faster, 27 percent cheaper, and a higher quality score on top of it. That is not a marginal improvement. That is the kind of number that makes you drop everything and actually do the migration.
The Bug That Was Not in the Model
First impressions were not all smooth. GPT-5.6 fills in default values for all 25 tool parameters. That sounds harmless, but it means 52 to 64 percent of file reads returned empty results when the agent thought it was reading actual content.
We tried adding instructions to the system prompt. We tried OpenAI strict mode. Neither fixed it. The behavior is baked into how GPT-5.6 handles tool schemas at the API level.
The workaround was straightforward, but it required going back and retesting every eval case we had already marked as failed. Which led us to the second surprise.
The Eval Harness Was Biased Against the New Model
After we dug into the results, a pattern emerged that nobody on the team had anticipated: our eval harness was tuned for Claude Opus, and we did not know it.
Opus uses sequential tool calls. GPT-5.6 fans out parallel calls and burns through our tool-call budgets on cases it is solving correctly. Our executor did not support batched file reads, which Opus rarely used and GPT-5.6 uses constantly. Roughly one-third of the raw failures in our first cross-model run traced back to harness assumptions, not model behavior.
The lesson: if you are evaluating a new model against an incumbent, triage the full traces before you trust the pass rate. Otherwise you are grading the new model on how well it imitates the old one.
What the Token Difference Actually Looks Like
One matched pair from our test suite illustrated the token gap better than any aggregate number. For a comparable page build, Opus produced a 17,957-character globals.css with 174 CSS variables. Most of those variables were color ramps that never got used. GPT-5.6 produced 2,508 characters and 45 variables for the same rendered result, and in some cases the GPT-5.6 version looked better.
That is the real story behind the 48 percent token reduction. GPT-5.6 does not just use fewer tokens. It writes leaner code that achieves the same or better output.
Where Each Model Still Has an Edge
Based on our migration experience, here is the practical breakdown for anyone running AI agents in production.
GPT-5.6 Sol wins on speed, cost, and coding efficiency. The 2.2 times speed improvement and 27 percent cost reduction compound in production environments where your agent runs hundreds of tasks per day. The leaner token usage means lower API bills and faster iteration cycles.
Claude Opus 4.8 still holds some advantages in specific scenarios. If you are running agentic workflows that rely heavily on sequential tool calls and your harness is already optimized for that pattern, the migration cost may not be worth it yet. Opus’s extended thinking mode also remains strong for complex scientific reasoning tasks that benefit from deeper deliberation.
The Migration Checklist We Wish We Had
Before you trust any benchmark numbers: audit your eval harness for model-specific assumptions. Check tool-call budgets, batched read support, and scoring thresholds.
Before you diagnose failures: trace every failed case back to its root cause before blaming the model. In our case, one-third of the failures were harness issues.
Before you go to production: add explicit null/default handling for every tool parameter. GPT-5.6 fills what other models leave empty.
Do not expect plug-and-play. Switching frontier models is not just an API call change. It surfaces every assumption your stack has quietly made about the old model.
My Take
We switched. GPT-5.6 Sol is now the default powering every Ploy workspace. The numbers justified the migration on their own, but what surprised me more was the harness audit. We spent two weeks re-evaluating cases we had written off, and a third of them turned out to be false failures.
If you are evaluating GPT-5.6 against your current model, do not stop at the first run of results. Audit your evals, fix your assumptions, and run again. The real gap might be smaller than the first numbers suggest, or bigger in the direction you did not expect.
For teams running AI agents at scale, GPT-5.6 Sol is worth serious consideration. The economics of 27 percent cost reduction and 2.2 times speed improvement are real and compound at production volume.
If you are thinking about AI agent architecture in general, also worth reading about setting up Codex from zero to production – relevant context for anyone building agentic workflows. And for a broader look at how different models handle agent tasks, this breakdown of video agent skills covers practical patterns across multiple model families.
