When Ploy, an AI-powered marketing agent, switched its production model from Claude Opus to GPT-5.6 Sol, the headline numbers looked great: builds finishing in half the wall-clock time at 27% lower cost. Behind those numbers lay weeks of work fixing harness assumptions, tool call schemas, prompt caching, and reasoning replay. The GPT-5.6 migration journey reveals what every team building AI agents needs to know about swapping models in production.

This article distills the engineering lessons from a real production migration, covering the four critical areas where provider differences materialized and how they were resolved.

Background: The Challenge

Ploy's agent builds and edits real marketing websites end to end. It plans pages, reads codebases, writes components, generates imagery, screenshots its own work, and decides when it is done. That workflow sets a very high quality bar. For four months, Claude Opus held the default slot because nothing else could match its output.

When OpenAI released GPT-5.6 Sol, the initial evaluation was promising but not straightforward. Raw failures in the first cross-model run traced back as much to testing harness assumptions as to model behavior. Roughly a third of the initial failures were false negatives caused by tool-call budgets sized for Opus's sequential style and an eval executor that did not support batched file reads.

How the Migration Unfolded

Step 1: Fixing Tool Call Schemas

The first blocker was subtle. GPT-5.6 sends every optional parameter on every tool call, inventing default values for the ones it does not need. For Ploy's code tool with 25 parameters, this meant every read call carried an offset of 0, a timeout of 120000, and a dummy site ID.

The result was catastrophic. The file-read implementation treated these invented values as real, causing 52% to 64% of GPT-5.6's file reads to come back empty. The tool returned success every time, so the model had no way to detect the corruption.

Behavior	Claude Opus 4.8	GPT-5.6 Sol
Carries all 25 optional params	0.1% of calls	100% of calls
Empty file reads from invented values	Rare	52-64%
Prompting fixes it	N/A	No

The fix was a schema transform at the provider boundary. Every optional property was rewritten as required but nullable using anyOf: [T, null], giving the model an explicit way to signal unused parameters. A seam at the tool invocation layer strips the nulls before validation so no tool implementation changes.

Results: empty file reads dropped to 0%, and tool call count fell by roughly 30%.

Step 2: Rebuilding Prompt Caching

This was the most instructive engineering difference, because both providers offer "prompt caching" but the implementations are completely different.

Claude's cache is org-scoped with a shared static prefix that hits at 92% to 96% across all conversations. GPT-5.6 uses keyed cache nodes with a throughput of roughly 15 requests per minute per key. The right key scope matters enormously:

• Per-conversation key: first-call hit rate of 0%
• Single global key: production traffic obliterates the 15 rpm budget
• Workspace-scoped key: first-call hit rate reached 83.7%

The solution required splitting the system prompt into breakpointed layers and using a workspace-scoped cache key. Cross-workspace sharing of the static prefix is structurally impossible on OpenAI, meaning every workspace pays one cold write per idle window. This is a real cost, but bounded and predictable.

Results after the fix: total uncached input tokens dropped 28%, and GPT-5.6's per-suite cost landed below Opus. "If you are cost-comparing models and one of them has a cold cache, you are comparing your config, not the models."

Step 3: Reasoning Replay

GPT-5.6's Responses API replays prior-turn reasoning as server-side item references by default. These references started failing mid-conversation with item-not-found errors. The fix was store: false, which requests encrypted reasoning content as self-contained blobs instead of pointers to server state.

Performance Benchmark

Metric	Claude Opus 4.8	GPT-5.6 Sol	Improvement
Cost per completed build	$3.06	$2.22	27% cheaper
Wall-clock time	8m 00s	3m 42s	2.2x faster
Input tokens per build	2.60M	1.70M	35% fewer
Output tokens per build	33.0K	17.1K	48% fewer
Visual score (0-1)	0.936	0.970	3.6% better

Frequently Asked Questions

Is GPT-5.6 Sol ready for production agent workloads? Yes, but with caveats. The model excels at clean, modern layouts and writes leaner code than Claude Opus. However, it requires schema transforms for tool calls and explicitly designed cache strategies to reach cost parity.

How does GPT-5.6 Luna compare for cost-sensitive workloads? Luna is significantly cheaper. One commenter noted you can get five Luna runs for the cost of one Sol run, making it attractive for sub-tasks that do not require Sol's orchestration capability.

Does switching models require changing prompts? Yes, and this is often underestimated. The system prompt and tool-calling patterns that work for one model may produce entirely different behavior on another. Ploy's experience shows that even minor differences in how a model handles optional parameters can corrupt results silently.

What is the biggest mistake teams make when migrating AI agents? Not fixing the evaluation harness first. Ploy found that roughly a third of initial cross-model failures traced back to harness assumptions tuned to the incumbent model, not the challenger's behavior.

Can I use the same prompt caching strategy across providers? No. Claude and OpenAI implement prompt caching fundamentally differently with respect to scope, keying, and throughput. Copying a caching strategy from one provider to the other without understanding these differences will produce incorrect cost comparisons.

Best Practices for AI Agent Migration

1. Fix your eval harness first before trusting a single benchmark number. Triage traces to distinguish harness artifacts from model behavior
2. Check tool call behavior early in the migration. Watch for invented optional parameters and schema compliance differences
3. Design caching for the provider's architecture, not yours. Understand scope, keying, and throughput limits before configuring
4. Test reasoning replay mechanics if your agent uses multi-turn conversations with reasoning traces
5. Compare apples to apples on cost by ensuring cache configurations are fair before making pricing decisions

Key Takeaways

• Migrating AI agents between model providers requires more than switching an API key
• GPT-5.6 Sol offers meaningful cost and performance improvements over Claude Opus when configured correctly
• Tool call schemas, prompt caching, and reasoning replay are the three areas where provider differences bite hardest
• Always triage eval failures before trusting cross-model benchmarks
• A cold cache makes any model look expensive -- configure caching before comparing costs

---

Source: ploy.ai | HN Discussion