Stet compares coding models on real merged-code tasks, then scores the patches above test pass rate: equivalence, code review, footprint risk, runtime, and cost.
Public comparisons show how model and harness choices behaved on these repos. A private Stet run answers the same question with your PR history, tests, AGENTS.md, and review standards. Compare Claude Code vs Codex on your repo.
Latest evidence
Current comparisons and reasoning studies, ordered by publication date. Each writeup links the decision back to task-level evidence and the limits of the local slice.
August 25, 2026
Opus 4.8 and Opus 5 tied on strict functional score across 25 matched Stet tasks, but their paired artifacts differed: Opus 4.8 had the lower footprint on 20 tasks, while Opus 5 ran more shell commands on 18 and more tests on 15.
Claude Opus 4.8 · Claude Opus 5
July 20, 2026
Six interventions meant to save tokens gave different answers depending on how you count. None cut the ten-task batch total twice. Caveman and the cheaper Terra model looked mildly cheaper per task, but within run-to-run noise.
GPT-5.6 Sol · GPT-5.6 Terra
July 8, 2026
On 24 real tasks, Sonnet scaled effort into more checking while Opus stayed flatter through high. The graders leaned Sonnet on clarity and Opus on diff minimality. Here is when I would use each.
Sonnet 5 · Opus 4.8
June 19, 2026
GLM 5.2 vs Composer 2.5 and the premium field on 50 real merged PRs from graphql-go-tools (Go) and sqlparser-rs (Rust). GLM lands last on craft and equivalence in both repos, costs about twice Composer, and writes more code than the human while missing the change. A routing guide for the new cheap model.
GLM 5.2 · Composer 2.5 · Opus 4.8 · GPT-5.5
June 18, 2026
Composer 2.5 vs Opus 4.8, GPT-5.5 and Opus 4.7 on 50 real merged PRs from sqlparser-rs (Rust) and graphql-go-tools (Go). Composer is 6.5-7x cheaper and ties the test gate, but finishes last on craft in both repos. A routing guide for where it's safe.
Composer 2.5 · Opus 4.8 · GPT-5.5 · Opus 4.7
June 14, 2026
Fable 5 vs Opus 4.8, GPT-5.5 and two more on 30 real GraphQL Go Tools and SQLParser Rust tasks. The useful read is taskwise W/D/L: which model won which metric, on which task, and why.
Fable 5 · Opus 4.8 · Opus 4.7 · GPT-5.5 · Composer 2.5
June 2, 2026
I graded four frontier coding models on 50 real merged PRs in Go and Rust - not just whether tests pass, but craft, equivalence, and cost. Opus 4.8 led on craft in both.
Claude Opus 4.8 · Opus 4.8 · GPT-5.5 · Claude Opus 4.7 · Opus 4.7 · Composer 2.5
May 27, 2026
Codex optimized its own AGENTS.md against real Stet repo tasks. The best candidate improved the training slice, then regressed enough on a clean holdout that it was not safe to ship.
Codex · GPT-5.5 · GPT-5.4
May 18, 2026
A fresh GPT-5.5 Codex high rerun on 21 clean GraphQL-go-tools tasks compared with the May 5 GPT-5.5 high run. The rerun was directionally worse on tests, equivalence, and review pass count, but the evidence is mixed and does not show a broad quality collapse.
GPT-5.5 · Codex
May 12, 2026
Claude Opus 4.7 reasoning-effort curve on 29 matched GraphQL-go-tools tasks: low, medium, high, xhigh, and max. Medium wins the behavioral metrics; more reasoning does not reliably buy better patches.
Claude Opus 4.7 · Opus 4.7
May 7, 2026
An interactive GPT-5.5 Codex reasoning-effort curve on 26 matched GraphQL-go-tools tasks: low, medium, high, and xhigh.
GPT-5.5 · Codex
May 1, 2026
Opus 4.7 vs GPT-5.5 vs GPT-5.4 on 56 real coding tasks across two open-source repos. Opus writes smaller patches; GPT-5.5 writes patches that more often survive review.
GPT-5.5 · GPT-5.4 · Claude Opus 4.7 · Opus 4.7
April 17, 2026
Three Opus snapshots, same 12/28 test pass rate. Above the gate, 4.7 is directionally better — more disciplined, not fundamentally smarter.
Claude Opus 4.7 · Opus 4.7 · Claude Opus 4.6 · Opus 4.6
30 tasks · GraphQL Go Tools and SQLParser Rust
Fable 5 led the cleaned n=30 local score, but the taskwise metric ledger points to routing: pay for Fable on hidden semantic and patch-shape risk, keep cheaper models in the loop for narrower work.
56 tasks · Zod and graphql-go-tools
GPT-5.5 beat GPT-5.4 on tests, equivalence, review pass, and clean pass across 56 real coding tasks.
56 tasks · Zod and graphql-go-tools
GPT-5.5 was the better shipping default, while Opus 4.7 remained the lower-footprint option.
56 tasks · Zod and graphql-go-tools
GPT-5.4 matched the human patch more often than Opus 4.7, but Opus kept patches smaller and lower-risk.
28 tasks · Zod
Opus 4.7 did not pass more tests, but it produced tighter and more equivalent patches.
26 tasks · graphql-go-tools
GPT-5.5 Codex at four reasoning settings on 26 matched graphql-go-tools tasks: equivalence and review pass climbed sharply with reasoning, while tests were not monotonic.
29 tasks · graphql-go-tools
Claude Opus 4.7 peaked at medium reasoning effort on 29 matched graphql-go-tools tasks; high, xhigh, and max were more expensive without improving the primary rollout metrics.
GPT-5.4 was stronger on equivalence in the 56-task benchmark: 35 of 56 patches matched the human change, versus 19 of 56 for Opus 4.7. Opus had the lower footprint risk, 0.20 versus 0.34, which makes it the more conservative patch writer.
| Metric | Opus 4.7 | GPT-5.4 |
|---|---|---|
| Tests pass | 33 / 56 | 31 / 56 |
| Equivalence | 19 / 56 | 35 / 56 |
| Clean pass | 10 / 56 | 11 / 56 |
| Footprint risk | 0.20 | 0.34 |