Stet compares coding models on real merged-code tasks, then scores the patches above test pass rate: equivalence, code review, footprint risk, runtime, and cost.
Public comparisons show how model and harness choices behaved on these repos. A private Stet run answers the same question with your PR history, tests, AGENTS.md, and review standards. Compare Claude Code vs Codex on your repo.
30 tasks · GraphQL Go Tools and SQLParser Rust
Fable 5 led the cleaned n=30 local score, but the taskwise metric ledger points to routing: pay for Fable on hidden semantic and patch-shape risk, keep cheaper models in the loop for narrower work.
56 tasks · Zod and graphql-go-tools
GPT-5.5 beat GPT-5.4 on tests, equivalence, review pass, and clean pass across 56 real coding tasks.
56 tasks · Zod and graphql-go-tools
GPT-5.5 was the better shipping default, while Opus 4.7 remained the lower-footprint option.
56 tasks · Zod and graphql-go-tools
GPT-5.4 matched the human patch more often than Opus 4.7, but Opus kept patches smaller and lower-risk.
28 tasks · Zod
Opus 4.7 did not pass more tests, but it produced tighter and more equivalent patches.
26 tasks · graphql-go-tools
GPT-5.5 Codex at four reasoning settings on 26 matched graphql-go-tools tasks: equivalence and review pass climbed sharply with reasoning, while tests were not monotonic.
29 tasks · graphql-go-tools
Claude Opus 4.7 peaked at medium reasoning effort on 29 matched graphql-go-tools tasks; high, xhigh, and max were more expensive without improving the primary rollout metrics.
GPT-5.4 was stronger on equivalence in the 56-task benchmark: 35 of 56 patches matched the human change, versus 19 of 56 for Opus 4.7. Opus had the lower footprint risk, 0.20 versus 0.34, which makes it the more conservative patch writer.
| Metric | Opus 4.7 | GPT-5.4 |
|---|---|---|
| Tests pass | 33 / 56 | 31 / 56 |
| Equivalence | 19 / 56 | 35 / 56 |
| Clean pass | 10 / 56 | 11 / 56 |
| Footprint risk | 0.20 | 0.34 |