Haiku 5.5 vs Opus 4.8
Haiku 5.5 cheaper in both pairings. Opus 4.8 has a slight quality edge in 1 of 2 pairings; none is a clear lead. 2 pairings; 24 merged changes in Stet-AI/stet. Compare the individual calls below to choose which setup to investigate.
What this comparison is
- 1 real repositoryStet-AI/stet, private; not public benchmark scores
- 24 merged changeseach run end to end by the coding agent from the repo as it was before the change
- Graded against the merged changeby the repo’s own tests and by judges comparing each patch with it
- Quality, cost and timea smaller sample than public benchmarks, but real agent work on a real codebase
Compare coding modelsLow vs high vs xhigh/max reasoning for coding agents
Recorded setup pairings
Each row is a separate recorded comparison. Scores are not pooled or averaged across effort settings. W/D/L means wins, draws and losses; undecided judgments remain undecided.
Stet score: A setup’s average chance of beating another setup on a task, from 0 to 100; 50 is even with the field.
| Setup pairing | Recorded quality call and record | Quality evidence | Cost per task | Agent time per task |
|---|---|---|---|---|
| Haiku 5.5 max vs Opus 4.8 mediumHaiku 5.5 max · max reasoningOpus 4.8 medium · medium reasoning23/24 tasks observed by both setups. | No clear quality difference: Haiku 5.5 max won 5, Opus 4.8 medium won 5, 12 tied, 2 undecided; Haiku 5.5 max leads on cost: 8.0× cheaper; Opus 4.8 medium leads on agent time: 2.5× faster.Head to head, Opus 4.8 medium won 5, tied 12 and lost 5 against Haiku 5.5 max, with 2 undecided. | No clear difference 5–5 head to head, 12 tiedHaiku 5.5 max: 48.2 (95% range 38.8–58.1)Opus 4.8 medium: 48.8 (95% range 39.7–58.0) | Haiku 5.5 max: $0.78 Opus 4.8 medium: $6.27 | Haiku 5.5 max: 58.4 min Opus 4.8 medium: 23.3 min |
| Haiku 5.5 low vs Opus 4.8 mediumHaiku 5.5 low · low reasoningOpus 4.8 medium · medium reasoning24/24 tasks observed by both setups. | Opus 4.8 medium has a slight quality edge, 5–3 head to head with 15 tied and 1 undecided; Haiku 5.5 low leads on cost and agent time: 97.4× cheaper and 2.4× faster.Head to head, Opus 4.8 medium won 5, tied 15 and lost 3 against Haiku 5.5 low, with 1 undecided. | Slight edge 5–3 head to head, 15 tiedHaiku 5.5 low: 44.7 (95% range 39.5–49.1)Opus 4.8 medium: 48.8 (95% range 39.7–58.0) | Haiku 5.5 low: $0.06 Opus 4.8 medium: $6.27 | Haiku 5.5 low: 9.9 min Opus 4.8 medium: 23.3 min |
Questions about these results
- Which is better for coding, Haiku 5.5 or Opus 4.8?
- Haiku 5.5 cheaper in both pairings. Opus 4.8 has a slight quality edge in 1 of 2 pairings; none is a clear lead. 2 pairings; 24 merged changes in Stet-AI/stet. Compare the individual calls below to choose which setup to investigate.
- Where do these coding results come from?
- Both coding agents worked the same 24 merged changes from the private Stet-AI/stet repository, each in the real repo as it was before the change. The repo’s tests and judges graded every patch against the merged change; no public benchmark scores are used.
- Are reasoning settings combined into one model score?
- No. Each effort pairing keeps its own recorded quality call, head-to-head record, cost and agent time. Incompatible scores are never averaged.