Haiku 5.5 max vs Opus 4.8 medium
Opus 4.8 medium leads on speed; Haiku 5.5 max on cost; quality is too close to call.
- Quality
- +0.6 pts
- Opus 4.8 medium
- 5–5 head to head, 12 tied
- Speed
- 2.5× faster
- Opus 4.8 medium
- 23.3 min vs 58.4 min per task
- Cost
- 8.0× cheaper
- Haiku 5.5 max
- $0.78 vs $6.27 per task
Stet score
Haiku 5.5 maxmax reasoningRank 12 of 17
48.295% range 38.8–58.1
Opus 4.8 mediummedium reasoningRank 11 of 17
48.895% range 39.7–58.00.6 points higher
Quality
Head-to-head matchups
Haiku 5.5 max won5
Tied12
Opus 4.8 medium won5
Task by task
Haiku 5.5 max solved 3 tasks that Opus 4.8 medium didn’t, and Opus 4.8 medium solved 5 that Haiku 5.5 max didn’t. Both solved 7; neither solved 8.
stet-232b50baSolved by 16 of 17 setups
- Haiku 5.5 maxFailed
- Opus 4.8 mediumPassed tests
A task counts as solved when its tests passed or a judge found the fix equivalent to the merged change, as in the leaderboard’s Solved column. Stet score also weighs code quality, so it can order the two differently. 1 of the 24 tasks is left out because one of the two has no scored result on it.
All 23 compared tasks
| Task | Haiku 5.5 max | Opus 4.8 medium | Solved by |
|---|---|---|---|
stet-3169aefa | Passed tests | Passed tests | Both |
stet-89dfbc27 | Passed tests | Passed tests | Both |
stet-b28f4fe7 | Passed tests | Passed tests | Both |
stet-232b50ba | Failed | Passed tests | Opus 4.8 medium only |
stet-5276aa3f | Passed tests | Passed tests | Both |
stet-commit-bbbbae09 | Passed tests | Passed tests | Both |
stet-66762b4c | Failed | Passed tests | Opus 4.8 medium only |
stet-commit-5d766dde | Equivalent to the merged fix | Equivalent to the merged fix | Both |
stet-43493c52 | Failed | Passed tests | Opus 4.8 medium only |
stet-commit-f0caded9 | Failed | Equivalent to the merged fix | Opus 4.8 medium only |
stet-commit-f8ae1177 | Failed | Passed tests | Opus 4.8 medium only |
stet-15439c21 | Passed tests | Failed | Haiku 5.5 max only |
stet-2450ca2d | Passed tests | Failed | Haiku 5.5 max only |
stet-commit-342d0ac0 | Equivalent to the merged fix | Failed | Haiku 5.5 max only |
stet-6f84e978 | Equivalent to the merged fix | Passed tests | Both |
stet-commit-5e075641 | Failed | Failed | Neither |
stet-commit-2826765e | Failed | Failed | Neither |
stet-commit-a8716b88 | Failed | Failed | Neither |
stet-commit-e928166f | Failed | Failed | Neither |
stet-commit-30506f8e | Failed | Failed | Neither |
stet-commit-87814583 | Failed | Failed | Neither |
stet-commit-d2ab0c25 | Failed | Failed | Neither |
stet-commit-e41c2e2d | Failed | Failed | Neither |
Side by side
| Metric | Haiku 5.5 max | Opus 4.8 medium |
|---|---|---|
| Reasoning effort | max | medium |
| Rank | 12 of 17 | 11 of 17 |
| Stet score | 48.295% range 38.8–58.1 | 48.895% range 39.7–58.0 |
| Solved | 10/247 passed, 3 judged equivalent | 13/2411 passed, 2 judged equivalent |
| Cost / task | $0.78 | $6.27 |
| Agent time / task | 58.4 min | 23.3 min |
See which setup wins on your repo
Stet replays your merged PRs, scores quality above pass/fail, and delivers comparison reports through your agent.
Get startedAbout this data
- 24 tasks, each a merged change replayed from the repo as it was just before the merge.
- Data from .
- Both setups come from the same leaderboard run of 17 setups:
bmk_d4ba310fdf50db4ccb2092e3.