GPT 6 Sol medium vs Haiku 5.5 low
GPT 6 Sol medium leads on quality; Haiku 5.5 low on speed and cost.
- Quality
- +8.6 pts
- GPT 6 Sol medium
- 4–0 head to head, 19 tied
- Not yet conclusive across all 136 comparisons on the board.
- Speed
- 1.6× faster
- Haiku 5.5 low
- 9.9 min vs 15.4 min per task
- Cost
- 46.7× cheaper
- Haiku 5.5 low
- $0.06 vs $3.01 per task
Stet score
GPT 6 Sol mediummedium reasoningRank 5 of 17
53.395% range 48.9–58.38.6 points higher
Haiku 5.5 lowlow reasoningRank 15 of 17
44.795% range 39.5–49.1
Quality
Head-to-head matchups
GPT 6 Sol medium won4
Tied19
Haiku 5.5 low won0
Task by task
GPT 6 Sol medium solved 2 tasks that Haiku 5.5 low didn’t, and Haiku 5.5 low solved 1 that GPT 6 Sol medium didn’t. Both solved 10; neither solved 11.
stet-commit-f8ae1177Solved by 10 of 17 setups
- GPT 6 Sol mediumPassed tests
- Haiku 5.5 lowFailed
A task counts as solved when its tests passed or a judge found the fix equivalent to the merged change, as in the leaderboard’s Solved column. Stet score also weighs code quality, so it can order the two differently.
All 24 compared tasks
| Task | GPT 6 Sol medium | Haiku 5.5 low | Solved by |
|---|---|---|---|
stet-3169aefa | Passed tests | Passed tests | Both |
stet-89dfbc27 | Passed tests | Passed tests | Both |
stet-b28f4fe7 | Passed tests | Passed tests | Both |
stet-232b50ba | Passed tests | Passed tests | Both |
stet-5276aa3f | Passed tests | Passed tests | Both |
stet-commit-bbbbae09 | Passed tests | Passed tests | Both |
stet-66762b4c | Passed tests | Equivalent to the merged fix | Both |
stet-commit-5d766dde | Equivalent to the merged fix | Equivalent to the merged fix | Both |
stet-43493c52 | Passed tests | Passed tests | Both |
stet-commit-f0caded9 | Equivalent to the merged fix | Equivalent to the merged fix | Both |
stet-commit-f8ae1177 | Passed tests | Failed | GPT 6 Sol medium only |
stet-15439c21 | Failed | Equivalent to the merged fix | Haiku 5.5 low only |
stet-2450ca2d | Failed | Failed | Neither |
stet-commit-342d0ac0 | Equivalent to the merged fix | Failed | GPT 6 Sol medium only |
stet-6f84e978 | Failed | Failed | Neither |
stet-commit-60a0e6c2 | Failed | Failed | Neither |
stet-commit-5e075641 | Failed | Failed | Neither |
stet-commit-2826765e | Failed | Failed | Neither |
stet-commit-a8716b88 | Failed | Failed | Neither |
stet-commit-e928166f | Failed | Failed | Neither |
stet-commit-30506f8e | Failed | Failed | Neither |
stet-commit-87814583 | Failed | Failed | Neither |
stet-commit-d2ab0c25 | Failed | Failed | Neither |
stet-commit-e41c2e2d | Failed | Failed | Neither |
Side by side
| Metric | GPT 6 Sol medium | Haiku 5.5 low |
|---|---|---|
| Reasoning effort | medium | low |
| Rank | 5 of 17 | 15 of 17 |
| Stet score | 53.395% range 48.9–58.3 | 44.795% range 39.5–49.1 |
| Solved | 12/249 passed, 3 judged equivalent | 11/247 passed, 4 judged equivalent |
| Cost / task | $3.01 | $0.06 |
| Agent time / task | 15.4 min | 9.9 min |
See which setup wins on your repo
Stet replays your merged PRs, scores quality above pass/fail, and delivers comparison reports through your agent.
Get startedAbout this data
- 24 tasks, each a merged change replayed from the repo as it was just before the merge.
- Data from .
- Both setups come from the same leaderboard run of 17 setups:
bmk_d4ba310fdf50db4ccb2092e3.