GPT 6.1 Sol medium vs Haiku 5.5 low
Haiku 5.5 low leads on speed and cost; quality is too close to call.
- Quality
- +9.4 pts
- GPT 6.1 Sol medium
- 4–0 head to head, 18 tied
- Speed
- 1.2× faster
- Haiku 5.5 low
- 9.9 min vs 11.8 min per task
- Cost
- 17.1× cheaper
- Haiku 5.5 low
- $0.06 vs $1.10 per task
Stet score
GPT 6.1 Sol mediummedium reasoningRank 4 of 17
54.195% range 47.7–61.59.4 points higher
Haiku 5.5 lowlow reasoningRank 15 of 17
44.795% range 39.5–49.1
Quality
Head-to-head matchups
GPT 6.1 Sol medium won4
Tied18
Haiku 5.5 low won0
Task by task
GPT 6.1 Sol medium solved 2 tasks that Haiku 5.5 low didn’t, and Haiku 5.5 low solved 2 that GPT 6.1 Sol medium didn’t. Both solved 9; neither solved 10.
stet-commit-f0caded9Solved by 12 of 17 setups
- GPT 6.1 Sol mediumFailed
- Haiku 5.5 lowEquivalent to the merged fix
A task counts as solved when its tests passed or a judge found the fix equivalent to the merged change, as in the leaderboard’s Solved column. Stet score also weighs code quality, so it can order the two differently. 1 of the 24 tasks is left out because one of the two has no scored result on it.
All 23 compared tasks
| Task | GPT 6.1 Sol medium | Haiku 5.5 low | Solved by |
|---|---|---|---|
stet-3169aefa | Passed tests | Passed tests | Both |
stet-89dfbc27 | Passed tests | Passed tests | Both |
stet-b28f4fe7 | Passed tests | Passed tests | Both |
stet-232b50ba | Passed tests | Passed tests | Both |
stet-5276aa3f | Passed tests | Passed tests | Both |
stet-commit-bbbbae09 | Passed tests | Passed tests | Both |
stet-66762b4c | Passed tests | Equivalent to the merged fix | Both |
stet-commit-5d766dde | Equivalent to the merged fix | Equivalent to the merged fix | Both |
stet-43493c52 | Passed tests | Passed tests | Both |
stet-commit-f0caded9 | Failed | Equivalent to the merged fix | Haiku 5.5 low only |
stet-commit-f8ae1177 | Passed tests | Failed | GPT 6.1 Sol medium only |
stet-15439c21 | Failed | Equivalent to the merged fix | Haiku 5.5 low only |
stet-2450ca2d | Failed | Failed | Neither |
stet-commit-342d0ac0 | Equivalent to the merged fix | Failed | GPT 6.1 Sol medium only |
stet-6f84e978 | Failed | Failed | Neither |
stet-commit-60a0e6c2 | Failed | Failed | Neither |
stet-commit-5e075641 | Failed | Failed | Neither |
stet-commit-2826765e | Failed | Failed | Neither |
stet-commit-a8716b88 | Failed | Failed | Neither |
stet-commit-e928166f | Failed | Failed | Neither |
stet-commit-87814583 | Failed | Failed | Neither |
stet-commit-d2ab0c25 | Failed | Failed | Neither |
stet-commit-e41c2e2d | Failed | Failed | Neither |
Side by side
| Metric | GPT 6.1 Sol medium | Haiku 5.5 low |
|---|---|---|
| Reasoning effort | medium | low |
| Rank | 4 of 17 | 15 of 17 |
| Stet score | 54.195% range 47.7–61.5 | 44.795% range 39.5–49.1 |
| Solved | 11/249 passed, 2 judged equivalent | 11/247 passed, 4 judged equivalent |
| Cost / task | $1.10 | $0.06 |
| Agent time / task | 11.8 min | 9.9 min |
See which setup wins on your repo
Stet replays your merged PRs, scores quality above pass/fail, and delivers comparison reports through your agent.
Get startedAbout this data
- 24 tasks, each a merged change replayed from the repo as it was just before the merge.
- Data from .
- Both setups come from the same leaderboard run of 17 setups:
bmk_d4ba310fdf50db4ccb2092e3.