Skip to the comparison

GPT 6 Sol medium vs Haiku 5.5 max

GPT 6 Sol medium leads on speed; Haiku 5.5 max on cost; quality is too close to call.

Quality
+5.1 pts
GPT 6 Sol medium
4–3 head to head, 15 tied
Speed
3.8× faster
GPT 6 Sol medium
15.4 min vs 58.4 min per task
Cost
3.9× cheaper
Haiku 5.5 max
$0.78 vs $3.01 per task

24 tasksData from

Stet score

GPT 6 Sol mediummedium reasoningRank 5 of 17

53.395% range 48.9–58.35.1 points higher

Haiku 5.5 maxmax reasoningRank 12 of 17

48.295% range 38.8–58.1

A directional read: their 95% score ranges overlap.

Quality

Head-to-head matchups

GPT 6 Sol medium won4

Tied15

Haiku 5.5 max won3

2 undecided
GPT 6 Sol medium’s win share 53%, 95% range 41–65%Ties are within judge noise.

Task by task

GPT 6 Sol medium solved 5 tasks that Haiku 5.5 max didn’t, and Haiku 5.5 max solved 3 that GPT 6 Sol medium didn’t. Both solved 7; neither solved 8.

stet-232b50baSolved by 16 of 17 setups

  • GPT 6 Sol mediumPassed tests
  • Haiku 5.5 maxFailed

A task counts as solved when its tests passed or a judge found the fix equivalent to the merged change, as in the leaderboard’s Solved column. Stet score also weighs code quality, so it can order the two differently. 1 of the 24 tasks is left out because one of the two has no scored result on it.

All 23 compared tasks
Task by task
TaskGPT 6 Sol mediumHaiku 5.5 maxSolved by
stet-3169aefaPassed testsPassed testsBoth
stet-89dfbc27Passed testsPassed testsBoth
stet-b28f4fe7Passed testsPassed testsBoth
stet-232b50baPassed testsFailedGPT 6 Sol medium only
stet-5276aa3fPassed testsPassed testsBoth
stet-commit-bbbbae09Passed testsPassed testsBoth
stet-66762b4cPassed testsFailedGPT 6 Sol medium only
stet-commit-5d766ddeEquivalent to the merged fixEquivalent to the merged fixBoth
stet-43493c52Passed testsFailedGPT 6 Sol medium only
stet-commit-f0caded9Equivalent to the merged fixFailedGPT 6 Sol medium only
stet-commit-f8ae1177Passed testsFailedGPT 6 Sol medium only
stet-15439c21FailedPassed testsHaiku 5.5 max only
stet-2450ca2dFailedPassed testsHaiku 5.5 max only
stet-commit-342d0ac0Equivalent to the merged fixEquivalent to the merged fixBoth
stet-6f84e978FailedEquivalent to the merged fixHaiku 5.5 max only
stet-commit-5e075641FailedFailedNeither
stet-commit-2826765eFailedFailedNeither
stet-commit-a8716b88FailedFailedNeither
stet-commit-e928166fFailedFailedNeither
stet-commit-30506f8eFailedFailedNeither
stet-commit-87814583FailedFailedNeither
stet-commit-d2ab0c25FailedFailedNeither
stet-commit-e41c2e2dFailedFailedNeither

Side by side

Side by side
MetricGPT 6 Sol mediumHaiku 5.5 max
Reasoning effortmediummax
Rank5 of 1712 of 17
Stet score53.395% range 48.9–58.348.295% range 38.8–58.1
Solved12/249 passed, 3 judged equivalent10/247 passed, 3 judged equivalent
Cost / task$3.01$0.78
Agent time / task15.4 min58.4 min

See which setup wins on your repo

Stet replays your merged PRs, scores quality above pass/fail, and delivers comparison reports through your agent.

Get started