GPT 5.6 Luna max vs GPT 6.1 Sol medium
GPT 6.1 Sol medium leads on speed and cost; quality is too close to call.
- Quality
- +7.8 pts
- GPT 5.6 Luna max
- 11–6 head to head, 3 tied
- Speed
- 2.4× faster
- GPT 6.1 Sol medium
- 11.8 min vs 28.5 min per task
- Cost
- 4.1× cheaper
- GPT 6.1 Sol medium
- $1.10 vs $4.49 per task
Stet score
GPT 5.6 Luna maxcodex · max reasoningRank 2 of 16
70.395% range 57.6–82.27.8 points higher
GPT 6.1 Sol mediumcodex · medium reasoningRank 4 of 16
62.595% range 53.0–71.7
Quality
Head-to-head matchups
GPT 5.6 Luna max won11
Tied34 undecided
GPT 6.1 Sol medium won6
Task by task
GPT 5.6 Luna max solved 0 tasks that GPT 6.1 Sol medium didn’t, and GPT 6.1 Sol medium solved 2 that GPT 5.6 Luna max didn’t. Both solved 10; neither solved 8.
On the 10 tasks both solved, judges preferred GPT 5.6 Luna max’s patch on 6, GPT 6.1 Sol medium’s on 1, and called 3 even.
stet-15439c21Solved by 13 of 16 setups
- GPT 5.6 Luna maxFailed
- GPT 6.1 Sol mediumPassed tests
Judges preferred GPT 6.1 Sol medium’s patch
A task counts as solved when its tests passed or a judge found the fix equivalent to the merged change, as in the leaderboard’s Solved column. Stet score also weighs code quality, so it can order the two differently. 4 of the 24 tasks are left out because one of the two has no scored result on them.
All 20 compared tasks
| Task | GPT 5.6 Luna max | GPT 6.1 Sol medium | Solved by | Better patch |
|---|---|---|---|---|
stet-3169aefa | Passed tests | Passed tests | Both | Even |
stet-89dfbc27 | Passed tests | Passed tests | Both | Even |
stet-b28f4fe7 | Passed tests | Passed tests | Both | GPT 5.6 Luna max |
stet-232b50ba | Passed tests | Passed tests | Both | GPT 5.6 Luna max |
stet-5276aa3f | Passed tests | Passed tests | Both | GPT 5.6 Luna max |
stet-commit-bbbbae09 | Passed tests | Passed tests | Both | GPT 5.6 Luna max |
stet-66762b4c | Equivalent to the merged fix | Passed tests | Both | GPT 6.1 Sol medium |
stet-15439c21 | Failed | Passed tests | GPT 6.1 Sol medium only | GPT 6.1 Sol medium |
stet-commit-5d766dde | Equivalent to the merged fix | Equivalent to the merged fix | Both | GPT 5.6 Luna max |
stet-43493c52 | Passed tests | Passed tests | Both | Even |
stet-6f84e978 | Failed | Equivalent to the merged fix | GPT 6.1 Sol medium only | GPT 6.1 Sol medium |
stet-2450ca2d | Failed | Failed | Neither | GPT 5.6 Luna max |
stet-commit-342d0ac0 | Equivalent to the merged fix | Equivalent to the merged fix | Both | GPT 5.6 Luna max |
stet-commit-60a0e6c2 | Failed | Failed | Neither | GPT 6.1 Sol medium |
stet-commit-5e075641 | Failed | Failed | Neither | GPT 5.6 Luna max |
stet-commit-a8716b88 | Failed | Failed | Neither | GPT 5.6 Luna max |
stet-commit-e928166f | Failed | Failed | Neither | GPT 6.1 Sol medium |
stet-commit-30506f8e | Failed | Failed | Neither | GPT 5.6 Luna max |
stet-commit-d2ab0c25 | Failed | Failed | Neither | GPT 5.6 Luna max |
stet-commit-e41c2e2d | Failed | Failed | Neither | GPT 6.1 Sol medium |
Side by side
| Metric | GPT 5.6 Luna max | GPT 6.1 Sol medium |
|---|---|---|
| Harness | codex | codex |
| Reasoning effort | max | medium |
| Rank | 2 of 16 | 4 of 16 |
| Stet score | 70.395% range 57.6–82.2 | 62.595% range 53.0–71.7 |
| Solved | 10/247 passed, 3 judged equivalent | 13/2410 passed, 3 judged equivalent |
| Cost / task | $4.49 | $1.10 |
| Agent time / task | 28.5 min | 11.8 min |
See which setup wins on your repo
Stet replays your merged PRs, scores quality above pass/fail, and delivers comparison reports through your agent.
Run it on your repoAbout this data
- 24 tasks, each a merged change replayed from the repo as it was just before the merge.
- Data from .
- Both setups come from the same leaderboard run of 16 setups:
bmk_d4ba310fdf50db4ccb2092e3.