Skip to the comparison

GPT 6.1 Sol medium vs GPT 6 Luna max

GPT 6.1 Sol medium leads on quality and speed; GPT 6 Luna max on cost.

Quality
+8.1 pts
GPT 6.1 Sol medium
3–0 head to head, 19 tied
Not yet conclusive across all 136 comparisons on the board.
Speed
5.1× faster
GPT 6.1 Sol medium
11.8 min vs 59.7 min per task
Cost
3.0× cheaper
GPT 6 Luna max
$0.37 vs $1.10 per task

24 tasksData from

Stet score

GPT 6.1 Sol mediummedium reasoningRank 4 of 17

54.195% range 47.7–61.58.1 points higher

GPT 6 Luna maxmax reasoningRank 14 of 17

46.095% range 40.4–51.6

A directional read: their 95% score ranges overlap.

Quality

Head-to-head matchups

GPT 6.1 Sol medium won3

Tied19

GPT 6 Luna max won0

2 undecided
GPT 6.1 Sol medium’s win share 58%, 95% range 52–66%Ties are within judge noise.

Task by task

GPT 6.1 Sol medium solved 1 task that GPT 6 Luna max didn’t, and GPT 6 Luna max solved 1 that GPT 6.1 Sol medium didn’t. Both solved 10; neither solved 11.

stet-commit-f0caded9Solved by 12 of 17 setups

  • GPT 6.1 Sol mediumFailed
  • GPT 6 Luna maxEquivalent to the merged fix

A task counts as solved when its tests passed or a judge found the fix equivalent to the merged change, as in the leaderboard’s Solved column. Stet score also weighs code quality, so it can order the two differently. 1 of the 24 tasks is left out because one of the two has no scored result on it.

All 23 compared tasks
Task by task
TaskGPT 6.1 Sol mediumGPT 6 Luna maxSolved by
stet-3169aefaPassed testsPassed testsBoth
stet-89dfbc27Passed testsPassed testsBoth
stet-b28f4fe7Passed testsPassed testsBoth
stet-232b50baPassed testsPassed testsBoth
stet-5276aa3fPassed testsPassed testsBoth
stet-commit-bbbbae09Passed testsPassed testsBoth
stet-66762b4cPassed testsEquivalent to the merged fixBoth
stet-commit-5d766ddeEquivalent to the merged fixEquivalent to the merged fixBoth
stet-43493c52Passed testsPassed testsBoth
stet-commit-f0caded9FailedEquivalent to the merged fixGPT 6 Luna max only
stet-commit-f8ae1177Passed testsFailedGPT 6.1 Sol medium only
stet-15439c21FailedFailedNeither
stet-2450ca2dFailedFailedNeither
stet-commit-342d0ac0Equivalent to the merged fixEquivalent to the merged fixBoth
stet-6f84e978FailedFailedNeither
stet-commit-60a0e6c2FailedFailedNeither
stet-commit-5e075641FailedFailedNeither
stet-commit-2826765eFailedFailedNeither
stet-commit-a8716b88FailedFailedNeither
stet-commit-e928166fFailedFailedNeither
stet-commit-87814583FailedFailedNeither
stet-commit-d2ab0c25FailedFailedNeither
stet-commit-e41c2e2dFailedFailedNeither

Side by side

Side by side
MetricGPT 6.1 Sol mediumGPT 6 Luna max
Reasoning effortmediummax
Rank4 of 1714 of 17
Stet score54.195% range 47.7–61.546.095% range 40.4–51.6
Solved11/249 passed, 2 judged equivalent11/247 passed, 4 judged equivalent
Cost / task$1.10$0.37
Agent time / task11.8 min59.7 min

See which setup wins on your repo

Stet replays your merged PRs, scores quality above pass/fail, and delivers comparison reports through your agent.

Get started