GPT 5.6 Luna low leads on speed and cost; no clear quality difference. Different harnesses: GPT 5.6 Luna low ran codex, SWE-2 max ran devin. This compares setups, not models alone.
Quality
Even
No clear difference
1–9 head to head, 13 tied, 1 undecided
Speed
10× faster
GPT 5.6 Luna low
6.7 min vs 68.3 min per task
Cost
54× cheaper
GPT 5.6 Luna low
$0.05 vs $2.76 per task; $2.76 from 23/24 tasks
What this comparison is
1 real repositoryStet-AI/stet, private; not public benchmark scores
24 merged changeseach run end to end by the coding agent from the repo as it was before the change
Graded against the merged changeby the repo’s own tests and by judges comparing each patch with it
Quality, cost and timea smaller sample than public benchmarks, but real agent work on a real codebase
GPT 5.6 Luna lowcodex · low reasoningRank 16 of 18
44.795% range 37.7–51.0
SWE-2 maxdevin · max reasoningRank 1 of 18
62.395% range 53.4–71.6
Other setups
Opus 5 medium
Opus 5 medium
60.9
Muse Wonder Full low
Muse Wonder Full low
57.2
GPT 6.1 Sol medium
GPT 6.1 Sol medium
53.1
GPT 6 Sol medium
GPT 6 Sol medium
52.9
Opus 4.8 medium
Opus 4.8 medium
52.5
Grok 4.7 high
Grok 4.7 high
50.9
DeepSeek V4.1 Flash max
DeepSeek V4.1 Flash max
50.9
Pareto 26.9 medium
Pareto 26.9 medium
50.6
GLM 5.3 Flash high
GLM 5.3 Flash high
49.0
DeepSeek V4.1 Flash (10/17) max
DeepSeek V4.1 Flash (10/17) max
48.4
Haiku 5.5 max
Haiku 5.5 max
48.0
GPT 6 Luna max
GPT 6 Luna max
47.7
Muse Spark 1.3 high
Muse Spark 1.3 high
47.7
GPT 5.6 Luna max
GPT 5.6 Luna max
45.4
Haiku 5.5 low
Haiku 5.5 low
43.7
GPT 6 Luna low
GPT 6 Luna low
39.8
304050607080even with the field
Their 95% score ranges don’t overlap.
Quality
Head-to-head matchups
GPT 5.6 Luna low won1
Tied13
1 undecided
SWE-2 max won9
SWE-2 max’s win share 67%, 95% range 56–79%Ties are within judge noise.
Other quality measures
Other quality measures
Metric
GPT 5.6 Luna low
SWE-2 max
Code review pass rate
30%23 tasks
33%24 tasks
Checklist score
0.6024 tasks
0.8424 tasks
Equivalence rate
38%24 tasks
54%24 tasks
These measures are not part of the headline Stet score and carry no confidence interval, so a small gap may be noise. The marked side has the higher recorded value, not a proven lead.
Task by task
GPT 5.6 Luna low solved 1 task that SWE-2 max didn’t, and SWE-2 max solved 6 that GPT 5.6 Luna low didn’t. Both solved 9; neither solved 7.
On the 9 tasks both solved, judges preferred SWE-2 max’s patch on 2 and called 7 even.
1Only GPT 5.6 Luna low solved
4 changed lines in 1 filestet-43493c52Judges preferred GPT 5.6 Luna low’s patch
6Only SWE-2 max solved
2 changed lines in 1 filestet-2450ca2dJudges preferred SWE-2 max’s patch
481 changed lines in 3 filesstet-commit-5e075641Judges preferred SWE-2 max’s patch
552 changed lines in 11 filesstet-commit-60a0e6c2Judges preferred SWE-2 max’s patch
564 changed lines in 10 filesstet-commit-a8716b88Judges preferred SWE-2 max’s patch
640 changed lines in 8 filesstet-commit-342d0ac0Judges preferred SWE-2 max’s patch
941 changed lines in 7 filesstet-commit-bbbbae09Judges preferred SWE-2 max’s patch
9Both solved
5 changed lines in 2 filesstet-15439c21
6 changed lines in 1 filestet-3169aefa
9 changed lines in 1 filestet-89dfbc27
10 changed lines in 1 filestet-232b50ba
23 changed lines in 2 filesstet-5276aa3f
29 changed lines in 1 filestet-b28f4fe7
84 changed lines in 1 filestet-66762b4cJudges preferred SWE-2 max’s patch
113 changed lines in 4 filesstet-commit-f0caded9Judges preferred SWE-2 max’s patch
218 changed lines in 2 filesstet-commit-5d766dde
7Neither solved
149 changed lines in 2 filesstet-commit-e928166fJudges preferred SWE-2 max’s patch
476 changed lines in 17 filesstet-commit-30506f8e
763 changed lines in 8 filesstet-commit-87814583
977 changed lines in 5 filesstet-commit-e41c2e2d
1,236 changed lines in 16 filesstet-commit-d2ab0c25
1,312 changed lines in 9 filesstet-commit-f8ae1177
1,420 changed lines in 19 filesstet-commit-2826765e
Bar: changed lines in the merged change, log scale
A task counts as solved when its tests passed or a judge found the fix equivalent to the merged change, as in the leaderboard’s Solved column. Stet score also weighs code quality, so it can order the two differently. 1 of the 24 tasks is left out because one of the two has no scored result on it.
All 23 compared tasks
Task by task
Task
GPT 5.6 Luna low
SWE-2 max
Solved by
Better patch
stet-89dfbc27 · 9 changed lines in 1 file
Passed tests$0.04 · 6.5 min
Passed tests$0.71 · 4.5 min
Both
Even
stet-b28f4fe7 · 29 changed lines in 1 file
Passed tests$0.03 · 2.6 min
Passed tests$2.02 · 16.7 min
Both
Even
stet-232b50ba · 10 changed lines in 1 file
Passed tests$0.04 · 14.5 min
Passed tests$1.56 · 11.9 min
Both
Even
stet-3169aefa · 6 changed lines in 1 file
Passed tests$0.02 · 4.9 min
Passed tests$1.03 · 5.3 min
Both
Even
stet-5276aa3f · 23 changed lines in 2 files
Passed tests$0.05 · 13.4 min
Passed tests$1.03 · 10.7 min
Both
Even
stet-66762b4c · 84 changed lines in 1 file
Equivalent to the merged fix$0.02 · 3.7 min
Passed tests$1.77 · 11.2 min
Both
SWE-2 max
stet-commit-bbbbae09 · 941 changed lines in 7 files
Failed$0.04 · 6.6 min
Passed tests$0.61 · 89.7 min
SWE-2 max only
SWE-2 max
stet-commit-5d766dde · 218 changed lines in 2 files
Equivalent to the merged fix$0.05 · 10.7 min
Equivalent to the merged fix$4.04 · 20.0 min
Both
Even
stet-43493c52 · 4 changed lines in 1 file
Passed tests$0.03 · 4.0 min
Failed$3.52 · 35.3 min
GPT 5.6 Luna low only
GPT 5.6 Luna low
stet-commit-f0caded9 · 113 changed lines in 4 files
Equivalent to the merged fix$0.04 · 2.9 min
Passed tests$3.91 · 22.6 min
Both
SWE-2 max
stet-15439c21 · 5 changed lines in 2 files
Equivalent to the merged fix$0.03 · 9.5 min
Passed tests$2.24 · 34.3 min
Both
Even
stet-commit-f8ae1177 · 1,312 changed lines in 9 files
Failed$0.05 · 6.7 min
Failed300.0 min
Neither
Even
stet-2450ca2d · 2 changed lines in 1 file
Failed$0.02 · 4.2 min
Passed tests$1.18 · 11.8 min
SWE-2 max only
SWE-2 max
stet-commit-342d0ac0 · 640 changed lines in 8 files
Failed$0.12 · 12.4 min
Equivalent to the merged fix$2.39 · 146.4 min
SWE-2 max only
SWE-2 max
stet-commit-5e075641 · 481 changed lines in 3 files
Failed$0.07 · 5.7 min
Passed tests$2.64 · 86.0 min
SWE-2 max only
SWE-2 max
stet-commit-60a0e6c2 · 552 changed lines in 11 files
Failed$0.04 · 2.6 min
Passed tests$0.66 · 106.1 min
SWE-2 max only
SWE-2 max
stet-commit-2826765e · 1,420 changed lines in 19 files
Failed$0.04 · 7.2 min
Failed$7.00 · 144.5 min
Neither
Even
stet-commit-a8716b88 · 564 changed lines in 10 files
Failed$0.10 · 5.8 min
Equivalent to the merged fix$1.29 · 102.3 min
SWE-2 max only
SWE-2 max
stet-commit-e928166f · 149 changed lines in 2 files
Failed$0.06 · 3.4 min
Failed$1.49 · 59.4 min
Neither
SWE-2 max
stet-commit-30506f8e · 476 changed lines in 17 files
Failed$0.12 · 10.1 min
Failed$5.49 · 164.8 min
Neither
Even
stet-commit-87814583 · 763 changed lines in 8 files
Failed$0.03 · 2.7 min
Failed$5.19 · 73.6 min
Neither
Even
stet-commit-d2ab0c25 · 1,236 changed lines in 16 files
Failed$0.05 · 5.0 min
Failed$5.66 · 68.9 min
Neither
Even
stet-commit-e41c2e2d · 977 changed lines in 5 files
Failed$0.10 · 8.3 min
Failed$6.08 · 69.8 min
Neither
Even
Cost and time on each task
Each dot is one task, on a log scale: filled where the setup solved it, hollow where it didn’t. The line marks the median.
Cost per task
GPT 5.6 Luna lowmean $0.05 · median $0.04
SWE-2 maxmean $2.80 · median $2.13from 22/23 tasks
$0.01$0.10$1$10
GPT 5.6 Luna low cost less on all 22 tasks.
Agent time per task
GPT 5.6 Luna lowmean 6.7 min · median 5.8 min
SWE-2 maxmean 69.4 min · median 59.4 min
1 min10 min100 min1000 min
GPT 5.6 Luna low finished faster on 20 of 23 tasks; SWE-2 max on 3.
Where each setup is the better choice
The leaderboard’s readings on groups of tasks. Each group holds fewer tasks than the overall standings, so treat these as leads to test on your repo.
Small fixes
Cheaper here without solving fewer: GPT 5.6 Luna low
10 tasks · 2–84 changed lines in 1–2 files
GPT 5.6 Luna low solved as many, 8 of 9, at 53× lower cost per task.
GPT 5.6 Luna low was 2.2× faster.
Small fixes
GPT 5.6 Luna low
SWE-2 max
Solved
8/9
8/9
Cost / task
$0.03
$1.67
Agent time / task
7.0 min
15.7 min
Large, cross-cutting changes
14 tasks · 113–1,420 changed lines in up to 19 files
SWE-2 max solved 7 of 14 to GPT 5.6 Luna low’s 2.
GPT 5.6 Luna low was 16× faster.
Large, cross-cutting changes
GPT 5.6 Luna low
SWE-2 max
Solved
2/14
7/14
Cost / task
$0.07
$3.57from 13/14 tasks
Agent time / task
6.4 min
103.9 min
Side by side
Side by side
Metric
GPT 5.6 Luna low
SWE-2 max
Harness
codex
devin
Reasoning effort
low
max
Rank
16 of 18
1 of 18
Stet score
44.795% range 37.7–51.0
62.395% range 53.4–71.6
Solved
10/246 passed, 4 judged equivalent
16/2413 passed, 3 judged equivalent
Cost / task
$0.05
$2.76from 23/24 tasks
Agent time / task
6.7 min
68.3 min
Questions
Which is better for coding, GPT 5.6 Luna low or SWE-2 max?
No clear quality difference: SWE-2 max won 9, GPT 5.6 Luna low won 1, 13 tied, 1 undecided; GPT 5.6 Luna low leads on cost and agent time: 54× cheaper and 10× faster. Measured on 24 merged changes from Stet-AI/stet.
How were GPT 5.6 Luna low and SWE-2 max tested?
Both coding agents worked the same 24 merged changes from the private Stet-AI/stet repository, each in the real repo as it was before the change. The repo’s tests and judges graded every patch against the merged change; no public benchmark scores are used. Stet score: A setup’s average chance of beating another setup on a task, from 0 to 100; 50 is even with the field.
See which setup wins on your repo
Stet replays your merged changes, scores quality above pass/fail, and delivers comparison reports through your agent.