DeepSeek V4.1 Flash (10/17) maxcodex · max reasoningRank 11 of 18
48.395% range 42.9–53.41.6 points higher
GPT 5.6 Luna maxmax reasoningRank 14 of 18
46.795% range 40.8–52.4
Other setups
SWE-2 Max high
SWE-2 Max high
63.7
Opus 5 medium
Opus 5 medium
61.2
Muse Wonder Full low
Muse Wonder Full low
56.5
GPT 6.1 Sol medium
GPT 6.1 Sol medium
54.5
GPT 6 Sol medium
GPT 6 Sol medium
53.2
Grok 4.7 high
Grok 4.7 high
52.6
Pareto 26.9 medium
Pareto 26.9 medium
52.2
DeepSeek V4.1 Flash max
DeepSeek V4.1 Flash max
50.9
Opus 4.8 medium
Opus 4.8 medium
48.9
Haiku 5.5 max
Haiku 5.5 max
48.8
Muse Spark 1.3 high
Muse Spark 1.3 high
47.5
GLM 5.3 Flash high
GLM 5.3 Flash high
47.1
GPT 6 Luna max
GPT 6 Luna max
45.2
Haiku 5.5 low
Haiku 5.5 low
44.8
GPT 5.6 Luna low
GPT 5.6 Luna low
43.4
GPT 6 Luna low
GPT 6 Luna low
40.7
304050607080even with the field
A directional read: their 95% score ranges overlap.
Quality
Head-to-head matchups
DeepSeek V4.1 Flash (10/17) max won3
Tied19
GPT 5.6 Luna max won2
DeepSeek V4.1 Flash (10/17) max’s win share 52%, 95% range 43–61%Ties are within judge noise.
Task by task
DeepSeek V4.1 Flash (10/17) max solved 1 task that GPT 5.6 Luna max didn’t, and GPT 5.6 Luna max solved 1 that DeepSeek V4.1 Flash (10/17) max didn’t. Both solved 9; neither solved 11.
On the 9 tasks both solved, judges preferred GPT 5.6 Luna max’s patch on 2, DeepSeek V4.1 Flash (10/17) max’s on 1, and called 6 even.
218 changed lines in 2 filesstet-commit-5d766ddeJudges preferred GPT 5.6 Luna max’s patch
941 changed lines in 7 filesstet-commit-bbbbae09Judges preferred GPT 5.6 Luna max’s patch
11Neither solved
149 changed lines in 2 filesstet-commit-e928166f
476 changed lines in 17 filesstet-commit-30506f8e
481 changed lines in 3 filesstet-commit-5e075641
552 changed lines in 11 filesstet-commit-60a0e6c2
564 changed lines in 10 filesstet-commit-a8716b88
640 changed lines in 8 filesstet-commit-342d0ac0
763 changed lines in 8 filesstet-commit-87814583
977 changed lines in 5 filesstet-commit-e41c2e2d
1,236 changed lines in 16 filesstet-commit-d2ab0c25
1,312 changed lines in 9 filesstet-commit-f8ae1177
1,420 changed lines in 19 filesstet-commit-2826765e
Bar: changed lines in the merged change, log scale
A task counts as solved when its tests passed or a judge found the fix equivalent to the merged change, as in the leaderboard’s Solved column. Stet score also weighs code quality, so it can order the two differently. 2 of the 24 tasks are left out because one of the two has no scored result on them.
All 22 compared tasks
Task by task
Task
DeepSeek V4.1 Flash (10/17) max
GPT 5.6 Luna max
Solved by
Better patch
stet-3169aefa · 6 changed lines in 1 file
Passed tests$0.06 · 15.9 min
Passed tests$0.10 · 3.8 min
Both
Even
stet-89dfbc27 · 9 changed lines in 1 file
Passed tests$0.05 · 4.9 min
Passed tests$0.25 · 6.8 min
Both
Even
stet-b28f4fe7 · 29 changed lines in 1 file
Passed tests$0.06 · 28.7 min
Passed tests$0.54 · 12.2 min
Both
Even
stet-232b50ba · 10 changed lines in 1 file
Passed tests$0.05 · 28.0 min
Passed tests$0.26 · 16.0 min
Both
Even
stet-5276aa3f · 23 changed lines in 2 files
Passed tests$0.02 · 14.4 min
Passed tests$0.18 · 5.8 min
Both
Even
stet-commit-bbbbae09 · 941 changed lines in 7 files
Passed tests$0.65 · 47.7 min
Passed tests$1.21 · 41.6 min
Both
GPT 5.6 Luna max
stet-66762b4c · 84 changed lines in 1 file
Passed tests$0.08 · 9.1 min
Equivalent to the merged fix$0.08 · 4.9 min
Both
DeepSeek V4.1 Flash (10/17) max
stet-commit-5d766dde · 218 changed lines in 2 files
Equivalent to the merged fix$0.09 · 18.5 min
Equivalent to the merged fix$0.48 · 19.3 min
Both
GPT 5.6 Luna max
stet-43493c52 · 4 changed lines in 1 file
Passed tests$0.14 · 16.0 min
Passed tests$0.14 · 7.3 min
Both
Even
stet-commit-f0caded9 · 113 changed lines in 4 files
Failed$0.09 · 14.6 min
Equivalent to the merged fix$0.90 · 25.2 min
GPT 5.6 Luna max only
Even
stet-commit-f8ae1177 · 1,312 changed lines in 9 files
Failed$0.54 · 75.0 min
Failed$1.38 · 52.9 min
Neither
Even
stet-2450ca2d · 2 changed lines in 1 file
Passed tests$0.07 · 10.1 min
Failed$0.09 · 9.1 min
DeepSeek V4.1 Flash (10/17) max only
DeepSeek V4.1 Flash (10/17) max
stet-commit-342d0ac0 · 640 changed lines in 8 files
Failed$0.20 · 38.3 min
Failed$2.15 · 66.8 min
Neither
Even
stet-commit-60a0e6c2 · 552 changed lines in 11 files
Failed$0.34 · 39.2 min
Failed$0.84 · 26.4 min
Neither
Even
stet-commit-5e075641 · 481 changed lines in 3 files
Failed$0.46 · 53.1 min
Failed$0.99 · 34.4 min
Neither
Even
stet-commit-2826765e · 1,420 changed lines in 19 files
Failed$0.56 · 50.2 min
Failed$1.52 · 57.1 min
Neither
Even
stet-commit-a8716b88 · 564 changed lines in 10 files
Failed$0.30 · 40.2 min
Failed$2.65 · 60.3 min
Neither
Even
stet-commit-e928166f · 149 changed lines in 2 files
Failed$0.29 · 42.0 min
Failed$1.25 · 37.4 min
Neither
Even
stet-commit-30506f8e · 476 changed lines in 17 files
Failed$0.38 · 48.8 min
Failed$2.47 · 54.8 min
Neither
Even
stet-commit-87814583 · 763 changed lines in 8 files
Failed$0.50 · 62.2 min
Failed$1.17 · 40.8 min
Neither
Even
stet-commit-d2ab0c25 · 1,236 changed lines in 16 files
Failed$0.71 · 76.4 min
Failed$1.27 · 36.4 min
Neither
Even
stet-commit-e41c2e2d · 977 changed lines in 5 files
Failed$0.96 · 110.4 min
Failed$0.93 · 32.4 min
Neither
Even
Cost and time on each task
Each dot is one task, on a log scale: filled where the setup solved it, hollow where it didn’t. The line marks the median.
Cost per task
DeepSeek V4.1 Flash (10/17) maxmean $0.30 · median $0.25
GPT 5.6 Luna maxmean $0.95 · median $0.92
$0.01$0.10$1$10
DeepSeek V4.1 Flash (10/17) max cost less on 20 of 22 tasks; GPT 5.6 Luna max on 2.
Agent time per task
DeepSeek V4.1 Flash (10/17) maxmean 38.3 min · median 38.7 min
GPT 5.6 Luna maxmean 29.6 min · median 29.4 min
1 min10 min100 min1000 min
GPT 5.6 Luna max finished faster on 15 of 22 tasks; DeepSeek V4.1 Flash (10/17) max on 7.
Where each setup is the better choice
The leaderboard’s readings on groups of tasks. Each group holds fewer tasks than the overall standings, so treat these as leads to test on your repo.
Small fixes
Cheaper here without solving fewer: DeepSeek V4.1 Flash (10/17) max
10 tasks · 2–84 changed lines in 1–2 files
DeepSeek V4.1 Flash (10/17) max solved more, 8 of 8 to 7, at 3.1× lower cost per task.
GPT 5.6 Luna max was 1.9× faster.
Small fixes
DeepSeek V4.1 Flash (10/17) max
GPT 5.6 Luna max
Solved
8/8
7/8
Cost / task
$0.07
$0.20
Agent time / task
15.9 min
8.2 min
Large, cross-cutting changes
14 tasks · 113–1,420 changed lines in up to 19 files
GPT 5.6 Luna max solved 1 more, 3 of 14 to 2; DeepSeek V4.1 Flash (10/17) max cost 3.2× less per task.
GPT 5.6 Luna max was 1.2× faster.
Large, cross-cutting changes
DeepSeek V4.1 Flash (10/17) max
GPT 5.6 Luna max
Solved
2/14
3/14
Cost / task
$0.43
$1.37
Agent time / task
51.2 min
41.9 min
Side by side
Side by side
Metric
DeepSeek V4.1 Flash (10/17) max
GPT 5.6 Luna max
Harness
codex
—
Reasoning effort
max
max
Rank
11 of 18
14 of 18
Stet score
48.395% range 42.9–53.4
46.795% range 40.8–52.4
Solved
10/249 passed, 1 judged equivalent
10/247 passed, 3 judged equivalent
Cost / task
$0.28
$0.90
Agent time / task
37.2 min
28.5 min
Questions
Which is better for coding, DeepSeek V4.1 Flash (10/17) max or GPT 5.6 Luna max?
No clear quality difference: DeepSeek V4.1 Flash (10/17) max won 3, GPT 5.6 Luna max won 2, 19 tied; DeepSeek V4.1 Flash (10/17) max leads on cost: 3.2× cheaper; GPT 5.6 Luna max leads on agent time: 1.3× faster. Measured on 24 merged changes from Stet-AI/stet.
How were DeepSeek V4.1 Flash (10/17) max and GPT 5.6 Luna max tested?
Both coding agents worked the same 24 merged changes from the private Stet-AI/stet repository, each in the real repo as it was before the change. The repo’s tests and judges graded every patch against the merged change; no public benchmark scores are used. Stet score: A setup’s average chance of beating another setup on a task, from 0 to 100; 50 is even with the field.
See which setup wins on your repo
Stet replays your merged changes, scores quality above pass/fail, and delivers comparison reports through your agent.