Skip to the comparison

DeepSeek V4.1 Flash (10/17) max vs SWE-2 Max high

SWE-2 Max high leads on quality; DeepSeek V4.1 Flash (10/17) max on speed.

Quality
+15.4 pts
SWE-2 Max high
Clear lead
6–1 head to head, 17 tied
Not yet conclusive across all 153 comparisons on the board.
Speed
1.8× faster
DeepSeek V4.1 Flash (10/17) max
37.2 min vs 68.3 min per task

What this comparison is

  • 1 real repositoryStet-AI/stet, private; not public benchmark scores
  • 24 merged changeseach run end to end by the coding agent from the repo as it was before the change
  • Graded against the merged changeby the repo’s own tests and by judges comparing each patch with it
  • Quality, cost and timea smaller sample than public benchmarks, but real agent work on a real codebase

24 tasksData from Cost not recorded for SWE-2 Max high

Stet score

DeepSeek V4.1 Flash (10/17) maxcodex · max reasoningRank 11 of 18

48.395% range 42.9–53.4

SWE-2 Max highhigh reasoningRank 1 of 18

63.795% range 54.9–72.415.4 points higher

Their 95% score ranges don’t overlap.

Quality

Head-to-head matchups

DeepSeek V4.1 Flash (10/17) max won1

Tied17

SWE-2 Max high won6

SWE-2 Max high’s win share 63%, 95% range 53–73%Ties are within judge noise.

Task by task

DeepSeek V4.1 Flash (10/17) max solved 1 task that SWE-2 Max high didn’t, and SWE-2 Max high solved 5 that DeepSeek V4.1 Flash (10/17) max didn’t. Both solved 9; neither solved 7.

On the 9 tasks both solved, judges preferred SWE-2 Max high’s patch on 1 and called 8 even.

1Only DeepSeek V4.1 Flash (10/17) max solved

  • 4 changed lines in 1 filestet-43493c52Judges preferred DeepSeek V4.1 Flash (10/17) max’s patch

5Only SWE-2 Max high solved

  • 113 changed lines in 4 filesstet-commit-f0caded9Judges preferred SWE-2 Max high’s patch
  • 481 changed lines in 3 filesstet-commit-5e075641Judges preferred SWE-2 Max high’s patch
  • 552 changed lines in 11 filesstet-commit-60a0e6c2Judges preferred SWE-2 Max high’s patch
  • 564 changed lines in 10 filesstet-commit-a8716b88
  • 640 changed lines in 8 filesstet-commit-342d0ac0Judges preferred SWE-2 Max high’s patch

9Both solved

  • 2 changed lines in 1 filestet-2450ca2d
  • 6 changed lines in 1 filestet-3169aefa
  • 9 changed lines in 1 filestet-89dfbc27
  • 10 changed lines in 1 filestet-232b50ba
  • 23 changed lines in 2 filesstet-5276aa3f
  • 29 changed lines in 1 filestet-b28f4fe7
  • 84 changed lines in 1 filestet-66762b4c
  • 218 changed lines in 2 filesstet-commit-5d766dde
  • 941 changed lines in 7 filesstet-commit-bbbbae09Judges preferred SWE-2 Max high’s patch

7Neither solved

  • 149 changed lines in 2 filesstet-commit-e928166f
  • 476 changed lines in 17 filesstet-commit-30506f8e
  • 763 changed lines in 8 filesstet-commit-87814583
  • 977 changed lines in 5 filesstet-commit-e41c2e2d
  • 1,236 changed lines in 16 filesstet-commit-d2ab0c25
  • 1,312 changed lines in 9 filesstet-commit-f8ae1177
  • 1,420 changed lines in 19 filesstet-commit-2826765e

Bar: changed lines in the merged change, log scale

A task counts as solved when its tests passed or a judge found the fix equivalent to the merged change, as in the leaderboard’s Solved column. Stet score also weighs code quality, so it can order the two differently. 2 of the 24 tasks are left out because one of the two has no scored result on them.

All 22 compared tasks
Task by task
TaskDeepSeek V4.1 Flash (10/17) maxSWE-2 Max highSolved byBetter patch
stet-3169aefa · 6 changed lines in 1 filePassed tests$0.06 · 15.9 minPassed tests5.3 minBothEven
stet-89dfbc27 · 9 changed lines in 1 filePassed tests$0.05 · 4.9 minPassed tests4.5 minBothEven
stet-b28f4fe7 · 29 changed lines in 1 filePassed tests$0.06 · 28.7 minPassed tests16.7 minBothEven
stet-232b50ba · 10 changed lines in 1 filePassed tests$0.05 · 28.0 minPassed tests11.9 minBothEven
stet-5276aa3f · 23 changed lines in 2 filesPassed tests$0.02 · 14.4 minPassed tests10.7 minBothEven
stet-commit-bbbbae09 · 941 changed lines in 7 filesPassed tests$0.65 · 47.7 minPassed tests89.7 minBothSWE-2 Max high
stet-66762b4c · 84 changed lines in 1 filePassed tests$0.08 · 9.1 minPassed tests11.2 minBothEven
stet-commit-5d766dde · 218 changed lines in 2 filesEquivalent to the merged fix$0.09 · 18.5 minEquivalent to the merged fix20.0 minBothEven
stet-43493c52 · 4 changed lines in 1 filePassed tests$0.14 · 16.0 minFailed35.3 minDeepSeek V4.1 Flash (10/17) max onlyDeepSeek V4.1 Flash (10/17) max
stet-commit-f0caded9 · 113 changed lines in 4 filesFailed$0.09 · 14.6 minPassed tests22.6 minSWE-2 Max high onlySWE-2 Max high
stet-commit-f8ae1177 · 1,312 changed lines in 9 filesFailed$0.54 · 75.0 minFailed300.0 minNeitherEven
stet-2450ca2d · 2 changed lines in 1 filePassed tests$0.07 · 10.1 minPassed tests11.8 minBothEven
stet-commit-342d0ac0 · 640 changed lines in 8 filesFailed$0.20 · 38.3 minEquivalent to the merged fix146.4 minSWE-2 Max high onlySWE-2 Max high
stet-commit-60a0e6c2 · 552 changed lines in 11 filesFailed$0.34 · 39.2 minPassed tests106.1 minSWE-2 Max high onlySWE-2 Max high
stet-commit-5e075641 · 481 changed lines in 3 filesFailed$0.46 · 53.1 minPassed tests86.0 minSWE-2 Max high onlySWE-2 Max high
stet-commit-2826765e · 1,420 changed lines in 19 filesFailed$0.56 · 50.2 minFailed144.5 minNeitherEven
stet-commit-a8716b88 · 564 changed lines in 10 filesFailed$0.30 · 40.2 minEquivalent to the merged fix102.3 minSWE-2 Max high onlyEven
stet-commit-e928166f · 149 changed lines in 2 filesFailed$0.29 · 42.0 minFailed59.4 minNeitherEven
stet-commit-30506f8e · 476 changed lines in 17 filesFailed$0.38 · 48.8 minFailed164.8 minNeitherEven
stet-commit-87814583 · 763 changed lines in 8 filesFailed$0.50 · 62.2 minFailed73.6 minNeitherEven
stet-commit-d2ab0c25 · 1,236 changed lines in 16 filesFailed$0.71 · 76.4 minFailed68.9 minNeitherEven
stet-commit-e41c2e2d · 977 changed lines in 5 filesFailed$0.96 · 110.4 minFailed69.8 minNeitherEven

Cost and time on each task

Each dot is one task, on a log scale: filled where the setup solved it, hollow where it didn’t. The line marks the median.

Cost per task

DeepSeek V4.1 Flash (10/17) maxmean $0.30 · median $0.25

SWE-2 Max highnot recorded

Agent time per task

DeepSeek V4.1 Flash (10/17) maxmean 38.3 min · median 38.7 min

SWE-2 Max highmean 71.0 min · median 64.2 min

DeepSeek V4.1 Flash (10/17) max finished faster on 15 of 22 tasks; SWE-2 Max high on 7.

Where each setup is the better choice

The leaderboard’s readings on groups of tasks. Each group holds fewer tasks than the overall standings, so treat these as leads to test on your repo.

Small fixes

10 tasks · 2–84 changed lines in 1–2 files

DeepSeek V4.1 Flash (10/17) max solved 8 of 8 to SWE-2 Max high’s 7.

SWE-2 Max high was 1.2× faster.

Small fixes
DeepSeek V4.1 Flash (10/17) maxSWE-2 Max high
Solved8/87/8
Cost / task$0.07not recorded
Agent time / task15.9 min13.4 min

Large, cross-cutting changes

14 tasks · 113–1,420 changed lines in up to 19 files

SWE-2 Max high solved 7 of 14 to DeepSeek V4.1 Flash (10/17) max’s 2.

DeepSeek V4.1 Flash (10/17) max was 2.0× faster.

Large, cross-cutting changes
DeepSeek V4.1 Flash (10/17) maxSWE-2 Max high
Solved2/147/14
Cost / task$0.43not recorded
Agent time / task51.2 min103.9 min

Side by side

Side by side
MetricDeepSeek V4.1 Flash (10/17) maxSWE-2 Max high
Harnesscodex—
Reasoning effortmaxhigh
Rank11 of 181 of 18
Stet score48.395% range 42.9–53.463.795% range 54.9–72.4
Solved10/249 passed, 1 judged equivalent16/2413 passed, 3 judged equivalent
Cost / task$0.28not recorded
Agent time / task37.2 min68.3 min

Questions

Which is better for coding, DeepSeek V4.1 Flash (10/17) max or SWE-2 Max high?
SWE-2 Max high leads on quality: won 6, tied 17, lost 1 head to head; DeepSeek V4.1 Flash (10/17) max leads on agent time: 1.8× faster; SWE-2 Max high’s cost wasn’t recorded. Measured on 24 merged changes from Stet-AI/stet.
How were DeepSeek V4.1 Flash (10/17) max and SWE-2 Max high tested?
Both coding agents worked the same 24 merged changes from the private Stet-AI/stet repository, each in the real repo as it was before the change. The repo’s tests and judges graded every patch against the merged change; no public benchmark scores are used. Stet score: A setup’s average chance of beating another setup on a task, from 0 to 100; 50 is even with the field.

See which setup wins on your repo

Stet replays your merged changes, scores quality above pass/fail, and delivers comparison reports through your agent.

Get started