Skip to the comparison

DeepSeek V4.1 Flash max vs SWE-2 max

DeepSeek V4.1 Flash max leads on speed and cost; no clear quality difference. Different harnesses: DeepSeek V4.1 Flash max ran deepseek-harness, SWE-2 max ran devin. This compares setups, not models alone.

Quality
Even
No clear difference
2–5 head to head, 11 tied, 6 undecided
Speed
1.4× faster
DeepSeek V4.1 Flash max
49.3 min vs 68.3 min per task
Cost
5.6× cheaper
DeepSeek V4.1 Flash max
$0.49 vs $2.76 per task; $0.49 from 23/24 tasks; $2.76 from 23/24 tasks

What this comparison is

  • 1 real repositoryStet-AI/stet, private; not public benchmark scores
  • 24 merged changeseach run end to end by the coding agent from the repo as it was before the change
  • Graded against the merged changeby the repo’s own tests and by judges comparing each patch with it
  • Quality, cost and timea smaller sample than public benchmarks, but real agent work on a real codebase

24 tasksData from

Stet score

DeepSeek V4.1 Flash maxdeepseek-harness · max reasoningRank 8 of 18

50.995% range 42.6–58.9

SWE-2 maxdevin · max reasoningRank 1 of 18

62.395% range 53.4–71.6

Their 95% score ranges overlap.

Quality

Head-to-head matchups

DeepSeek V4.1 Flash max won2

Tied11

6 undecided

SWE-2 max won5

SWE-2 max’s win share 58%, 95% range 44–72%Ties are within judge noise.

Other quality measures

Other quality measures
MetricDeepSeek V4.1 Flash maxSWE-2 max
Code review pass rate36%22 tasks33%24 tasks
Checklist score0.8324 tasks0.8424 tasks
Equivalence rate43%23 tasks54%24 tasks

These measures are not part of the headline Stet score and carry no confidence interval, so a small gap may be noise. The marked side has the higher recorded value, not a proven lead.

Task by task

DeepSeek V4.1 Flash max solved 2 tasks that SWE-2 max didn’t, and SWE-2 max solved 4 that DeepSeek V4.1 Flash max didn’t. Both solved 8; neither solved 4.

On the 8 tasks both solved, judges called every patch even.

2Only DeepSeek V4.1 Flash max solved

  • 4 changed lines in 1 filestet-43493c52Judges preferred DeepSeek V4.1 Flash max’s patch
  • 1,312 changed lines in 9 filesstet-commit-f8ae1177Judges preferred DeepSeek V4.1 Flash max’s patch

4Only SWE-2 max solved

  • 2 changed lines in 1 filestet-2450ca2dJudges preferred SWE-2 max’s patch
  • 23 changed lines in 1 filestet-6f84e978Judges preferred SWE-2 max’s patch
  • 113 changed lines in 4 filesstet-commit-f0caded9Judges preferred SWE-2 max’s patch
  • 552 changed lines in 11 filesstet-commit-60a0e6c2Judges preferred SWE-2 max’s patch

8Both solved

  • 5 changed lines in 2 filesstet-15439c21
  • 9 changed lines in 1 filestet-89dfbc27
  • 10 changed lines in 1 filestet-232b50ba
  • 23 changed lines in 2 filesstet-5276aa3f
  • 29 changed lines in 1 filestet-b28f4fe7
  • 84 changed lines in 1 filestet-66762b4c
  • 218 changed lines in 2 filesstet-commit-5d766dde
  • 640 changed lines in 8 filesstet-commit-342d0ac0

4Neither solved

  • 149 changed lines in 2 filesstet-commit-e928166fJudges preferred SWE-2 max’s patch
  • 476 changed lines in 17 filesstet-commit-30506f8e
  • 763 changed lines in 8 filesstet-commit-87814583
  • 1,420 changed lines in 19 filesstet-commit-2826765e

Bar: changed lines in the merged change, log scale

A task counts as solved when its tests passed or a judge found the fix equivalent to the merged change, as in the leaderboard’s Solved column. Stet score also weighs code quality, so it can order the two differently. 6 of the 24 tasks are left out because one of the two has no scored result on them.

All 18 compared tasks
Task by task
TaskDeepSeek V4.1 Flash maxSWE-2 maxSolved byBetter patch
stet-89dfbc27 · 9 changed lines in 1 filePassed tests$0.09 · 9.3 minPassed tests$0.71 · 4.5 minBothEven
stet-b28f4fe7 · 29 changed lines in 1 filePassed tests$0.22 · 21.5 minPassed tests$2.02 · 16.7 minBothEven
stet-232b50ba · 10 changed lines in 1 filePassed tests$0.34 · 59.2 minPassed tests$1.56 · 11.9 minBothEven
stet-5276aa3f · 23 changed lines in 2 filesPassed tests$0.16 · 37.9 minPassed tests$1.03 · 10.7 minBothEven
stet-66762b4c · 84 changed lines in 1 filePassed tests$0.09 · 15.1 minPassed tests$1.77 · 11.2 minBothEven
stet-commit-5d766dde · 218 changed lines in 2 filesEquivalent to the merged fix$0.27 · 35.4 minEquivalent to the merged fix$4.04 · 20.0 minBothEven
stet-43493c52 · 4 changed lines in 1 filePassed tests$0.22 · 63.4 minFailed$3.52 · 35.3 minDeepSeek V4.1 Flash max onlyDeepSeek V4.1 Flash max
stet-commit-f0caded9 · 113 changed lines in 4 filesFailed$0.25 · 42.5 minPassed tests$3.91 · 22.6 minSWE-2 max onlySWE-2 max
stet-15439c21 · 5 changed lines in 2 filesPassed tests$0.15 · 45.6 minPassed tests$2.24 · 34.3 minBothEven
stet-commit-f8ae1177 · 1,312 changed lines in 9 filesPassed tests$1.72 · 113.6 minFailed300.0 minDeepSeek V4.1 Flash max onlyDeepSeek V4.1 Flash max
stet-2450ca2d · 2 changed lines in 1 fileFailed$0.15 · 29.5 minPassed tests$1.18 · 11.8 minSWE-2 max onlySWE-2 max
stet-commit-342d0ac0 · 640 changed lines in 8 filesEquivalent to the merged fix$1.05 · 59.4 minEquivalent to the merged fix$2.39 · 146.4 minBothEven
stet-6f84e978 · 23 changed lines in 1 fileFailed$0.24 · 28.7 minPassed tests$2.06 · 42.4 minSWE-2 max onlySWE-2 max
stet-commit-60a0e6c2 · 552 changed lines in 11 filesFailed$0.58 · 47.5 minPassed tests$0.66 · 106.1 minSWE-2 max onlySWE-2 max
stet-commit-2826765e · 1,420 changed lines in 19 filesFailed$0.80 · 70.4 minFailed$7.00 · 144.5 minNeitherEven
stet-commit-e928166f · 149 changed lines in 2 filesFailed$0.58 · 58.7 minFailed$1.49 · 59.4 minNeitherSWE-2 max
stet-commit-30506f8e · 476 changed lines in 17 filesFailed71.2 minFailed$5.49 · 164.8 minNeitherEven
stet-commit-87814583 · 763 changed lines in 8 filesFailed$0.38 · 61.5 minFailed$5.19 · 73.6 minNeitherEven

Cost and time on each task

Each dot is one task, on a log scale: filled where the setup solved it, hollow where it didn’t. The line marks the median.

Cost per task

DeepSeek V4.1 Flash maxmean $0.43 · median $0.25from 17/18 tasks

SWE-2 maxmean $2.72 · median $2.06from 17/18 tasks

DeepSeek V4.1 Flash max cost less on all 16 tasks.

Agent time per task

DeepSeek V4.1 Flash maxmean 48.4 min · median 46.6 min

SWE-2 maxmean 67.6 min · median 34.8 min

SWE-2 max finished faster on 10 of 18 tasks; DeepSeek V4.1 Flash max on 8.

Where each setup is the better choice

The leaderboard’s readings on groups of tasks. Each group holds fewer tasks than the overall standings, so treat these as leads to test on your repo.

Small fixes

10 tasks · 2–84 changed lines in 1–2 files

SWE-2 max solved 1 more, 8 of 9 to 7; DeepSeek V4.1 Flash max cost 9.7× less per task.

SWE-2 max was 1.7× faster.

Small fixes
DeepSeek V4.1 Flash maxSWE-2 max
Solved7/98/9
Cost / task$0.18$1.79
Agent time / task34.5 min19.9 min

Large, cross-cutting changes

14 tasks · 113–1,420 changed lines in up to 19 files

SWE-2 max solved 4 of 9 to DeepSeek V4.1 Flash max’s 3.

DeepSeek V4.1 Flash max was 1.9× faster.

Large, cross-cutting changes
DeepSeek V4.1 Flash maxSWE-2 max
Solved3/94/9
Cost / task$0.70from 8/9 tasks$3.77from 8/9 tasks
Agent time / task62.3 min115.3 min

Side by side

Side by side
MetricDeepSeek V4.1 Flash maxSWE-2 max
Harnessdeepseek-harnessdevin
Reasoning effortmaxmax
Rank8 of 181 of 18
Stet score50.995% range 42.6–58.962.395% range 53.4–71.6
Solved10/248 passed, 2 judged equivalent16/2413 passed, 3 judged equivalent
Cost / task$0.49from 23/24 tasks$2.76from 23/24 tasks
Agent time / task49.3 min68.3 min

Questions

Which is better for coding, DeepSeek V4.1 Flash max or SWE-2 max?
No clear quality difference: SWE-2 max won 5, DeepSeek V4.1 Flash max won 2, 11 tied, 6 undecided; DeepSeek V4.1 Flash max leads on cost and agent time: 5.6× cheaper and 1.4× faster. Measured on 24 merged changes from Stet-AI/stet.
How were DeepSeek V4.1 Flash max and SWE-2 max tested?
Both coding agents worked the same 24 merged changes from the private Stet-AI/stet repository, each in the real repo as it was before the change. The repo’s tests and judges graded every patch against the merged change; no public benchmark scores are used. Stet score: A setup’s average chance of beating another setup on a task, from 0 to 100; 50 is even with the field.

See which setup wins on your repo

Stet replays your merged changes, scores quality above pass/fail, and delivers comparison reports through your agent.

Get started