Skip to the comparison

Pareto 26.9 medium vs SWE-2 max

Pareto 26.9 medium leads on speed; SWE-2 max on cost; no clear quality difference. Different harnesses: Pareto 26.9 medium ran claude-code, SWE-2 max ran devin. This compares setups, not models alone.

Quality
Even
No clear difference
3–8 head to head, 13 tied
Speed
1.7× faster
Pareto 26.9 medium
39.2 min vs 68.3 min per task
Cost
2.2× cheaper
SWE-2 max
$2.76 vs $5.98 per task; $2.76 from 23/24 tasks

What this comparison is

  • 1 real repositoryStet-AI/stet, private; not public benchmark scores
  • 24 merged changeseach run end to end by the coding agent from the repo as it was before the change
  • Graded against the merged changeby the repo’s own tests and by judges comparing each patch with it
  • Quality, cost and timea smaller sample than public benchmarks, but real agent work on a real codebase

24 tasksData from

Stet score

Pareto 26.9 mediumclaude-code · medium reasoningRank 9 of 18

50.695% range 43.8–57.1

SWE-2 maxdevin · max reasoningRank 1 of 18

62.395% range 53.4–71.6

Their 95% score ranges overlap.

Quality

Head-to-head matchups

Pareto 26.9 medium won3

Tied13

SWE-2 max won8

SWE-2 max’s win share 61%, 95% range 48–73%Ties are within judge noise.

Other quality measures

Other quality measures
MetricPareto 26.9 mediumSWE-2 max
Code review pass rate38%21 tasks33%24 tasks
Checklist score0.8024 tasks0.8424 tasks
Equivalence rate39%23 tasks54%24 tasks

These measures are not part of the headline Stet score and carry no confidence interval, so a small gap may be noise. The marked side has the higher recorded value, not a proven lead.

Task by task

Pareto 26.9 medium solved 2 tasks that SWE-2 max didn’t, and SWE-2 max solved 7 that Pareto 26.9 medium didn’t. Both solved 9; neither solved 6.

On the 9 tasks both solved, judges preferred Pareto 26.9 medium’s patch on 1, SWE-2 max’s on 1, and called 7 even.

2Only Pareto 26.9 medium solved

  • 4 changed lines in 1 filestet-43493c52Judges preferred Pareto 26.9 medium’s patch
  • 1,312 changed lines in 9 filesstet-commit-f8ae1177Judges preferred Pareto 26.9 medium’s patch

7Only SWE-2 max solved

  • 5 changed lines in 2 filesstet-15439c21Judges preferred SWE-2 max’s patch
  • 23 changed lines in 1 filestet-6f84e978Judges preferred SWE-2 max’s patch
  • 218 changed lines in 2 filesstet-commit-5d766ddeJudges preferred SWE-2 max’s patch
  • 481 changed lines in 3 filesstet-commit-5e075641Judges preferred SWE-2 max’s patch
  • 552 changed lines in 11 filesstet-commit-60a0e6c2Judges preferred SWE-2 max’s patch
  • 564 changed lines in 10 filesstet-commit-a8716b88Judges preferred SWE-2 max’s patch
  • 640 changed lines in 8 filesstet-commit-342d0ac0Judges preferred SWE-2 max’s patch

9Both solved

  • 2 changed lines in 1 filestet-2450ca2d
  • 6 changed lines in 1 filestet-3169aefa
  • 9 changed lines in 1 filestet-89dfbc27
  • 10 changed lines in 1 filestet-232b50ba
  • 23 changed lines in 2 filesstet-5276aa3f
  • 29 changed lines in 1 filestet-b28f4fe7
  • 84 changed lines in 1 filestet-66762b4c
  • 113 changed lines in 4 filesstet-commit-f0caded9Judges preferred SWE-2 max’s patch
  • 941 changed lines in 7 filesstet-commit-bbbbae09Judges preferred Pareto 26.9 medium’s patch

6Neither solved

  • 149 changed lines in 2 filesstet-commit-e928166f
  • 476 changed lines in 17 filesstet-commit-30506f8e
  • 763 changed lines in 8 filesstet-commit-87814583
  • 977 changed lines in 5 filesstet-commit-e41c2e2d
  • 1,236 changed lines in 16 filesstet-commit-d2ab0c25
  • 1,420 changed lines in 19 filesstet-commit-2826765e

Bar: changed lines in the merged change, log scale

A task counts as solved when its tests passed or a judge found the fix equivalent to the merged change, as in the leaderboard’s Solved column. Stet score also weighs code quality, so it can order the two differently.

All 24 compared tasks
Task by task
TaskPareto 26.9 mediumSWE-2 maxSolved byBetter patch
stet-89dfbc27 · 9 changed lines in 1 filePassed tests$3.94 · 47.2 minPassed tests$0.71 · 4.5 minBothEven
stet-b28f4fe7 · 29 changed lines in 1 filePassed tests$3.17 · 22.7 minPassed tests$2.02 · 16.7 minBothEven
stet-232b50ba · 10 changed lines in 1 filePassed tests$3.09 · 42.8 minPassed tests$1.56 · 11.9 minBothEven
stet-3169aefa · 6 changed lines in 1 filePassed tests$2.42 · 14.8 minPassed tests$1.03 · 5.3 minBothEven
stet-5276aa3f · 23 changed lines in 2 filesPassed tests$0.69 · 13.0 minPassed tests$1.03 · 10.7 minBothEven
stet-66762b4c · 84 changed lines in 1 filePassed tests$3.39 · 50.6 minPassed tests$1.77 · 11.2 minBothEven
stet-commit-bbbbae09 · 941 changed lines in 7 filesPassed tests$14.28 · 95.4 minPassed tests$0.61 · 89.7 minBothPareto 26.9 medium
stet-commit-5d766dde · 218 changed lines in 2 filesFailed$1.02 · 7.0 minEquivalent to the merged fix$4.04 · 20.0 minSWE-2 max onlySWE-2 max
stet-43493c52 · 4 changed lines in 1 filePassed tests$2.04 · 29.5 minFailed$3.52 · 35.3 minPareto 26.9 medium onlyPareto 26.9 medium
stet-commit-f0caded9 · 113 changed lines in 4 filesEquivalent to the merged fix$8.97 · 60.8 minPassed tests$3.91 · 22.6 minBothSWE-2 max
stet-15439c21 · 5 changed lines in 2 filesFailed$2.73 · 36.0 minPassed tests$2.24 · 34.3 minSWE-2 max onlySWE-2 max
stet-commit-f8ae1177 · 1,312 changed lines in 9 filesPassed tests$9.84 · 52.6 minFailed300.0 minPareto 26.9 medium onlyPareto 26.9 medium
stet-2450ca2d · 2 changed lines in 1 filePassed tests$0.61 · 12.3 minPassed tests$1.18 · 11.8 minBothEven
stet-commit-342d0ac0 · 640 changed lines in 8 filesFailed$7.95 · 42.6 minEquivalent to the merged fix$2.39 · 146.4 minSWE-2 max onlySWE-2 max
stet-6f84e978 · 23 changed lines in 1 fileFailed$2.15 · 20.9 minPassed tests$2.06 · 42.4 minSWE-2 max onlySWE-2 max
stet-commit-5e075641 · 481 changed lines in 3 filesFailed$2.54 · 23.6 minPassed tests$2.64 · 86.0 minSWE-2 max onlySWE-2 max
stet-commit-60a0e6c2 · 552 changed lines in 11 filesFailed$5.64 · 20.5 minPassed tests$0.66 · 106.1 minSWE-2 max onlySWE-2 max
stet-commit-2826765e · 1,420 changed lines in 19 filesFailed$8.80 · 40.3 minFailed$7.00 · 144.5 minNeitherEven
stet-commit-a8716b88 · 564 changed lines in 10 filesFailed$13.86 · 42.7 minEquivalent to the merged fix$1.29 · 102.3 minSWE-2 max onlySWE-2 max
stet-commit-e928166f · 149 changed lines in 2 filesFailed$7.09 · 34.0 minFailed$1.49 · 59.4 minNeitherEven
stet-commit-30506f8e · 476 changed lines in 17 filesFailed$16.87 · 66.0 minFailed$5.49 · 164.8 minNeitherEven
stet-commit-87814583 · 763 changed lines in 8 filesFailed$11.99 · 89.0 minFailed$5.19 · 73.6 minNeitherEven
stet-commit-d2ab0c25 · 1,236 changed lines in 16 filesFailed$1.11 · 8.3 minFailed$5.66 · 68.9 minNeitherEven
stet-commit-e41c2e2d · 977 changed lines in 5 filesFailed$9.40 · 67.9 minFailed$6.08 · 69.8 minNeitherEven

Cost and time on each task

Each dot is one task, on a log scale: filled where the setup solved it, hollow where it didn’t. The line marks the median.

Cost per task

Pareto 26.9 mediummean $5.98 · median $3.66

SWE-2 maxmean $2.76 · median $2.06from 23/24 tasks

SWE-2 max cost less on 17 of 23 tasks; Pareto 26.9 medium on 6.

Agent time per task

Pareto 26.9 mediummean 39.2 min · median 38.2 min

SWE-2 maxmean 68.3 min · median 50.9 min

Pareto 26.9 medium finished faster on 13 of 24 tasks; SWE-2 max on 11.

Where each setup is the better choice

The leaderboard’s readings on groups of tasks. Each group holds fewer tasks than the overall standings, so treat these as leads to test on your repo.

Small fixes

Cheaper here without solving fewer: SWE-2 max

10 tasks · 2–84 changed lines in 1–2 files

SWE-2 max solved more, 9 of 10 to 8, at 1.4× lower cost per task.

SWE-2 max was 1.6× faster.

Small fixes
Pareto 26.9 mediumSWE-2 max
Solved8/109/10
Cost / task$2.42$1.71
Agent time / task29.0 min18.4 min

Large, cross-cutting changes

14 tasks · 113–1,420 changed lines in up to 19 files

SWE-2 max solved 7 of 14 to Pareto 26.9 medium’s 3.

Pareto 26.9 medium was 2.2× faster.

Large, cross-cutting changes
Pareto 26.9 mediumSWE-2 max
Solved3/147/14
Cost / task$8.53$3.57from 13/14 tasks
Agent time / task46.5 min103.9 min

Side by side

Side by side
MetricPareto 26.9 mediumSWE-2 max
Harnessclaude-codedevin
Reasoning effortmediummax
Rank9 of 181 of 18
Stet score50.695% range 43.8–57.162.395% range 53.4–71.6
Solved11/2410 passed, 1 judged equivalent16/2413 passed, 3 judged equivalent
Cost / task$5.98$2.76from 23/24 tasks
Agent time / task39.2 min68.3 min

Questions

Which is better for coding, Pareto 26.9 medium or SWE-2 max?
No clear quality difference: SWE-2 max won 8, Pareto 26.9 medium won 3, 13 tied; SWE-2 max leads on cost: 2.2× cheaper; Pareto 26.9 medium leads on agent time: 1.7× faster. Measured on 24 merged changes from Stet-AI/stet.
How were Pareto 26.9 medium and SWE-2 max tested?
Both coding agents worked the same 24 merged changes from the private Stet-AI/stet repository, each in the real repo as it was before the change. The repo’s tests and judges graded every patch against the merged change; no public benchmark scores are used. Stet score: A setup’s average chance of beating another setup on a task, from 0 to 100; 50 is even with the field.

See which setup wins on your repo

Stet replays your merged changes, scores quality above pass/fail, and delivers comparison reports through your agent.

Get started