Skip to the comparison

Opus 5 medium vs SWE-2 max

Opus 5 medium leads on speed; SWE-2 max on cost; no clear quality difference. Different harnesses: Opus 5 medium ran claude-code, SWE-2 max ran devin. This compares setups, not models alone.

Quality
Even
No clear difference
4–4 head to head, 16 tied
Speed
2.9× faster
Opus 5 medium
23.2 min vs 68.3 min per task
Cost
2.5× cheaper
SWE-2 max
$2.76 vs $6.78 per task; $2.76 from 23/24 tasks

What this comparison is

  • 1 real repositoryStet-AI/stet, private; not public benchmark scores
  • 24 merged changeseach run end to end by the coding agent from the repo as it was before the change
  • Graded against the merged changeby the repo’s own tests and by judges comparing each patch with it
  • Quality, cost and timea smaller sample than public benchmarks, but real agent work on a real codebase

24 tasksData from

Stet score

Opus 5 mediumclaude-code · medium reasoningRank 2 of 18

60.995% range 54.2–68.3

SWE-2 maxdevin · max reasoningRank 1 of 18

62.395% range 53.4–71.6

Their 95% score ranges overlap.

Quality

Head-to-head matchups

Opus 5 medium won4

Tied16

SWE-2 max won4

SWE-2 max’s win share 50%, 95% range 38–60%Ties are within judge noise.

Other quality measures

Other quality measures
MetricOpus 5 mediumSWE-2 max
Code review pass rate39%23 tasks33%24 tasks
Checklist score0.8924 tasks0.8424 tasks
Equivalence rate54%24 tasks54%24 tasks

These measures are not part of the headline Stet score and carry no confidence interval, so a small gap may be noise. The marked side has the higher recorded value, not a proven lead.

Task by task

Opus 5 medium solved 3 tasks that SWE-2 max didn’t, and SWE-2 max solved 4 that Opus 5 medium didn’t. Both solved 12; neither solved 5.

On the 12 tasks both solved, judges preferred Opus 5 medium’s patch on 1 and called 11 even.

3Only Opus 5 medium solved

  • 4 changed lines in 1 filestet-43493c52Judges preferred Opus 5 medium’s patch
  • 1,312 changed lines in 9 filesstet-commit-f8ae1177Judges preferred Opus 5 medium’s patch
  • 1,420 changed lines in 19 filesstet-commit-2826765eJudges preferred Opus 5 medium’s patch

4Only SWE-2 max solved

  • 481 changed lines in 3 filesstet-commit-5e075641Judges preferred SWE-2 max’s patch
  • 552 changed lines in 11 filesstet-commit-60a0e6c2Judges preferred SWE-2 max’s patch
  • 564 changed lines in 10 filesstet-commit-a8716b88Judges preferred SWE-2 max’s patch
  • 640 changed lines in 8 filesstet-commit-342d0ac0Judges preferred SWE-2 max’s patch

12Both solved

  • 2 changed lines in 1 filestet-2450ca2d
  • 5 changed lines in 2 filesstet-15439c21
  • 6 changed lines in 1 filestet-3169aefa
  • 9 changed lines in 1 filestet-89dfbc27
  • 10 changed lines in 1 filestet-232b50ba
  • 23 changed lines in 2 filesstet-5276aa3f
  • 23 changed lines in 1 filestet-6f84e978Judges preferred Opus 5 medium’s patch
  • 29 changed lines in 1 filestet-b28f4fe7
  • 84 changed lines in 1 filestet-66762b4c
  • 113 changed lines in 4 filesstet-commit-f0caded9
  • 218 changed lines in 2 filesstet-commit-5d766dde
  • 941 changed lines in 7 filesstet-commit-bbbbae09

5Neither solved

  • 149 changed lines in 2 filesstet-commit-e928166f
  • 476 changed lines in 17 filesstet-commit-30506f8e
  • 763 changed lines in 8 filesstet-commit-87814583
  • 977 changed lines in 5 filesstet-commit-e41c2e2d
  • 1,236 changed lines in 16 filesstet-commit-d2ab0c25

Bar: changed lines in the merged change, log scale

A task counts as solved when its tests passed or a judge found the fix equivalent to the merged change, as in the leaderboard’s Solved column. Stet score also weighs code quality, so it can order the two differently.

All 24 compared tasks
Task by task
TaskOpus 5 mediumSWE-2 maxSolved byBetter patch
stet-89dfbc27 · 9 changed lines in 1 filePassed tests$1.53 · 32.6 minPassed tests$0.71 · 4.5 minBothEven
stet-b28f4fe7 · 29 changed lines in 1 filePassed tests$0.69 · 19.4 minPassed tests$2.02 · 16.7 minBothEven
stet-232b50ba · 10 changed lines in 1 filePassed tests$1.21 · 7.9 minPassed tests$1.56 · 11.9 minBothEven
stet-3169aefa · 6 changed lines in 1 filePassed tests$1.09 · 9.2 minPassed tests$1.03 · 5.3 minBothEven
stet-5276aa3f · 23 changed lines in 2 filesPassed tests$1.03 · 7.7 minPassed tests$1.03 · 10.7 minBothEven
stet-66762b4c · 84 changed lines in 1 filePassed tests$1.74 · 36.7 minPassed tests$1.77 · 11.2 minBothEven
stet-commit-bbbbae09 · 941 changed lines in 7 filesPassed tests$8.24 · 18.7 minPassed tests$0.61 · 89.7 minBothEven
stet-commit-5d766dde · 218 changed lines in 2 filesEquivalent to the merged fix$3.36 · 11.0 minEquivalent to the merged fix$4.04 · 20.0 minBothEven
stet-43493c52 · 4 changed lines in 1 filePassed tests$0.52 · 2.3 minFailed$3.52 · 35.3 minOpus 5 medium onlyOpus 5 medium
stet-commit-f0caded9 · 113 changed lines in 4 filesPassed tests$3.43 · 10.8 minPassed tests$3.91 · 22.6 minBothEven
stet-15439c21 · 5 changed lines in 2 filesPassed tests$0.53 · 4.9 minPassed tests$2.24 · 34.3 minBothEven
stet-commit-f8ae1177 · 1,312 changed lines in 9 filesPassed tests$18.97 · 27.5 minFailed300.0 minOpus 5 medium onlyOpus 5 medium
stet-2450ca2d · 2 changed lines in 1 filePassed tests$0.82 · 21.7 minPassed tests$1.18 · 11.8 minBothEven
stet-commit-342d0ac0 · 640 changed lines in 8 filesFailed$5.56 · 16.4 minEquivalent to the merged fix$2.39 · 146.4 minSWE-2 max onlySWE-2 max
stet-6f84e978 · 23 changed lines in 1 filePassed tests$1.39 · 34.9 minPassed tests$2.06 · 42.4 minBothOpus 5 medium
stet-commit-5e075641 · 481 changed lines in 3 filesFailed$10.93 · 31.1 minPassed tests$2.64 · 86.0 minSWE-2 max onlySWE-2 max
stet-commit-60a0e6c2 · 552 changed lines in 11 filesFailed$7.75 · 24.8 minPassed tests$0.66 · 106.1 minSWE-2 max onlySWE-2 max
stet-commit-2826765e · 1,420 changed lines in 19 filesEquivalent to the merged fix$17.19 · 30.1 minFailed$7.00 · 144.5 minOpus 5 medium onlyOpus 5 medium
stet-commit-a8716b88 · 564 changed lines in 10 filesFailed$12.40 · 43.1 minEquivalent to the merged fix$1.29 · 102.3 minSWE-2 max onlySWE-2 max
stet-commit-e928166f · 149 changed lines in 2 filesFailed$5.94 · 25.2 minFailed$1.49 · 59.4 minNeitherEven
stet-commit-30506f8e · 476 changed lines in 17 filesFailed$15.36 · 43.4 minFailed$5.49 · 164.8 minNeitherEven
stet-commit-87814583 · 763 changed lines in 8 filesFailed$14.95 · 33.3 minFailed$5.19 · 73.6 minNeitherEven
stet-commit-d2ab0c25 · 1,236 changed lines in 16 filesFailed$15.17 · 25.8 minFailed$5.66 · 68.9 minNeitherEven
stet-commit-e41c2e2d · 977 changed lines in 5 filesFailed$12.80 · 38.3 minFailed$6.08 · 69.8 minNeitherEven

Cost and time on each task

Each dot is one task, on a log scale: filled where the setup solved it, hollow where it didn’t. The line marks the median.

Cost per task

Opus 5 mediummean $6.78 · median $4.50

SWE-2 maxmean $2.76 · median $2.06from 23/24 tasks

SWE-2 max cost less on 14 of 23 tasks; Opus 5 medium on 9.

Agent time per task

Opus 5 mediummean 23.2 min · median 25.0 min

SWE-2 maxmean 68.3 min · median 50.9 min

Opus 5 medium finished faster on 19 of 24 tasks; SWE-2 max on 5.

Where each setup is the better choice

The leaderboard’s readings on groups of tasks. Each group holds fewer tasks than the overall standings, so treat these as leads to test on your repo.

Small fixes

Cheaper here without solving fewer: Opus 5 medium

10 tasks · 2–84 changed lines in 1–2 files

Opus 5 medium solved more, 10 of 10 to 9, at 1.6× lower cost per task.

Opus 5 medium was a little faster.

Small fixes
Opus 5 mediumSWE-2 max
Solved10/109/10
Cost / task$1.05$1.71
Agent time / task17.7 min18.4 min

Large, cross-cutting changes

14 tasks · 113–1,420 changed lines in up to 19 files

SWE-2 max solved 7 of 14 to Opus 5 medium’s 5.

Opus 5 medium was 3.8× faster.

Large, cross-cutting changes
Opus 5 mediumSWE-2 max
Solved5/147/14
Cost / task$10.86$3.57from 13/14 tasks
Agent time / task27.1 min103.9 min

Side by side

Side by side
MetricOpus 5 mediumSWE-2 max
Harnessclaude-codedevin
Reasoning effortmediummax
Rank2 of 181 of 18
Stet score60.995% range 54.2–68.362.395% range 53.4–71.6
Solved15/2413 passed, 2 judged equivalent16/2413 passed, 3 judged equivalent
Cost / task$6.78$2.76from 23/24 tasks
Agent time / task23.2 min68.3 min

Questions

Which is better for coding, Opus 5 medium or SWE-2 max?
No clear quality difference: Opus 5 medium won 4, SWE-2 max won 4, 16 tied; SWE-2 max leads on cost: 2.5× cheaper; Opus 5 medium leads on agent time: 2.9× faster. Measured on 24 merged changes from Stet-AI/stet.
How were Opus 5 medium and SWE-2 max tested?
Both coding agents worked the same 24 merged changes from the private Stet-AI/stet repository, each in the real repo as it was before the change. The repo’s tests and judges graded every patch against the merged change; no public benchmark scores are used. Stet score: A setup’s average chance of beating another setup on a task, from 0 to 100; 50 is even with the field.

See which setup wins on your repo

Stet replays your merged changes, scores quality above pass/fail, and delivers comparison reports through your agent.

Get started

About this data

  • 24 tasks, each a merged change from the private Stet-AI/stet repository replayed from the repo as it was just before the change.
  • Data from .
  • Both setups come from the same leaderboard run of 18 setups: bmk_d4ba310fdf50db4ccb2092e3.