Pareto 26.9 medium vs SWE-2 Max high
Pareto 26.9 medium leads on speed; quality is too close to call.
- Quality
- +2.8 pts
- SWE-2 Max high
- 4–3 head to head, 16 tied
- Speed
- 1.7× faster
- Pareto 26.9 medium
- 39.2 min vs 68.3 min per task
Stet score
Pareto 26.9 mediummedium reasoningRank 3 of 17
55.295% range 50.5–60.5
SWE-2 Max highhigh reasoningRank 2 of 17
58.095% range 49.7–66.82.8 points higher
Quality
Head-to-head matchups
Pareto 26.9 medium won3
Tied16
SWE-2 Max high won4
Task by task
Pareto 26.9 medium solved 2 tasks that SWE-2 Max high didn’t, and SWE-2 Max high solved 7 that Pareto 26.9 medium didn’t. Both solved 9; neither solved 6.
stet-commit-5d766ddeSolved by 14 of 17 setups
- Pareto 26.9 mediumFailed
- SWE-2 Max highEquivalent to the merged fix
A task counts as solved when its tests passed or a judge found the fix equivalent to the merged change, as in the leaderboard’s Solved column. Stet score also weighs code quality, so it can order the two differently.
All 24 compared tasks
| Task | Pareto 26.9 medium | SWE-2 Max high | Solved by |
|---|---|---|---|
stet-3169aefa | Passed tests | Passed tests | Both |
stet-89dfbc27 | Passed tests | Passed tests | Both |
stet-b28f4fe7 | Passed tests | Passed tests | Both |
stet-232b50ba | Passed tests | Passed tests | Both |
stet-5276aa3f | Passed tests | Passed tests | Both |
stet-commit-bbbbae09 | Passed tests | Passed tests | Both |
stet-66762b4c | Passed tests | Passed tests | Both |
stet-commit-5d766dde | Failed | Equivalent to the merged fix | SWE-2 Max high only |
stet-43493c52 | Passed tests | Failed | Pareto 26.9 medium only |
stet-commit-f0caded9 | Equivalent to the merged fix | Passed tests | Both |
stet-commit-f8ae1177 | Passed tests | Failed | Pareto 26.9 medium only |
stet-15439c21 | Failed | Equivalent to the merged fix | SWE-2 Max high only |
stet-2450ca2d | Passed tests | Passed tests | Both |
stet-commit-342d0ac0 | Failed | Equivalent to the merged fix | SWE-2 Max high only |
stet-6f84e978 | Failed | Passed tests | SWE-2 Max high only |
stet-commit-60a0e6c2 | Failed | Passed tests | SWE-2 Max high only |
stet-commit-5e075641 | Failed | Passed tests | SWE-2 Max high only |
stet-commit-2826765e | Failed | Failed | Neither |
stet-commit-a8716b88 | Failed | Equivalent to the merged fix | SWE-2 Max high only |
stet-commit-e928166f | Failed | Failed | Neither |
stet-commit-30506f8e | Failed | Failed | Neither |
stet-commit-87814583 | Failed | Failed | Neither |
stet-commit-d2ab0c25 | Failed | Failed | Neither |
stet-commit-e41c2e2d | Failed | Failed | Neither |
Side by side
| Metric | Pareto 26.9 medium | SWE-2 Max high |
|---|---|---|
| Reasoning effort | medium | high |
| Rank | 3 of 17 | 2 of 17 |
| Stet score | 55.295% range 50.5–60.5 | 58.095% range 49.7–66.8 |
| Solved | 11/2410 passed, 1 judged equivalent | 16/2412 passed, 4 judged equivalent |
| Cost / task | $5.98 | not recorded |
| Agent time / task | 39.2 min | 68.3 min |
See which setup wins on your repo
Stet replays your merged PRs, scores quality above pass/fail, and delivers comparison reports through your agent.
Get startedAbout this data
- 24 tasks, each a merged change replayed from the repo as it was just before the merge.
- Data from .
- Both setups come from the same leaderboard run of 17 setups:
bmk_d4ba310fdf50db4ccb2092e3.