Opus 4.8 medium vs Pareto 26.9 medium
Opus 4.8 medium leads on speed; quality is too close to call.
- Quality
- +6.4 pts
- Pareto 26.9 medium
- 6–4 head to head, 13 tied
- Speed
- 1.7× faster
- Opus 4.8 medium
- 23.3 min vs 39.2 min per task
- Cost
- a little cheaper
- Pareto 26.9 medium
- $5.98 vs $6.27 per task
Stet score
Opus 4.8 mediummedium reasoningRank 11 of 17
48.895% range 39.7–58.0
Pareto 26.9 mediummedium reasoningRank 3 of 17
55.295% range 50.5–60.56.4 points higher
Quality
Head-to-head matchups
Opus 4.8 medium won4
Tied13
Pareto 26.9 medium won6
Task by task
Opus 4.8 medium solved 3 tasks that Pareto 26.9 medium didn’t, and Pareto 26.9 medium solved 1 that Opus 4.8 medium didn’t. Both solved 10; neither solved 10.
stet-commit-5d766ddeSolved by 14 of 17 setups
- Opus 4.8 mediumEquivalent to the merged fix
- Pareto 26.9 mediumFailed
A task counts as solved when its tests passed or a judge found the fix equivalent to the merged change, as in the leaderboard’s Solved column. Stet score also weighs code quality, so it can order the two differently.
All 24 compared tasks
| Task | Opus 4.8 medium | Pareto 26.9 medium | Solved by |
|---|---|---|---|
stet-3169aefa | Passed tests | Passed tests | Both |
stet-89dfbc27 | Passed tests | Passed tests | Both |
stet-b28f4fe7 | Passed tests | Passed tests | Both |
stet-232b50ba | Passed tests | Passed tests | Both |
stet-5276aa3f | Passed tests | Passed tests | Both |
stet-commit-bbbbae09 | Passed tests | Passed tests | Both |
stet-66762b4c | Passed tests | Passed tests | Both |
stet-commit-5d766dde | Equivalent to the merged fix | Failed | Opus 4.8 medium only |
stet-43493c52 | Passed tests | Passed tests | Both |
stet-commit-f0caded9 | Equivalent to the merged fix | Equivalent to the merged fix | Both |
stet-commit-f8ae1177 | Passed tests | Passed tests | Both |
stet-15439c21 | Failed | Failed | Neither |
stet-2450ca2d | Failed | Passed tests | Pareto 26.9 medium only |
stet-commit-342d0ac0 | Failed | Failed | Neither |
stet-6f84e978 | Passed tests | Failed | Opus 4.8 medium only |
stet-commit-60a0e6c2 | Passed tests | Failed | Opus 4.8 medium only |
stet-commit-5e075641 | Failed | Failed | Neither |
stet-commit-2826765e | Failed | Failed | Neither |
stet-commit-a8716b88 | Failed | Failed | Neither |
stet-commit-e928166f | Failed | Failed | Neither |
stet-commit-30506f8e | Failed | Failed | Neither |
stet-commit-87814583 | Failed | Failed | Neither |
stet-commit-d2ab0c25 | Failed | Failed | Neither |
stet-commit-e41c2e2d | Failed | Failed | Neither |
Side by side
| Metric | Opus 4.8 medium | Pareto 26.9 medium |
|---|---|---|
| Reasoning effort | medium | medium |
| Rank | 11 of 17 | 3 of 17 |
| Stet score | 48.895% range 39.7–58.0 | 55.295% range 50.5–60.5 |
| Solved | 13/2411 passed, 2 judged equivalent | 11/2410 passed, 1 judged equivalent |
| Cost / task | $6.27 | $5.98 |
| Agent time / task | 23.3 min | 39.2 min |
See which setup wins on your repo
Stet replays your merged PRs, scores quality above pass/fail, and delivers comparison reports through your agent.
Get startedAbout this data
- 24 tasks, each a merged change replayed from the repo as it was just before the merge.
- Data from .
- Both setups come from the same leaderboard run of 17 setups:
bmk_d4ba310fdf50db4ccb2092e3.