Skip to result
STET
Get a free trial

Living benchmark · 24 merged changes from Stet’s own repo, replayed by 15 AI coding setups · Updated October 21, 2026

GPT 6.1 Sol medium leads on quality vs. cost

Second of 15 on quality, 0.1 points above Opus 5. SWE-2 Max ranked first, but its cost wasn’t recorded.

Stet score
65.8
Rank 2 of 15 · Opus 5 65.7
Average chance out of 100 of beating another setup on a task
Cost per task
$1.10
6.2× cheaper than Opus 5
Agent time per task
11.8 min
2.0× faster than Opus 5

A directional read from 24 tasks in one repo; the top three are within 1.0 points. How this was measured

Score against cost

6 setups, from GPT 6 Luna low to GPT 6.1 Sol medium, give the best score for their cost

Compare Stet score with cost or time; the red line connects the best averages.

Stet scoreStet score summarizes task matchup wins on a 0–100 scale. 50 means a setup is even with the field.

GPT 6.1 Sol mediumrank 2 of 15

Stet score 65.8 (56.0–75.1) · $1.10 a task · 11.8 min a task

On the line: no cheaper setup scores higher on average, and no setup reliably beats it on both score and cost.

Hover, tap or tab to another point to read it.

Standings

Setups ranked by Stet score

Stet score is a setup’s average chance, out of 100, of beating another setup on a task. Solved counts tasks whose tests passed or whose fix matched the merged change.

#SetupStet scoreStet score summarizes task matchup wins on a 0–100 scale. 50 means a setup is even with the field.SolvedCost / taskTime / task
1SWE-2 Maxdevin · high reasoningRank 1. Stet score 66.7, 95% range 56.8 to 75.9. Solved 16 of 24: 13 passed tests, 3 judged equivalent to the merged fix. No cost recorded. 68.3 min a task on average.
More metrics
  • Tasks solved16/24 (13 passed, 3 judged equivalent)
  • Time / task68.3 min
2GPT 6.1 Sol mediumcodex · medium reasoningLeads on quality vs. costRank 2. Stet score 65.8, 95% range 56.0 to 75.1. Solved 13 of 24: 10 passed tests, 3 judged equivalent to the merged fix. $1.10 a task. 11.8 min a task on average.
More metrics
  • Tasks solved13/24 (10 passed, 3 judged equivalent)
  • Time / task11.8 min
3Opus 5medium reasoningRank 3. Stet score 65.7, 95% range 55.3 to 75.5. Solved 14 of 24: 12 passed tests, 2 judged equivalent to the merged fix. $6.78 a task. 23.2 min a task on average.
More metrics
  • Tasks solved14/24 (12 passed, 2 judged equivalent)
  • Time / task23.2 min
4Grok 4.7 Highcursor · high reasoningRank 4. Stet score 64.0, 95% range 53.2 to 73.8. Solved 13 of 24: 11 passed tests, 2 judged equivalent to the merged fix. $5.76 a task. 18.9 min a task on average.
More metrics
  • Tasks solved13/24 (11 passed, 2 judged equivalent)
  • Time / task18.9 min
5Pareto 26.9claude-code · medium reasoningRank 5. Stet score 60.0, 95% range 49.5 to 70.0. Solved 13 of 24: 12 passed tests, 1 judged equivalent to the merged fix. $12.47 a task. 39.2 min a task on average.
More metrics
  • Tasks solved13/24 (12 passed, 1 judged equivalent)
  • Time / task39.2 min
6Haiku 5.5 maxclaude-code · max reasoningRank 6. Stet score 58.9, 95% range 47.6 to 69.4. Solved 12 of 24: 9 passed tests, 3 judged equivalent to the merged fix. $3.85 a task. 58.4 min a task on average.
More metrics
  • Tasks solved12/24 (9 passed, 3 judged equivalent)
  • Time / task58.4 min
7GPT 6 Sol mediumcodex · medium reasoningRank 7. Stet score 58.5, 95% range 50.6 to 66.3. Solved 14 of 24: 11 passed tests, 3 judged equivalent to the merged fix. $3.01 a task. 15.4 min a task on average.
More metrics
  • Tasks solved14/24 (11 passed, 3 judged equivalent)
  • Time / task15.4 min
8GPT 6 Luna maxcodex · max reasoningRank 8. Stet score 50.8, 95% range 41.2 to 60.0. Solved 12 of 24: 8 passed tests, 4 judged equivalent to the merged fix. $0.37 a task. 59.7 min a task on average.
More metrics
  • Tasks solved12/24 (8 passed, 4 judged equivalent)
  • Time / task59.7 min
9DeepSeek V4.1 Flashdeepseek-harness · max reasoningRank 9. Stet score 48.1, 95% range 37.6 to 58.5. Solved 12 of 24: 11 passed tests, 1 judged equivalent to the merged fix. $0.49 a task over 23 of 24 tasks. 49.3 min a task on average.
More metrics
  • Tasks solved12/24 (11 passed, 1 judged equivalent)
  • Time / task49.3 min
10Haiku 5.5 low+guidanceclaude-code · low reasoningRank 10. Stet score 42.4, 95% range 33.2 to 51.7. Solved 9 of 24: 8 passed tests, 1 judged equivalent to the merged fix. $0.29 a task. 12.4 min a task on average.
More metrics
  • Tasks solved9/24 (8 passed, 1 judged equivalent)
  • Time / task12.4 min
11Haiku 5.5 lowclaude-code · low reasoningRank 11. Stet score 41.9, 95% range 35.3 to 48.5. Solved 11 of 24: 7 passed tests, 4 judged equivalent to the merged fix. $0.25 a task. 9.9 min a task on average.
More metrics
  • Tasks solved11/24 (7 passed, 4 judged equivalent)
  • Time / task9.9 min
12Opus 4.8claude-code · medium reasoningRank 12. Stet score 40.8, 95% range 30.8 to 50.9. Solved 13 of 24: 11 passed tests, 2 judged equivalent to the merged fix. $7.05 a task. 23.3 min a task on average.
More metrics
  • Tasks solved13/24 (11 passed, 2 judged equivalent)
  • Time / task23.3 min
13Muse controlmuse · high reasoningRank 13. Stet score 34.1, 95% range 25.3 to 43.3. Solved 10 of 24: 8 passed tests, 2 judged equivalent to the merged fix. $0.10 a task. 13.2 min a task on average.
More metrics
  • Tasks solved10/24 (8 passed, 2 judged equivalent)
  • Time / task13.2 min
14GLM 5.3 Flashcodex · high reasoningRank 14. Stet score 31.1, 95% range 21.5 to 41.9. Solved 9 of 24: 9 passed tests, 0 judged equivalent to the merged fix. $0.41 a task. 26.7 min a task on average.
More metrics
  • Tasks solved9/24 (9 passed, 0 judged equivalent)
  • Time / task26.7 min
15GPT 6 Luna lowcodex · low reasoningRank 15. Stet score 21.2, 95% range 12.5 to 31.2. Solved 5 of 24: 5 passed tests, 0 judged equivalent to the merged fix. $0.05 a task. 6.9 min a task on average.
More metrics
  • Tasks solved5/24 (5 passed, 0 judged equivalent)
  • Time / task6.9 min
passed tests review judged equivalent to the fix. Score whiskers show the 95% range.

Task by task

SWE-2 Max solved most (16/24) · 5 unsolved

SWE-2 Max16
Opus 514
GPT 6 Sol medium14
GPT 6.1 Sol medium13
Grok 4.7 High13
Pareto 26.913
Opus 4.813
Haiku 5.5 max12
GPT 6 Luna max12
DeepSeek V4.1 Flash12
Haiku 5.5 low11
Muse control10
Haiku 5.5 low+guidance9
GLM 5.3 Flash9
GPT 6 Luna low5
Passed testsEquivalent to the merged fixFailedNo change madeNot scored

Compare two setups

Any two setups side by side on the same 24 tasks: Stet score, cost, agent time and which tasks each solved.

All 105 comparisons

See which setup wins on your repo

Stet replays your merged PRs, scores quality above pass/fail, and delivers comparison reports through your agent.

Run it on your repo

How this was measured

Each setup ran the same 24 tasks. Each task is a change merged into Stet’s repo, started from the repo as it was just before the merge. AI judges read each fix: “passed review” means an AI code reviewer approved it, and “matched the merged change” means a judge found it does what the merged change did.

Stet score is a setup’s average chance of beating another setup on a task, from 0 to 100, so 50 is even with the field. On each task the more correct fix wins: passing the tests beats matching the merged change, which beats failing. Between equally correct fixes the judges’ code-quality scores decide, and near-equal scores tie. The score’s range comes from resampling the tasks.

Cost is the coding cost each harness reported, or Stet’s estimate from recorded tokens. Agent time is how long the agent worked on a task, not end-to-end delivery time. Setups ran different test commands on some tasks, so those test results carry no weight in Stet score. The solved counts include them and are shown for reference only: tasks whose tests passed plus fixes judged equivalent to the merged change.

Setups are whole bundles: model, reasoning effort, harness and its prompts. Stet’s own read of this run is low confidence: check the evidence before rolling out any one setup. Code review didn’t cover every task, and setups recorded cost in different ways. The order above is a directional read.

Download the results (JSON)Download the summary (Markdown)

Report identity
Generated
Published
Report SHA-256
0bbc1f6210cf4ed08f9a425024a6e7e45e25a003fc5fe3585536c3c694d96f46
Published version
sha256:d34123b5df1396ebc24f274af135704f492fee23dd2f05ba6770e7a6d0970661
The 24 tasks
  • stet-15439c21
  • stet-232b50ba
  • stet-2450ca2d
  • stet-3169aefa
  • stet-43493c52
  • stet-5276aa3f
  • stet-66762b4c
  • stet-6f84e978
  • stet-89dfbc27
  • stet-b28f4fe7
  • stet-commit-2826765e
  • stet-commit-30506f8e
  • stet-commit-342d0ac0
  • stet-commit-5d766dde
  • stet-commit-5e075641
  • stet-commit-60a0e6c2
  • stet-commit-87814583
  • stet-commit-a8716b88
  • stet-commit-bbbbae09
  • stet-commit-d2ab0c25
  • stet-commit-e41c2e2d
  • stet-commit-e928166f
  • stet-commit-f0caded9
  • stet-commit-f8ae1177