GPT 6.1 Sol vs GPT 6 Sol
A cheaper replacement that holds quality.
Head to head
9-10-5
9 wins10 ties5 losses
Cost per task
$1.10 vs $3.01
cheaper on 22 of 24
From merged work to a decision you can defend.
The four steps
Builds tasks from your merged PRs
Each task is a merged change in your repo, with the repo snapshot from just before it merged, a prompt and the tests, which the agent never sees.
The agent is prevented from fetching the result via GitHub.
Runs each setup on the same work
Gives each setup the same tasks, tests, and quality criteria.
Each setup works in its own container, built from your repo with its dependencies installed.
Runs on the Claude, Codex or Cursor login you already have.
- Queued
- Onboarding repository
- Building tasks
- Choosing tasks
- Running and grading the tasks
- Publishing report
- Complete
Grades the work above pass/fail
Combines correctness, checklist and review into one score per patch.
- Correctness
- Original tests passing or failing, or equivalence to the merged change.
- Checklist
- Each task’s required behaviors, written without seeing any agent’s patch and judged item by item.
- Code review
- Code reuse, abstraction levels, test style, dead code, guards and scope. Basically: would a senior engineer want to merge this?
- Cost and time
- Tokens, coding cost and minutes for every task.
Sends you the report
Compares quality, cost and time per task, and recommends a setup.
It’s not just prompts.
Model swaps, harness changes, skill edits, config flips. Each one checked the same way.
repo/
AGENTS.mdunchanged
CLAUDE.mdunchanged
.claude/skills/unchanged
session
modelhaiku 5.5
effortmax → low
−93%
cost, giving up max’s narrow edge
- max vs low
- 12-4-8
- cost/task
- $3.85 → $0.25
Tell your coding agent to improve its own setup.
Ask your coding agent, or click Create Benchmark at app.stet.sh. Stet builds the tasks from your merged changes and you get the report back.
What should my default reasoning effort be?
Where it runs
Local
Keeps your code, tests, and credentials on your machine.
- Your code
- Stet never receives your code or tests.
- Logins
- The agents call your model provider directly, on your own login.
- Stet sees
- Your sign-in, and a usage log of the commands and model names you run.
Hosted
Uploads your code, tests, and model login to Stet to run the eval.
- Sandbox
- Your repo’s commands and the agent run only inside an isolated sandbox.
- Logins
- Your model login never enters the sandbox. A relay outside it forwards the agent’s requests, and logins sent with a run are kept encrypted for at most 30 days so retries work.
- Your code
- Stet keeps a snapshot of your repo only to run and rerun your benchmarks. Only your org can read the snapshot, the reports and the traces.
- Deletion
- Delete a run or all of your org’s data at any time. Stet purges the code, reports and logins from storage, its database and sandbox snapshots.
Ready to test what works on your codebase?
stetsh