# GPT 6.1 Sol medium vs Haiku 5.5 max: real-repository coding test

Stet publishes a public search API over its indexed pages; the guide is at https://www.stet.sh/agents/agent-access.

Source: https://www.stet.sh/leaderboard/compare/gpt-6-1-sol-medium-vs-haiku-5-5-max
Source observed: 2026-11-19T11:08:21.216Z
Last checked: 2026-11-19T11:08:21.216+00:00
Indexed: 2026-11-19T11:19:14.399+00:00

---

# GPT 6.1 Sol medium vs Haiku 5.5 max

我们在不同 harness 上比较两个编码 agent 配置：GPT 6.1 Sol medium 在 `codex`、medium reasoning 中运行；Haiku 5.5 max 在 `claude-code`、max reasoning 中运行。因此结果比较的是 setups，不是模型单独的能力。

## Headline result

- **Quality:** 没有明确差异。GPT 6.1 Sol medium 赢 5 个 head-to-head，Haiku 5.5 max 赢 3 个，15 个打平，1 个未决。
- **Speed:** GPT 6.1 Sol medium **5.0× faster**：11.8 versus 58.4 分钟 per task。
- **Cost:** GPT 6.1 Sol medium **3.5× cheaper**：$1.10 versus $3.83 per task。

## What this comparison is

- One private real repository: `Stet-AI/stet`; not public benchmark scores.
- 24 merged changes, each run end to end from the repository state before the change.
- Tests from the repository and judges comparing each patch with the merged change graded the work.
- We measured quality, cost, and time on real agent work; this is a smaller sample than public benchmarks.

[GPT 6.1 Sol vs Haiku 5.5: all effort pairings](https://www.stet.sh/leaderboard/compare/gpt-6-1-sol-vs-haiku-5-5)

Data: November 19, 2026. Both setups came from leaderboard run `bmk_d4ba310fdf50db4ccb2092e3` of 18 setups.

## Stet score and rank

| Setup | Harness | Reasoning | Rank | Stet score | 95% range |
| --- | --- | --- | --- | --- | --- |
| GPT 6.1 Sol medium | `codex` | medium | 4 of 18 | 54.7% | 46.0–63.4 |
| Haiku 5.5 max | `claude-code` | max | 10 of 18 | 48.7% | 38.3–58.7 |

The score ranges overlap. GPT 6.1 Sol medium’s win share was 55%, with a 95% range of 43–67%; ties are within judge noise.

## Other quality measures

These measures are not part of the headline Stet score and have no confidence interval.

| Metric | GPT 6.1 Sol medium | Haiku 5.5 max |
| --- | --- | --- |
| Code review pass rate | 43% — 23 tasks | 38% — 24 tasks |
| Checklist score | 0.88 — 24 tasks | 0.81 — 24 tasks |
| Equivalence rate | 43% — 23 tasks | 42% — 24 tasks |

The higher recorded value is not a proven lead.

## Task-by-task outcomes

GPT 6.1 Sol medium solved 4 tasks that Haiku did not; Haiku solved 3 that GPT 6.1 did not. Both solved 7; neither solved 9. On the 7 tasks both solved, judges preferred GPT 6.1’s patch on 1 and called 6 even. One of the 24 tasks is excluded because one setup has no scored result.

### Only GPT 6.1 Sol medium solved

- `stet-43493c52` — 4 changed lines in 1 file; judges preferred GPT 6.1’s patch.
- `stet-232b50ba` — 10 changed lines in 1 file; judges preferred GPT 6.1’s patch.
- `stet-66762b4c` — 84 changed lines in 1 file; judges preferred GPT 6.1’s patch.
- `stet-commit-f8ae1177` — 1,312 changed lines in 9 files; judges preferred GPT 6.1’s patch.

### Only Haiku 5.5 max solved

- `stet-2450ca2d` — 2 changed lines in 1 file; judges preferred Haiku’s patch.
- `stet-15439c21` — 5 changed lines in 2 files; judges preferred Haiku’s patch.
- `stet-6f84e978` — 23 changed lines in 1 file; judges preferred Haiku’s patch.

### Both solved

- `stet-3169aefa` — 6 changed lines in 1 file.
- `stet-89dfbc27` — 9 changed lines in 1 file.
- `stet-5276aa3f` — 23 changed lines in 2 files.
- `stet-b28f4fe7` — 29 changed lines in 1 file.
- `stet-commit-5d766dde` — 218 changed lines in 2 files.
- `stet-commit-342d0ac0` — 640 changed lines in 8 files.
- `stet-commit-bbbbae09` — 941 changed lines in 7 files; judges preferred GPT 6.1’s patch.

### Neither solved

- `stet-commit-f0caded9` — 113 changed lines in 4 files.
- `stet-commit-e928166f` — 149 changed lines in 2 files.
- `stet-commit-30506f8e` — 476 changed lines in 17 files.
- `stet-commit-5e075641` — 481 changed lines in 3 files.
- `stet-commit-a8716b88` — 564 changed lines in 10 files.
- `stet-commit-87814583` — 763 changed lines in 8 files.
- `stet-commit-e41c2e2d` — 977 changed lines in 5 files.
- `stet-commit-d2ab0c25` — 1,236 changed lines in 16 files.
- `stet-commit-2826765e` — 1,420 changed lines in 19 files.

## All 23 compared tasks

| Task | GPT 6.1 Sol medium | Haiku 5.5 max | Solved by | Better patch |
| --- | --- | --- | --- | --- |
| `stet-89dfbc27` · 9 lines, 1 file | Passed; $0.18 · 2.8 min | Passed; $0.19 · 6.3 min | Both | Even |
| `stet-b28f4fe7` · 29 lines, 1 file | Passed; $0.73 · 7.1 min | Passed; $1.06 · 22.2 min | Both | Even |
| `stet-232b50ba` · 10 lines, 1 file | Passed; $0.24 · 3.8 min | Failed; $3.21 · 45.5 min | GPT 6.1 only | GPT 6.1 |
| `stet-3169aefa` · 6 lines, 1 file | Passed; $0.27 · 3.6 min | Passed; $0.94 · 38.5 min | Both | Even |
| `stet-5276aa3f` · 23 lines, 2 files | Passed; $0.54 · 6.5 min | Passed; $0.38 · 25.3 min | Both | Even |
| `stet-66762b4c` · 84 lines, 1 file | Passed; $0.32 · 3.6 min | Failed; $0.68 · 39.4 min | GPT 6.1 only | GPT 6.1 |
| `stet-commit-bbbbae09` · 941 lines, 7 files | Passed; $1.63 · 25.4 min | Passed; $3.83 · 47.8 min | Both | GPT 6.1 |
| `stet-commit-5d766dde` · 218 lines, 2 files | Equivalent; $0.77 · 9.8 min | Equivalent; $1.86 · 31.7 min | Both | Even |
| `stet-43493c52` · 4 lines, 1 file | Passed; $0.39 · 5.7 min | Failed; $1.15 · 71.0 min | GPT 6.1 only | GPT 6.1 |
| `stet-commit-f0caded9` · 113 lines, 4 files | Failed; $0.87 · 9.2 min | Failed; $8.99 · 153.7 min | Neither | Even |
| `stet-15439c21` · 5 lines, 2 files | Failed; $0.61 · 8.6 min | Passed; $0.53 · 24.5 min | Haiku only | Haiku |
| `stet-commit-f8ae1177` · 1,312 lines, 9 files | Passed; $1.83 · 28.7 min | Failed; $12.35 · 126.3 min | GPT 6.1 only | GPT 6.1 |
| `stet-2450ca2d` · 2 lines, 1 file | Failed; $0.26 · 4.2 min | Passed; $1.48 · 56.7 min | Haiku only | Haiku |
| `stet-commit-342d0ac0` · 640 lines, 8 files | Equivalent; $1.49 · 14.4 min | Equivalent; $7.41 · 82.3 min | Both | Even |
| `stet-6f84e978` · 23 lines, 1 file | Failed; $0.50 · 5.4 min | Equivalent; $1.47 · 34.2 min | Haiku only | Haiku |
| `stet-commit-5e075641` · 481 lines, 3 files | Failed; $2.02 · 14.3 min | Failed; $4.10 · 59.0 min | Neither | Even |
| `stet-commit-2826765e` · 1,420 lines, 19 files | Failed; $1.60 · 13.8 min | Failed; $4.11 · 78.2 min | Neither | Even |
| `stet-commit-a8716b88` · 564 lines, 10 files | Failed; $1.52 · 16.6 min | Failed; $4.94 · 95.4 min | Neither | Even |
| `stet-commit-e928166f` · 149 lines, 2 files | Failed; $1.14 · 12.2 min | Failed; $1.72 · 35.6 min | Neither | Even |
| `stet-commit-30506f8e` · 476 lines, 17 files | Failed; $0.81 · 20.7 min | Failed; $8.29 · 100.0 min | Neither | Even |
| `stet-commit-87814583` · 763 lines, 8 files | Failed; $3.24 · 17.9 min | Failed; $4.38 · 61.9 min | Neither | Even |
| `stet-commit-d2ab0c25` · 1,236 lines, 16 files | Failed; $2.16 · 15.3 min | Failed; $10.68 · 92.8 min | Neither | Even |
| `stet-commit-e41c2e2d` · 977 lines, 5 files | Failed; $1.92 · 18.9 min | Failed; $3.05 · 44.0 min | Neither | Even |

## Cost and time

- **GPT 6.1 Sol medium:** mean $1.09; median $0.81; cost less on 21 of 23 tasks.
- **Haiku 5.5 max:** mean $3.77; median $3.05; cost less on 2 tasks.
- **GPT 6.1 Sol medium:** mean 11.7 minutes; median 9.8 minutes.
- **Haiku 5.5 max:** mean 59.7 minutes; median 47.8 minutes.
- GPT 6.1 Sol medium finished faster on all 23 tasks.

## Where each setup may fit

### Small fixes

10 tasks; 2–84 changed lines in 1–2 files. Both solved 7/10. GPT 6.1 Sol medium cost 2.8× less and was 7.1× faster.

| Metric | GPT 6.1 Sol medium | Haiku 5.5 max |
| --- | ---: | ---: |
| Solved | 7/10 | 7/10 |
| Cost per task | $0.40 | $1.11 |
| Agent time per task | 5.1 min | 36.4 min |

### Large, cross-cutting changes

14 tasks; 113–1,420 changed lines in up to 19 files. GPT 6.1 solved 4/13 versus Haiku’s 3/13, cost 3.6× less, and was 4.6× faster.

| Metric | GPT 6.1 Sol medium | Haiku 5.5 max |
| --- | ---: | ---: |
| Solved | 4/13 | 3/13 |
| Cost per task | $1.62 | $5.82 |
| Agent time per task | 16.7 min | 77.6 min |

## Side by side

| Metric | GPT 6.1 Sol medium | Haiku 5.5 max |
| --- | --- | --- |
| Harness | `codex` | `claude-code` |
| Reasoning effort | medium | max |
| Rank | 4 of 18 | 10 of 18 |
| Stet score | 54.7%; 95% range 46.0–63.4 | 48.7%; 95% range 38.3–58.7 |
| Solved | 11/24 — 9 passed, 2 judged equivalent | 10/24 — 7 passed, 3 judged equivalent |
| Cost per task | $1.10 | $3.83 |
| Agent time per task | 11.8 min | 58.4 min |

## Research links

- [Cost per successful task ranked the coding agent that lost every head to head 1st](https://www.stet.sh/blog/cost-per-successful-task)
- [A benchmark mined from my own repo agreed with FrontierCode](https://www.stet.sh/blog/benchmark-agreed-with-frontier-code)
- [I ran GPT 6.1 Sol against GPT 6 Sol on 24 tasks from my own repo, and found that it maintained quality while cutting cost in half](https://www.stet.sh/blog/gpt-6-1-sol-vs-gpt-6-sol)
- [Get started](https://www.stet.sh/private)
- [See all 18 setups](https://www.stet.sh/leaderboard)
- [How Stet measures](https://www.stet.sh/methodology)

## FAQ

**Which is better for coding?** Quality was not clearly different: GPT 6.1 won 5, Haiku won 3, 15 tied, and 1 was undecided. GPT 6.1 led on cost and agent time: 3.5× cheaper and 5.0× faster.

**How did we test them?** Both coding agents worked on the same 24 merged changes from private `Stet-AI/stet`, starting from the repository state before each change. Tests and judges graded each patch against the merged change; no public benchmark scores were used.

We replay your merged changes, score quality above pass/fail, and deliver comparison reports through your agent. [Get started](https://www.stet.sh/private)

---

[Read this page’s summary](https://www.stet.sh/agents/api/public/v1/page-tldr?url=https%3A%2F%2Fwww.stet.sh%2Fleaderboard%2Fcompare%2Fgpt-6-1-sol-medium-vs-haiku-5-5-max&trace=t_5da45423e484)

## Ask a follow-up about this page

Need a more specific answer? Ask the exact question you have. It returns a fresh, cited answer from Stet’s knowledge base. No account or API key required.

Open this URL with your own question, URL-encoded. Include your question in q. All research inputs are optional; no account or setup.

`GET https://www.stet.sh/agents/agent/faq?q={question}&trace=t_5da45423e484`

[Tool guide and examples](https://www.stet.sh/agents/agent-access) · [OpenAPI](https://www.stet.sh/agents/api/public/v1/openapi.json) · [Page TLDR form](https://www.stet.sh/agents/agent-tools/page-tldr?url=https%3A%2F%2Fwww.stet.sh%2Fleaderboard%2Fcompare%2Fgpt-6-1-sol-medium-vs-haiku-5-5-max). Opening a form does not execute a tool.