# GPT 6.1 Sol medium vs Haiku 5.5 low: real-repo coding test

Stet publishes a public search API over its indexed pages; the guide is at https://www.stet.sh/agents/agent-access.

Source: https://www.stet.sh/leaderboard/compare/gpt-6-1-sol-medium-vs-haiku-5-5-low
Source observed: 2026-11-19T11:08:37.687Z
Last checked: 2026-11-19T11:08:37.687+00:00
Indexed: 2026-11-19T11:16:42.773+00:00

---

# GPT 6.1 Sol medium vs Haiku 5.5 low

We compare two coding-agent setups on real repository work. Haiku 5.5 low leads on speed and cost; we found no clear quality difference. GPT 6.1 Sol medium ran `codex` with medium reasoning, while Haiku 5.5 low ran `claude-code` with low reasoning, so this compares setups rather than models alone.

## At a glance

- **Quality:** Even; no clear difference. GPT 6.1 Sol medium won 6 head-to-head tasks, Haiku 5.5 low won 2, and 16 were tied.
- **Speed:** Haiku 5.5 low was **1.2× faster**: 9.9 minutes versus 11.8 minutes per task.
- **Cost:** Haiku 5.5 low was **4.4× cheaper**: $0.25 versus $1.10 per task.

## What this comparison is

- **1 real repository:** private `Stet-AI/stet`; these are not public benchmark scores.
- **24 merged changes:** each setup ran end to end from the repository state before the change.
- **Grading against the merged change:** we used the repository’s tests and judges comparing each patch with the merged change.
- **Quality, cost, and time:** this is a smaller sample than public benchmarks, but it measures agent work on a real codebase.

[GPT 6.1 Sol vs Haiku 5.5: all effort pairings](https://www.stet.sh/leaderboard/compare/gpt-6-1-sol-vs-haiku-5-5)

**24 tasks · Data from November 19, 2026**

## Stet score

| Setup | Harness and reasoning | Rank | Stet score | 95% range |
| --- | --- | --- | --- | --- |
| GPT 6.1 Sol medium | `codex` · medium reasoning | 4 of 18 | 54.7 | 46.0–63.4 |
| Haiku 5.5 low | `claude-code` · low reasoning | 16 of 18 | 44.9 | 39.1–50.3 |

Their 95% score ranges overlap. Stet score is a setup’s average chance of beating another setup on a task, from 0 to 100; 50 is even with the field.

## Quality

### Head-to-head matchups

- GPT 6.1 Sol medium won **6**.
- **16** were tied.
- Haiku 5.5 low won **2**.
- GPT 6.1 Sol medium’s win share was **57%**, with a 95% range of **46–69%**.

Ties are within judge noise.

### Other quality measures

| Metric | GPT 6.1 Sol medium | Haiku 5.5 low |
| --- | --- | --- |
| Code review pass rate | 43% — 23 tasks | 33% — 24 tasks |
| Checklist score | 0.88 — 24 tasks | 0.77 — 24 tasks |
| Equivalence rate | 43% — 23 tasks | 38% — 24 tasks |

These measures are not part of the headline Stet score and have no confidence interval. A small gap may be noise; the higher recorded value is not a proven lead.

## Task by task

GPT 6.1 Sol medium solved 2 tasks that Haiku 5.5 low did not, and Haiku 5.5 low solved 2 that GPT 6.1 Sol medium did not. Both solved 9; neither solved 11. On the 9 tasks both solved, judges preferred GPT 6.1 Sol medium’s patch on 3 and called 6 even.

### Only GPT 6.1 Sol medium solved — 2

- 640 changed lines in 8 files — `stet-commit-342d0ac0`; judges preferred GPT 6.1 Sol medium’s patch.
- 1,312 changed lines in 9 files — `stet-commit-f8ae1177`; judges preferred GPT 6.1 Sol medium’s patch.

### Only Haiku 5.5 low solved — 2

- 5 changed lines in 2 files — `stet-15439c21`; judges preferred Haiku 5.5 low’s patch.
- 113 changed lines in 4 files — `stet-commit-f0caded9`; judges preferred Haiku 5.5 low’s patch.

### Both solved — 9

- 4 changed lines in 1 file — `stet-43493c52`; judges preferred GPT 6.1 Sol medium’s patch.
- 6 changed lines in 1 file — `stet-3169aefa`.
- 9 changed lines in 1 file — `stet-89dfbc27`.
- 10 changed lines in 1 file — `stet-232b50ba`.
- 23 changed lines in 2 files — `stet-5276aa3f`.
- 29 changed lines in 1 file — `stet-b28f4fe7`.
- 84 changed lines in 1 file — `stet-66762b4c`; judges preferred GPT 6.1 Sol medium’s patch.
- 218 changed lines in 2 files — `stet-commit-5d766dde`.
- 941 changed lines in 7 files — `stet-commit-bbbbae09`; judges preferred GPT 6.1 Sol medium’s patch.

### Neither solved — 11

- 2 changed lines in 1 file — `stet-2450ca2d`.
- 23 changed lines in 1 file — `stet-6f84e978`.
- 149 changed lines in 2 files — `stet-commit-e928166f`; judges preferred GPT 6.1 Sol medium’s patch.
- 476 changed lines in 17 files — `stet-commit-30506f8e`.
- 481 changed lines in 3 files — `stet-commit-5e075641`.
- 552 changed lines in 11 files — `stet-commit-60a0e6c2`.
- 564 changed lines in 10 files — `stet-commit-a8716b88`.
- 763 changed lines in 8 files — `stet-commit-87814583`.
- 977 changed lines in 5 files — `stet-commit-e41c2e2d`.
- 1,236 changed lines in 16 files — `stet-commit-d2ab0c25`.
- 1,420 changed lines in 19 files — `stet-commit-2826765e`.

A task counts as solved when its tests passed or a judge found the fix equivalent to the merged change, as in the leaderboard’s Solved column. Stet score also weighs code quality, so it can order setups differently.

## All 24 compared tasks

| Task | GPT 6.1 Sol medium | Haiku 5.5 low | Solved by | Better patch |
| --- | --- | --- | --- | --- |
| `stet-89dfbc27` · 9 changed lines in 1 file | Passed tests · $0.18 · 2.8 min | Passed tests · $0.01 · 1.3 min | Both | Even |
| `stet-b28f4fe7` · 29 changed lines in 1 file | Passed tests · $0.73 · 7.1 min | Passed tests · $0.02 · 3.4 min | Both | Even |
| `stet-232b50ba` · 10 changed lines in 1 file | Passed tests · $0.24 · 3.8 min | Passed tests · $0.01 · 1.1 min | Both | Even |
| `stet-3169aefa` · 6 changed lines in 1 file | Passed tests · $0.27 · 3.6 min | Passed tests · $0.01 · 0.6 min | Both | Even |
| `stet-5276aa3f` · 23 changed lines in 2 files | Passed tests · $0.54 · 6.5 min | Passed tests · $0.02 · 3.6 min | Both | Even |
| `stet-66762b4c` · 84 changed lines in 1 file | Passed tests · $0.32 · 3.6 min | Equivalent to the merged fix · $0.02 · 2.9 min | Both | GPT 6.1 Sol medium |
| `stet-commit-bbbbae09` · 941 changed lines in 7 files | Passed tests · $1.63 · 25.4 min | Passed tests · $0.06 · 5.9 min | Both | GPT 6.1 Sol medium |
| `stet-commit-5d766dde` · 218 changed lines in 2 files | Equivalent to the merged fix · $0.77 · 9.8 min | Equivalent to the merged fix · $0.03 · 3.7 min | Both | Even |
| `stet-43493c52` · 4 changed lines in 1 file | Passed tests · $0.39 · 5.7 min | Passed tests · $0.01 · 0.8 min | Both | GPT 6.1 Sol medium |
| `stet-commit-f0caded9` · 113 changed lines in 4 files | Failed · $0.87 · 9.2 min | Equivalent to the merged fix · $0.09 · 10.5 min | Haiku 5.5 low only | Haiku 5.5 low |
| `stet-15439c21` · 5 changed lines in 2 files | Failed · $0.61 · 8.6 min | Equivalent to the merged fix · $0.01 · 10.6 min | Haiku 5.5 low only | Haiku 5.5 low |
| `stet-commit-f8ae1177` · 1,312 changed lines in 9 files | Passed tests · $1.83 · 28.7 min | Failed · $0.83 · 12.0 min | GPT 6.1 Sol medium only | GPT 6.1 Sol medium |
| `stet-2450ca2d` · 2 changed lines in 1 file | Failed · $0.26 · 4.2 min | Failed · $0.01 · 1.4 min | Neither | Even |
| `stet-commit-342d0ac0` · 640 changed lines in 8 files | Equivalent to the merged fix · $1.49 · 14.4 min | Failed · $0.12 · 32.1 min | GPT 6.1 Sol medium only | GPT 6.1 Sol medium |
| `stet-6f84e978` · 23 changed lines in 1 file | Failed · $0.50 · 5.4 min | Failed · $0.01 · 1.4 min | Neither | Even |
| `stet-commit-5e075641` · 481 changed lines in 3 files | Failed · $2.02 · 14.3 min | Failed · $0.45 · 14.9 min | Neither | Even |
| `stet-commit-60a0e6c2` · 552 changed lines in 11 files | Failed · $1.38 · 14.2 min | Failed · $0.08 · 2.9 min | Neither | Even |
| `stet-commit-2826765e` · 1,420 changed lines in 19 files | Failed · $1.60 · 13.8 min | Failed · $0.95 · 13.8 min | Neither | Even |
| `stet-commit-a8716b88` · 564 changed lines in 10 files | Failed · $1.52 · 16.6 min | Failed · $0.48 · 12.8 min | Neither | Even |
| `stet-commit-e928166f` · 149 changed lines in 2 files | Failed · $1.14 · 12.2 min | Failed · $0.05 · 3.3 min | Neither | GPT 6.1 Sol medium |
| `stet-commit-30506f8e` · 476 changed lines in 17 files | Failed · $0.81 · 20.7 min | Failed · $0.90 · 30.4 min | Neither | Even |
| `stet-commit-87814583` · 763 changed lines in 8 files | Failed · $3.24 · 17.9 min | Failed · $0.04 · 5.6 min | Neither | Even |
| `stet-commit-d2ab0c25` · 1,236 changed lines in 16 files | Failed · $2.16 · 15.3 min | Failed · $1.74 · 50.2 min | Neither | Even |
| `stet-commit-e41c2e2d` · 977 changed lines in 5 files | Failed · $1.92 · 18.9 min | Failed · $0.03 · 11.9 min | Neither | Even |

## Cost and time on each task

Each dot represents one task on a log scale. Filled dots indicate a solved task; hollow dots indicate an unsolved task. The line marks the median.

- **Cost per task:** GPT 6.1 Sol medium — mean $1.10, median $0.84. Haiku 5.5 low — mean $0.25, median $0.04. Haiku 5.5 low cost less on 23 of 24 tasks; GPT 6.1 Sol medium cost less on 1.
- **Agent time per task:** GPT 6.1 Sol medium — mean 11.8 min, median 11.0 min. Haiku 5.5 low — mean 9.9 min, median 4.7 min. Haiku 5.5 low finished faster on 17 of 24 tasks; GPT 6.1 Sol medium finished faster on 7.

## Where each setup is the better choice

These are readings on smaller task groups, so use them as leads to test on your repository.

### Small fixes

**10 tasks · 2–84 changed lines in 1–2 files**

Haiku 5.5 low solved more: 8 of 10 versus 7 of 10, at 28× lower cost per task. It was 1.9× faster.

| Metric | GPT 6.1 Sol medium | Haiku 5.5 low |
| --- | --- | --- |
| Solved | 7/10 | 8/10 |
| Cost / task | $0.40 | $0.01 |
| Agent time / task | 5.1 min | 2.7 min |

### Large, cross-cutting changes

**14 tasks · 113–1,420 changed lines in up to 19 files**

GPT 6.1 Sol medium solved 1 more: 4 of 14 versus 3 of 14. Haiku 5.5 low cost 3.8× less per task and was 1.1× faster.

| Metric | GPT 6.1 Sol medium | Haiku 5.5 low |
| --- | --- | --- |
| Solved | 4/14 | 3/14 |
| Cost / task | $1.60 | $0.42 |
| Agent time / task | 16.5 min | 15.0 min |

## Side by side

| Metric | GPT 6.1 Sol medium | Haiku 5.5 low |
| --- | --- | --- |
| Harness | `codex` | `claude-code` |
| Reasoning effort | medium | low |
| Rank | 4 of 18 | 16 of 18 |
| Stet score | 54.7; 95% range 46.0–63.4 | 44.9; 95% range 39.1–50.3 |
| Solved | 11/24 — 9 passed, 2 judged equivalent | 11/24 — 7 passed, 4 judged equivalent |
| Cost / task | $1.10 | $0.25 |
| Agent time / task | 11.8 min | 9.9 min |

## Research on these models

- [Cost per successful task ranked the coding agent that lost every head to head 1st](https://www.stet.sh/blog/cost-per-successful-task)
- [A benchmark mined from my own repo agreed with FrontierCode](https://www.stet.sh/blog/benchmark-agreed-with-frontier-code)
- [I ran GPT 6.1 Sol against GPT 6 Sol on 24 tasks from my own repo, and found that it maintained quality while cutting cost in half](https://www.stet.sh/blog/gpt-6-1-sol-vs-gpt-6-sol)

## Questions

### Which is better for coding, GPT 6.1 Sol medium or Haiku 5.5 low?

We found no clear quality difference: GPT 6.1 Sol medium won 6, Haiku 5.5 low won 2, and 16 were tied. Haiku 5.5 low led on cost and agent time: 4.4× cheaper and 1.2× faster. We measured this on 24 merged changes from `Stet-AI/stet`.

### How were GPT 6.1 Sol medium and Haiku 5.5 low tested?

Both coding agents worked on the same 24 merged changes from our private `Stet-AI/stet` repository, each in the real repository state before the change. The repository’s tests and judges graded every patch against the merged change; we used no public benchmark scores.

## See which setup wins on your repository

We replay your merged changes, score quality above pass/fail, and deliver comparison reports through your agent.

[Get started](https://www.stet.sh/private)

## About this data

- 24 tasks, each a merged change from the private `Stet-AI/stet` repository, replayed from the repository state just before the change.
- Data from November 19, 2026.
- Both setups came from the same leaderboard run of 18 setups: `bmk_d4ba310fdf50db4ccb2092e3`.

[See all 18 setups on the leaderboard](https://www.stet.sh/leaderboard) · [How Stet measures](https://www.stet.sh/methodology)

## More comparisons

- [GPT 6.1 Sol medium vs Muse Wonder Full low](https://www.stet.sh/leaderboard/compare/gpt-6-1-sol-medium-vs-muse-wonder-full-low)
- [GPT 6.1 Sol medium vs GPT 6 Sol medium](https://www.stet.sh/leaderboard/compare/gpt-6-1-sol-medium-vs-gpt-6-sol-medium)
- [GPT 6.1 Sol medium vs Opus 5 medium](https://www.stet.sh/leaderboard/compare/gpt-6-1-sol-medium-vs-opus-5-medium)
- [GPT 6.1 Sol medium vs Grok 4.7 high](https://www.stet.sh/leaderboard/compare/gpt-6-1-sol-medium-vs-grok-4-7-high)
- [GPT 6.1 Sol medium vs SWE-2 Max high](https://www.stet.sh/leaderboard/compare/gpt-6-1-sol-medium-vs-swe-2-max-high)
- [Haiku 5.5 low vs GPT 6 Luna max](https://www.stet.sh/leaderboard/compare/gpt-6-luna-max-vs-haiku-5-5-low)
- [Haiku 5.5 low vs GPT 5.6 Luna low](https://www.stet.sh/leaderboard/compare/gpt-5-6-luna-low-vs-haiku-5-5-low)
- [Haiku 5.5 low vs GPT 5.6 Luna max](https://www.stet.sh/leaderboard/compare/gpt-5-6-luna-max-vs-haiku-5-5-low)
- [Haiku 5.5 low vs GPT 6 Luna low](https://www.stet.sh/leaderboard/compare/gpt-6-luna-low-vs-haiku-5-5-low)
- [Haiku 5.5 low vs Opus 4.8 medium](https://www.stet.sh/leaderboard/compare/haiku-5-5-low-vs-opus-4-8-medium)
- [All 153 comparisons](https://www.stet.sh/leaderboard/compare)

---

[Read this page’s summary](https://www.stet.sh/agents/api/public/v1/page-tldr?url=https%3A%2F%2Fwww.stet.sh%2Fleaderboard%2Fcompare%2Fgpt-6-1-sol-medium-vs-haiku-5-5-low&trace=t_5da45423e484)

## Ask a follow-up about this page

Need a more specific answer? Ask the exact question you have. It returns a fresh, cited answer from Stet’s knowledge base. No account or API key required.

Open this URL with your own question, URL-encoded. Include your question in q. All research inputs are optional; no account or setup.

`GET https://www.stet.sh/agents/agent/faq?q={question}&trace=t_5da45423e484`

[Tool guide and examples](https://www.stet.sh/agents/agent-access) · [OpenAPI](https://www.stet.sh/agents/api/public/v1/openapi.json) · [Page TLDR form](https://www.stet.sh/agents/agent-tools/page-tldr?url=https%3A%2F%2Fwww.stet.sh%2Fleaderboard%2Fcompare%2Fgpt-6-1-sol-medium-vs-haiku-5-5-low). Opening a form does not execute a tool.