Low vs high vs xhigh/max reasoning for coding agents
Higher reasoning effort is no guarantee of better code. These 3 same-model effort comparisons show what moving up in effort bought in recorded quality, cost and agent time.
What this comparison is
- 1 real repositoryStet-AI/stet, private; not public benchmark scores
- 24 merged changeseach run end to end by the coding agent from the repo as it was before the change
- Graded against the merged changeby the repo’s own tests and by judges comparing each patch with it
- Quality, cost and timea smaller sample than public benchmarks, but real agent work on a real codebase
Recorded setup pairings
Each row is a separate recorded comparison. Scores are not pooled or averaged across effort settings. W/D/L means wins, draws and losses; undecided judgments remain undecided.
Stet score: A setup’s average chance of beating another setup on a task, from 0 to 100; 50 is even with the field.
Known settings run from lower to higher effort: minimal, low, medium, high, xhigh, max; other settings follow as recorded. Effort names are provider settings, not comparable compute budgets across models. Harnesses and other recorded setup details can differ, so these observations do not isolate effort as a cause.
| Setup pairing | Recorded quality call and record | Quality evidence | Cost per task | Agent time per task |
|---|---|---|---|---|
| GPT 5.6 Luna low vs GPT 5.6 Luna maxGPT 5.6 Luna low · codex · low reasoningGPT 5.6 Luna max · max reasoning24/24 tasks observed by both setups. | No clear quality difference: GPT 5.6 Luna max won 1, GPT 5.6 Luna low won 0, 22 tied, 1 undecided; GPT 5.6 Luna low leads on cost and agent time: 17.6× cheaper and 4.3× faster.Head to head, GPT 5.6 Luna max won 1, tied 22 and lost 0 against GPT 5.6 Luna low, with 1 undecided. | No clear difference 0–1 head to head, 22 tiedGPT 5.6 Luna low: 44.6 (95% range 38.3–50.3)GPT 5.6 Luna max: 49.7 (95% range 44.0–55.6) | GPT 5.6 Luna low: $0.05 GPT 5.6 Luna max: $0.90 | GPT 5.6 Luna low: 6.7 min GPT 5.6 Luna max: 28.5 min |
| Haiku 5.5 low vs Haiku 5.5 maxHaiku 5.5 low · low reasoningHaiku 5.5 max · max reasoning23/24 tasks observed by both setups. | No clear quality difference: Haiku 5.5 max won 3, Haiku 5.5 low won 2, 17 tied, 2 undecided; Haiku 5.5 low leads on cost and agent time: 12.1× cheaper and 5.9× faster.Head to head, Haiku 5.5 max won 3, tied 17 and lost 2 against Haiku 5.5 low, with 2 undecided. | No clear difference 2–3 head to head, 17 tiedHaiku 5.5 low: 44.7 (95% range 39.5–49.1)Haiku 5.5 max: 48.2 (95% range 38.8–58.1) | Haiku 5.5 low: $0.06 Haiku 5.5 max: $0.78 | Haiku 5.5 low: 9.9 min Haiku 5.5 max: 58.4 min |
| GPT 6 Luna low vs GPT 6 Luna maxGPT 6 Luna low · low reasoningGPT 6 Luna max · max reasoning24/24 tasks observed by both setups. | GPT 6 Luna max leads on quality, 6–2 head to head with 15 tied and 1 undecided (directional at this sample size); GPT 6 Luna low leads on cost and agent time: 6.9× cheaper and 8.7× faster.Head to head, GPT 6 Luna max won 6, tied 15 and lost 2 against GPT 6 Luna low, with 1 undecided. | Directional lead at this sample size 6–2 head to head, 15 tiedGPT 6 Luna low: 37.8 (95% range 27.8–47.7)GPT 6 Luna max: 46.0 (95% range 40.4–51.6) | GPT 6 Luna low: $0.05 GPT 6 Luna max: $0.37 | GPT 6 Luna low: 6.9 min GPT 6 Luna max: 59.7 min |
Questions about these results
- Is high, xhigh or max reasoning better than low or medium for coding?
- More effort does not guarantee better code. Read each same-model pair's quality call alongside cost and agent time; a close or mixed result does not establish a quality gain.
- Does this board test high vs xhigh or Codex medium vs high?
- Only the effort pairs listed here have recorded same-model results on this board. An absent setting, including high vs xhigh or Codex medium vs high, has no measured result here; max and xhigh are not interchangeable.
- What does moving up in reasoning effort cost?
- Each row shows both setups' recorded cost and agent time per task, including missing readings and token-based estimates. Compare the values within a pair rather than assuming a fixed effort multiplier.