# Opus 4.7 reasoning: medium led on 29 GraphQL tasks

Stet publishes a public search API over its indexed pages; the guide is at https://www.stet.sh/agents/agent-access.

Source: https://www.stet.sh/blog/opus-47-graphql-reasoning-curve
Source observed: 2026-11-08T19:51:42.062Z
Last checked: 2026-11-08T19:51:42.062+00:00
Indexed: 2026-11-08T20:16:15.542+00:00

---

# Opus 4.7 Low Vs Medium Vs High Vs Xhigh Vs Max: the Reasoning Curve on 29 Real Tasks from an Open Source Repo

**May 12, 2026 · 21 min read** · [Ben, Founder, Stet](https://www.stet.sh/about)

We ran Opus 4.7 in Claude Code at low, medium, high, xhigh, and max reasoning effort on the same 29 real tasks from the open-source GraphQL-go-tools repository in Go.

On this slice, performance did not improve linearly with more reasoning. Medium led the main quality measures; low was cheaper and faster but less correct; high, xhigh, and max cost more without beating medium on the primary metrics.

## Summary data

| Metric | Low | Medium | High | Xhigh | Max |
| --- | ---: | ---: | ---: | ---: | ---: |
| All-task pass | 23/29 | 28/29 | 26/29 | 25/29 | 27/29 |
| Equivalent | 10/29 | 14/29 | 12/29 | 11/29 | 13/29 |
| Code-review pass | 5/29 | 10/29 | 7/29 | 4/29 | 8/29 |
| Code-review rubric mean | 2.426 | 2.716 | 2.509 | 2.482 | 2.431 |
| Footprint risk mean | 0.155 | 0.189 | 0.206 | 0.238 | 0.227 |
| All custom graders | 2.598 | 2.759 | 2.670 | 2.669 | 2.690 |
| Mean cost/task | $2.50 | $3.15 | $5.01 | $6.51 | $8.84 |
| Mean duration/task | 383.8s | 450.7s | 716.4s | 803.8s | 996.9s |
| Equivalent passes per dollar | 0.138 | 0.153 | 0.083 | 0.058 | 0.051 |

“Equivalent” means the patch matched the intent of the merged human PR. “Code-review pass” means an AI reviewer judged it acceptable. Craft/discipline is a 0–4 maintainability/style rubric, and footprint risk measures extra code touched relative to the human patch.

## Why we ran this

A previous 28-task Zod run showed a non-monotonic pattern: all four tested settings passed 12/28 tests, while equivalence and review moved unevenly. We reran the question on a more discriminating 29-task GraphQL-go-tools slice.

This is intended to inform reasoning-setting choices for real repository work, not to serve as a universal benchmark. The practical question is whether an agent makes the kind of change a human merged and whether the resulting patch is maintainable.

The GPT-5.5 Codex GraphQL run provides a contrast: on that separate run, semantic and review quality generally increased with reasoning effort. [Read that comparison](https://www.stet.sh/blog/gpt-55-codex-graphql-reasoning-curve).

## Experimental setup

Each task came from a real merged PR or commit. We gave the model a frozen repository snapshot, a prompt describing the change, and one attempt to produce a patch in a Docker container. We applied the patch and ran tests in an isolated container.

We graded beyond pass/fail:

- **Equivalence:** whether the candidate patch accomplished the same behavioral change as the human patch.
- **Code review:** whether a reviewer would accept it for correctness, introduced-bug risk, maintainability, and edge cases.
- **Footprint risk:** additional code touched compared with the human patch.
- **Craft and discipline:** clarity, simplicity, coherence, intentionality, robustness, instruction adherence, scope discipline, and diff minimality.

Each model ran once per task with a single seed. The LLM-as-judge was GPT-5.4, blinded to the model and effort that produced each patch. We manually inspected representative examples. There was no human calibration pass, so the direction of changes is more informative than any single absolute score.

The harness and dataset details were:

- Model: Opus 4.7.
- Harness: Claude Code 2.1.126–2.1.138, varying by run date; the latest npm-installed version was used for each run.
- Dataset: 29 real GraphQL-go-tools tasks. Running this used most of the weekly 20× quota.
- Main metrics: test pass, semantic equivalence, code-review pass, footprint risk, craft/discipline graders, cost, and runtime.

We also use these evaluations as an “autoresearch” loop: an agent can propose an `AGENTS.md` change, run Stet on historical tasks, inspect where it improved or regressed, and iterate.

## Low: cheaper, shallower, and incomplete

| Metric | Low | Medium | Difference |
| --- | ---: | ---: | ---: |
| All-task pass | 23/29, 79.3% | 28/29, 96.6% | −17.2pp |
| Equivalent | 10/29, 34.5% | 14/29, 48.3% | −13.8pp |
| Code-review pass | 5/29, 17.2% | 10/29, 34.5% | −17.2pp |
| Footprint risk mean | 0.155 | 0.189 | −0.034 |
| Craft/discipline average | 2.598 | 2.759 | −0.161 |
| Mean cost/task | $2.50 | $3.15 | −$0.65, 0.79× |
| Mean duration | 383.8s | 450.7s | −66.8s |

Low was faster, cheaper, and lower-footprint, but it missed important pieces of tasks. Medium cost about 26% more ($2.50 → $3.15) and performed better across the reported quality measures.

For PR #1230, which fixed GraphQL federation query-planner bugs and added an empty-selection-set guard, low implemented behavior in `graphql_datasource.go` instead of registering a planner-scoped validation rule. Tests failed and the patch was non-equivalent. Medium used a dedicated rule wired into the planner’s `printKitPool` and matched the human PR’s two-pass shape.

## Medium: balance of restraint and correctness

| Metric | Medium |
| --- | ---: |
| All-task pass | 28/29, 96.6% |
| Equivalent | 14/29, 48.3% |
| Code-review pass | 10/29, 34.5% |
| Footprint risk mean | 0.189 |
| Craft/discipline average | 2.759 |
| Mean cost/task | $3.15 |
| Mean duration | 450.7s |

Medium did enough repository modeling without drifting into oversized patches or rationalized no-ops. It led test pass, equivalence, code review, and craft/discipline on this slice.

PR #1260 exposed the difference. The task required making an existing `SkipLoader` early return reachable for plan-only requests and surfacing trigger metadata. Low stopped without a patch after becoming confused by partial scaffolding. High and xhigh searched prior commit hashes (`34cc4fa8` and `69485dfe`) and concluded incorrectly that no code changes were needed. Medium read the control flow, found the branch after a `Trigger.Source == nil` guard, and made the minimum hoist-and-extract fix; max made the same fix plus a regression test.

## High: the limits of more thinking

| Metric | Medium | High | Difference |
| --- | ---: | ---: | ---: |
| All-task pass | 28/29, 96.6% | 26/29, 89.7% | −6.9pp |
| Equivalent | 14/29, 48.3% | 12/29, 41.4% | −6.9pp |
| Code-review pass | 10/29, 34.5% | 7/29, 24.1% | −10.3pp |
| Footprint risk mean | 0.189 | 0.206 | +0.017 |
| Craft/discipline average | 2.759 | 2.670 | −0.089 |
| Mean cost/task | $3.15 | $5.01 | +$1.86, 1.59× |
| Mean duration | 450.7s | 716.4s | +265.7s |

High spent more time and money while producing lower pass, equivalence, review, and aggregate-quality results. In PR #1293, high, xhigh, and max skipped a one-character `go.work` toolchain fix and Makefile changes; the resulting toolchain resolution failed before Go code ran. Medium completed the full 18-file change, including the load-bearing build fixes.

## Xhigh: larger surface, worse results

| Metric | Medium | Xhigh | Difference |
| --- | ---: | ---: | ---: |
| All-task pass | 28/29, 96.6% | 25/29, 86.2% | −10.3pp |
| Equivalent | 14/29, 48.3% | 11/29, 37.9% | −10.3pp |
| Code-review pass | 10/29, 34.5% | 4/29, 13.8% | −20.7pp |
| Footprint risk mean | 0.189 | 0.238 | +0.049 |
| Craft/discipline average | 2.759 | 2.669 | −0.090 |
| Mean cost/task | $3.15 | $6.51 | +$3.36, 2.07× |
| Mean duration | 450.7s | 803.8s | +353.2s |

Xhigh was Claude Code’s default for Opus 4.7 and Anthropic’s stated best coding option, but on this slice it cost $6.51 per task, touched the most files, and added 7,764 lines, 47.5% of them in test/fixture surface. It did not run more tests or use more tools than medium.

For PR #859, medium changed 2 files and added 85 lines for the requested map-backed indexes. Xhigh changed 5 files and added 263 lines, including an unrequested 170-line federation metadata cache. Tests passed, but footprint risk worsened and the broader cache introduced stale-index or semantic-drift concerns.

## Max: busier, but not better than medium

| Metric | Medium | Max | Difference |
| --- | ---: | ---: | ---: |
| All-task pass | 28/29, 96.6% | 27/29, 93.1% | −3.4pp |
| Equivalent | 14/29, 48.3% | 13/29, 44.8% | −3.4pp |
| Code-review pass | 10/29, 34.5% | 8/29, 27.6% | −6.9pp |
| Footprint risk mean | 0.189 | 0.227 | +0.038 |
| Craft/discipline average | 2.759 | 2.690 | −0.069 |
| Mean cost/task | $3.15 | $8.84 | +$5.70, 2.81× |
| Mean duration | 450.7s | 996.9s | +546.2s |

Max ran 294 test commands versus medium’s 132, made 1,153 shell calls versus 582, and produced 3,719 assistant turns versus 2,042. It added 8,102 lines versus medium’s 6,700. Even so, it trailed medium on equivalence, review, review-rubric mean, and aggregate craft/discipline; it produced 0.051 equivalent passes per dollar versus medium’s 0.153.

PR #1076 was the clearest max-over-medium example. It handled a concurrency-heavy subscription rewrite and added safety guards, while medium left a shared event loop that caused a deterministic test failure. This was one task out of 29, not a monotonic improvement across the slice.

## Craft and discipline

| Metric | Low | Medium | High | Xhigh | Max |
| --- | ---: | ---: | ---: | ---: | ---: |
| Craft average | 2.572 | 2.788 | 2.691 | 2.702 | 2.724 |
| Discipline average | 2.624 | 2.729 | 2.649 | 2.635 | 2.655 |
| All custom graders | 2.598 | 2.759 | 2.670 | 2.669 | 2.690 |
| Simplicity | 2.745 | 3.034 | 2.886 | 2.910 | 2.859 |
| Coherence | 2.504 | 2.552 | 2.561 | 2.600 | 2.576 |
| Intentionality | 3.114 | 3.300 | 3.303 | 3.366 | 3.362 |
| Robustness | 1.926 | 2.266 | 2.014 | 1.932 | 2.100 |
| Clarity | 2.811 | 2.797 | 2.764 | 2.796 | 2.779 |
| Instruction adherence | 1.990 | 2.338 | 2.169 | 2.200 | 2.266 |
| Scope discipline | 2.907 | 2.934 | 2.776 | 2.697 | 2.766 |
| Diff minimality | 2.790 | 2.848 | 2.886 | 2.848 | 2.810 |

Medium led on simplicity, robustness, instruction adherence, and scope discipline. Higher settings scored better on intentionality and coherence in places, but the extra deliberation did not reliably produce patches that were easier to maintain.

## Cost and runtime

| Reasoning effort | Mean cost/task | Median cost/task | Mean duration | Median duration |
| --- | ---: | ---: | ---: | ---: |
| Low | $2.50 | $2.00 | 383.8s | 316.6s |
| Medium | $3.15 | $2.72 | 450.7s | 404.2s |
| High | $5.01 | $5.05 | 716.4s | 724.4s |
| Xhigh | $6.51 | $6.48 | 803.8s | 770.9s |
| Max | $8.84 | $8.59 | 996.9s | 991.4s |

Equivalent passes per dollar were 0.138 for low, 0.153 for medium, 0.083 for high, 0.058 for xhigh, and 0.051 for max. Medium produced about three times as many equivalent passes per dollar as max.

## Why this may happen

One hypothesis is adaptive thinking. [Anthropic’s adaptive-thinking documentation](https://platform.claude.com/docs/en/build-with-claude/adaptive-thinking) says Opus 4.7 dynamically determines when and how much extended thinking to use; the effort setting biases that policy rather than simply adding intelligence. [Claude Code’s model configuration documentation](https://code.claude.com/docs/en/model-config) warns that max may show diminishing returns and be prone to overthinking. [Anthropic’s inverse-scaling research](https://alignment.anthropic.com/2025/inverse-scaling/) describes cases where extended reasoning can worsen outputs, although that research is not coding-specific.

## GPT-5.5 contrast

On a separate 26-task GPT-5.5 GraphQL run, quality above tests rose much more clearly with reasoning effort:

| Metric | Low | Medium | High | Xhigh |
| --- | ---: | ---: | ---: | ---: |
| Task count | 26 | 26 | 26 | 26 |
| Tests pass | 21/26, 80.8% | 21/26, 80.8% | 25/26, 96.2% | 24/26, 92.3% |
| Equivalent | 4/26, 15.4% | 11/26, 42.3% | 18/26, 69.2% | 23/26, 88.5% |
| Code-review pass | 3/26, 11.5% | 5/26, 19.2% | 10/26, 38.5% | 18/26, 69.2% |
| Craft/discipline average | 2.311 | 2.604 | 2.736 | 3.071 |
| Cost/task | $2.65 | $3.13 | $4.49 | $9.77 |

[Read the GPT-5.5 GraphQL reasoning-curve post](https://www.stet.sh/blog/gpt-55-codex-graphql-reasoning-curve).

## Limitations

- Single seed per task.
- 29 matched GraphQL-go-tools tasks, with the original 28-task Zod run as context.
- GPT-5.4 was the blinded LLM-as-judge.
- No grader calibration on this task set.
- No-patch rows reduced publishable denominators for low, high, and xhigh; built-in equivalence and review coverage was partial on those rows.
- Max was repaired for targeted no-patch and infrastructure issues, but remains an inspect result rather than a rollout result because its primary metrics were mixed and below medium.

## Conclusion

On this slice, medium was the practical default: it combined the highest test pass, equivalence, review pass, and aggregate craft/discipline with lower cost and duration than high, xhigh, and max. Low remains cheaper and faster, while xhigh or max may be worth testing selectively for exploratory or cross-cutting work. We recommend measuring your own repository and harness before changing a default.

We build Stet, a local evaluation tool that can test coding-agent setup changes—such as improving `AGENTS.md`—against historical repository tasks. [Get started](https://www.stet.sh/private)

## FAQ

### Does higher Opus 4.7 reasoning effort improve coding-agent quality?

On this 29-task GraphQL-go-tools slice, no. Medium led test pass, equivalence, code-review pass, and aggregate craft/discipline. Higher settings cost more without improving those primary measures.

### Why did Opus 4.7 peak at medium while GPT-5.5 improved with more reasoning?

A possible explanation is adaptive thinking: Opus 4.7 already selects a reasoning budget per task, so forcing higher effort may bias an adaptive policy rather than add useful intelligence.

### Is this a decision-grade comparison?

It is an inspect-grade five-arm curve on 29 matched tasks, with one seed per task. Max was repaired for targeted no-patch and infrastructure issues, but the overall result remains an inspect result because its metrics were mixed and below medium.

---

[Read this page’s summary](https://www.stet.sh/agents/api/public/v1/page-tldr?url=https%3A%2F%2Fwww.stet.sh%2Fblog%2Fopus-47-graphql-reasoning-curve&trace=t_5da45423e484)

## Ask a follow-up about this page

Need a more specific answer? Ask the exact question you have. It returns a fresh, cited answer from Stet’s knowledge base. No account or API key required.

Open this URL with your own question, URL-encoded. Include your question in q. All research inputs are optional; no account or setup.

`GET https://www.stet.sh/agents/agent/faq?q={question}&trace=t_5da45423e484`

[Tool guide and examples](https://www.stet.sh/agents/agent-access) · [OpenAPI](https://www.stet.sh/agents/api/public/v1/openapi.json) · [Page TLDR form](https://www.stet.sh/agents/agent-tools/page-tldr?url=https%3A%2F%2Fwww.stet.sh%2Fblog%2Fopus-47-graphql-reasoning-curve). Opening a form does not execute a tool.