# GPT-5.5 vs GPT-5.4 vs Opus 4.7 on 56 real coding tasks from 2 open source repos

Stet publishes a public search API over its indexed pages; the guide is at https://www.stet.sh/agents/agent-access.

Source: https://www.stet.sh/blog/gpt-55-vs-opus-47
Source observed: 2026-11-16T18:25:37.550Z
Last checked: 2026-11-16T18:25:37.55+00:00
Indexed: 2026-11-16T18:45:41.437+00:00

---

# GPT-5.5 vs GPT-5.4 vs Opus 4.7 on 56 real coding tasks from 2 open source repos

**May 1, 2026 · 14 min read**  
By [Ben, Founder, Stet](https://www.stet.sh/about)

Opus 4.7 writes smaller patches. GPT-5.5 writes patches that more often survive review. We ran GPT-5.5, GPT-5.4, and Opus 4.7 on **56 real coding tasks** from two open-source repositories: **27 tasks from Zod** and **29 from `graphql-go-tools`**. Each model ran once per task in its native agent harness at default settings: Anthropic models in Claude Code and OpenAI models in OpenAI Codex CLI.

The result is a workflow tradeoff rather than one model winning every category. Across these runs, GPT-5.5 was our best shipping default: it most often produced patches that passed tests, matched the intended human change, and survived code review. Opus 4.7 produced much smaller patches and had lower footprint risk. On Zod, GPT-5.5 and Opus tied on test passes, but GPT-5.5 led reviewer judgment. On `graphql-go-tools`, GPT-5.5 led more clearly because Opus’s small-patch strategy missed integration surfaces.

> **GPT-5.5 ships more often. Opus 4.7 ships smaller.** The better choice depends on whether your bottleneck is review or footprint.

We used [Stet](https://www.stet.sh/) to grade more than test pass/fail: behavioral equivalence to the human patch, code-review acceptability, footprint risk, and craft/discipline rubrics. This is a concrete comparison of three models on two real codebases, not a statement about every coding task.

## Model setups

| Model | Harness | Reasoning level |
| --- | --- | --- |
| Opus 4.7 | Claude Code | high |
| GPT-5.4 | Codex CLI | high |
| GPT-5.5 | Codex CLI | high |

## The short version

### Overall results, 56 tasks

| Measure | Opus 4.7 | GPT-5.4 | GPT-5.5 |
| --- | ---: | ---: | ---: |
| Tests pass | 33/56 | 31/56 | 38/56 |
| Equivalent to human patch | 19/56 | 35/56 | 40/56 |
| Clean pass: tests + review | 10/56 | 11/56 | 28/56 |
| Footprint risk, lower is better | 0.20 | 0.34 | 0.32 |
| Mean time / task, lower is better | 11m18s | 8m24s | 6m56s |
| Cost / task, lower is better | $3.43 | $2.39 | $2.86 |

GPT-5.5 passed the most tests, matched the human patch most often, and produced 28 clean passes. Opus cleared review 10 times and had the lowest footprint risk. GPT-5.5 used fewer input and output tokens and less mean agent time than either competitor. GPT-5.4 was cheapest per task because its pricing was lower, but its cost advantage did not offset the clean-pass gap in these runs.

### Results by repository

#### Zod — 27 tasks

Zod is a TypeScript schema library with parallel Node and Deno trees.

| Measure | Opus 4.7 | GPT-5.4 | GPT-5.5 |
| --- | ---: | ---: | ---: |
| Tests pass | 12/27 | 9/27 | 12/27 |
| Equivalent | 11/27 | 18/27 | 18/27 |
| Review pass | 6/27 | 10/27 | 14/27 |
| Clean pass | 5/27 | 5/27 | 10/27 |

GPT-5.5 and Opus tied on tests. GPT-5.5 led reviewer judgment; Opus produced smaller diffs.

#### `graphql-go-tools` — 29 tasks

`graphql-go-tools` is a Go federation engine spanning planner, datasource, and runtime surfaces.

| Measure | Opus 4.7 | GPT-5.4 | GPT-5.5 |
| --- | ---: | ---: | ---: |
| Tests pass | 21/29 | 22/29 | 26/29 |
| Equivalent | 8/29 | 17/29 | 22/29 |
| Review pass | 5/29 | 6/29 | 19/29 |
| Clean pass | 5/29 | 6/29 | 18/29 |

GPT-5.5 led outright: it passed more tests, produced many more clean passes, and more often matched the human patch. Opus still produced the smallest patches, but missed too much integration work.

## Full scorecard

The quality rows prevent us from treating test passes as the entire result. Code review includes correctness, introduced-bug risk, and maintainability where available. The separate custom grader uses eight additive rubrics: five craft dimensions and three discipline dimensions. Scores run from **0 to 4**, with higher better.

| Review or rubric measure | Opus 4.7 | GPT-5.4 | GPT-5.5 |
| --- | ---: | ---: | ---: |
| Code-review pass | 11/56 | 16/56 | 33/56 |
| Correctness + bug safety, mean | 2.33 | 2.59 | 3.08 |
| Correctness | 2.11 | 2.60 | 3.16 |
| Introduced-bug safety | 2.55 | 2.56 | 3.04 |
| Maintainability — GraphQL only | 2.07 | 2.55 | 3.03 |
| Rubric average | 2.33 | 2.40 | 2.62 |
| Craft — mean of 5 | 2.41 | 2.54 | 2.78 |
| Clarity | 2.56 | 2.75 | 2.91 |
| Coherence | 1.95 | 2.18 | 2.51 |
| Robustness | 1.92 | 2.43 | 2.69 |
| Discipline — mean of 3 | 2.20 | 2.16 | 2.36 |
| Scope discipline | 2.39 | 2.18 | 2.45 |
| Diff minimality | 2.42 | 2.28 | 2.46 |
| Input tokens, full run; lower better | 239.1M | 222.3M | 201.8M |
| Output tokens, full run; lower better | 1.29M | 1.09M | 0.72M |

GPT-5.5 scored higher on correctness, introduced-bug safety, GraphQL maintainability, coherence, robustness, scope discipline, and task-relative diff minimality. Opus won the mechanical footprint row: smaller diffs, but not consistently more disciplined diffs.

## How the benchmark works

Each task came from a real merged commit. Each model received a frozen repository snapshot, a prompt describing the change, and one attempt to produce a patch in its native shipped agent harness with no Stet-side scaffolding:

- Opus 4.7 ran in Claude Code using `claude -p`.
- GPT-5.5 and GPT-5.4 ran in OpenAI Codex CLI using `codex exec`.
- All ran at default settings.

We applied each patch and ran the task’s tests in an isolated container. We then graded:

- **Tests:** whether executable acceptance tests passed.
- **Equivalence:** whether the candidate patch accomplished the same behavioral change as the original human patch.
- **Code review:** whether a reviewer would accept it, considering correctness, introduced-bug risk, maintainability, and edge cases.
- **Footprint risk:** the review and regression surface created by the patch.
- **Craft/discipline:** clarity, simplicity, coherence, intentionality, robustness, instruction adherence, scope discipline, and diff minimality.

Every model ran once per task with a single seed. GPT-5.4 judged equivalence and rubrics using identical rubric versions across all three arms. Each patch was scored independently without exposing the arm label or producing model to the judge. There was no dual-rater calibration, so absolute scores are directional; cross-arm differences are the useful comparison.

## GPT-5.5 widened the gap

Across these two repositories, GPT-5.5 outperformed GPT-5.4 for coding-agent work on tests, human-equivalent patches, reviewer-accepted patches, and clean passes. GPT-5.5 was more shippable; GPT-5.4 was cheaper per task. GPT-5.4 remained a possible cost-control choice, but its quality gap was large enough that we would not use it as the default shipping model for these repositories.

## Tests are signal, not the finish line

A **clean pass** requires both passing tests and acceptance by the code-review grader.

- On Zod, Opus and GPT-5.5 both passed **12/27** tests, but GPT-5.5 produced **10** clean passes versus Opus’s **5**.
- On `graphql-go-tools`, GPT-5.5 passed **26/29** tests and produced **18** clean passes. Opus passed **21/29** and produced **5** clean passes.

One GraphQL example was PR **#1001**, which changed an HTTP datasource `OnFinished` hook so consumers could inspect request and response metadata. All three models passed tests and were judged equivalent, but only GPT-5.5 cleared code review. The other patches received warnings around API shape, raw HTTP object exposure, and robustness at the hook boundary.

## What the reviewer saw

### Zod async codecs and defaults

The task required async codec pipelines, prevention of defaults becoming `undefined`, and generated stub package manifests. All three models failed tests.

Opus changed eight files but missed central semantics: defaults could still allow `undefined`, core codec definitions remained synchronous, generated stubs were not published, and `prefault()` was tightened even though the request concerned `.default()`.

GPT-5.4 produced an 11-file patch and was judged behaviorally equivalent, but it also over-tightened adjacent API by restricting `prefault`.

GPT-5.5 failed tests too, but was judged equivalent and scored better on correctness and introduced-bug risk. It covered codec/default tests, version metadata, stub-manifest scripts, and the relevant `packages/zod/src/v4/*/schemas.ts` surfaces more cleanly.

### GraphQL Apollo-compatible validation

PR **#1169** aligned field-selection validation errors with GraphQL specification and Apollo Router conventions. All three models produced patches and passed tests, but only GPT-5.5 cleared equivalence and review.

Opus touched 11 files but missed enum and wrapped-scalar leaf validation, used some field locations instead of selection-set locations, left an inline-fragment message non-spec-compliant, and did not apply validation status uniformly.

GPT-5.4 touched 12 files and passed tests, but broadened behavior in the wrong places: unconditional validation metadata, incomplete enum/wrapped-scalar handling, broad request-error conversion, and stale compatibility API.

GPT-5.5 touched 10 files, six non-test, and added more targeted behavior: aligned field-selection messages, requested locations, and centralized Apollo validation metadata.

GPT-5.5’s correctness score was **3.16**, versus **2.60** for GPT-5.4 and **2.11** for Opus. Its discipline lead was narrower: **2.36**, versus **2.20** for Opus and **2.16** for GPT-5.4. Opus won raw footprint; GPT-5.5 narrowly won task-relative discipline.

## What Opus is doing

Opus 4.7 is cautious: it writes smaller patches, touches fewer files, and has the lowest footprint risk in both repositories. That is attractive for contained Zod tasks where a precise source edit and small test update are sufficient.

Its recurring failure mode is implementing the headline behavior while stopping before companion work is complete. Zod’s parallel Node and Deno trees made this visible: a CIDR validation change passed tests after Opus touched four files, but GPT-5.5 touched eleven to update the parallel distribution surface. The human patch included that companion work, so the judge marked Opus non-equivalent.

On `graphql-go-tools`, the same behavior was more consequential because planner, datasource, hook, validation, and runtime paths must align. PR **#1155** covered repeated scalar fields in a gRPC datasource, request building, response marshaling, null and invalid responses, error status information, disabled datasources, and dynamically created clients. Opus produced no patch; GPT-5.5 passed tests, matched the human patch, and cleared review.

Opus’s small patches can be disciplined on local tasks and under-implementation on integration-heavy tasks.

## What changed from GPT-5.4 to GPT-5.5

GPT-5.5 was not simply GPT-5.4 with higher pass rates; its failure modes shifted.

- On Zod, GPT-5.4 had **18 equivalence-positive judgments**, matching GPT-5.5, but only **9 test passes**. GPT-5.5 closed that execution gap.
- For a schema-to-TypeScript generator, Opus and GPT-5.5 built the requested recursive visitor; GPT-5.4 produced repository-instruction files instead of the feature.
- For a recursive parser fix, GPT-5.4 added an `inProgress` sentinel and reset logic. GPT-5.5 kept count-and-cache-error behavior with fewer moving parts and passing tests.
- For CIDR validation, both GPT models used similar address and prefix validation, but GPT-5.5 updated the Deno mirrors and GPT-5.4 did not.
- On `graphql-go-tools` PR **#1232**, which required deduplicating identical single fetches and rewriting dependencies pointing to removed duplicates, GPT-5.5 was the only model to pass tests, match human behavior, and clear review.

Our pattern is that GPT-5.5 does more integration work that turns a local fix into a shippable repository change.

## The cost of doing more

On `graphql-go-tools`, average patch size was about **33 KB for GPT-5.5**, **27 KB for GPT-5.4**, and **19 KB for Opus**. Footprint scores were **0.34**, **0.32**, and **0.19**, respectively.

Bigger patches are harder to review, easier to conflict, and more likely to touch sensitive paths. Opus therefore retains a real advantage when auditability and compact diffs dominate. But a patch that misses required surfaces is not more minimal than a larger patch that completes the task.

### Operational comparison by repository

#### Zod — 27 tasks

| Measure | Opus 4.7 | GPT-5.5 |
| --- | ---: | ---: |
| Input tokens | 53.0M | 50.4M |
| Output tokens | 359K | 290K |
| Mean time / task | 4m25s | 5m10s |
| Cost / task | $1.69 | $1.73 |

#### `graphql-go-tools` — 29 tasks

| Measure | Opus 4.7 | GPT-5.5 |
| --- | ---: | ---: |
| Input tokens | 186.1M | 151.4M |
| Output tokens | 934K | 431K |
| Mean time / task | 17m43s | 8m36s |
| Explicit planning calls / task | 3.2 | 0.0 |
| Patch calls / task | 10.2 | 9.9 |

On Zod, Opus was faster and slightly cheaper. On GraphQL, Opus used more tokens and time while converging on smaller, less complete patches. GPT-5.5 made more repository-inspection shell and search calls; Opus spent more of its budget on planning and patch-rewrite churn.

## Model profiles

- **Opus 4.7 — under-reach:** conservative, precise, and low-footprint. Strong when the task is local and narrow; weaker when companion surfaces are required but incompletely tested.
- **GPT-5.4 — right shape, wrong execution:** often finds the intended approach, but is more prone to stale mirrors, extra bookkeeping, unearned refactors, and patches the judge prefers more than the test suite does.
- **GPT-5.5 — broader, bigger footprint:** more complete across integration surfaces and more likely to pass review, with a larger patch-footprint risk when wrong.

## Why this matters

The practical question is not which model is universally best. It is which model produces patches we trust for our repository, harness, task mix, and review standards.

Zod made GPT-5.5 versus Opus look like a tradeoff: equal test passes, stronger GPT-5.5 reviewer alignment, and smaller Opus patches. `graphql-go-tools` made the tradeoff less symmetrical: GPT-5.5 was more shippable, while Opus’s small-patch advantage came with missed integration work.

That is why we build Stet around real repository tasks. Each codebase has its own mirror trees, code-generation surfaces, test blind spots, hook conventions, planner invariants, review standards, `AGENTS.md`, skills, model settings, and harness settings.

## Caveats

- 56 scored tasks is still a small sample; one task can move a repository-level rate by several points.
- Each model ran once per task, so some close calls could change on rerun.
- GPT-5.4 judged equivalence and rubrics, which can introduce family bias.
- Results are harness-conditional. Claude Code and Codex CLI provide different prompts, planning loops, and tool surfaces. Running Opus in Codex or GPT-5.5 in Claude Code would change the comparison.

## Takeaway

- GPT-5.5 was our best default shipping model across these two repositories.
- Opus 4.7 remained the low-footprint choice and may be preferable when narrow diffs matter most.
- GPT-5.4 was cheaper per task, but not enough cheaper to overcome its clean-pass gap here.
- Tests alone hid important differences in review quality and integration completeness.
- The ranking changed by repository, which is why measuring your own work matters.

Your repository is different. That is what [Stet](https://www.stet.sh/) is for.

## FAQ

### GPT-5.4 vs GPT-5.5: which was better in this benchmark?

GPT-5.5 passed **38/56** tests, matched the human patch on **40/56** tasks, and produced **28** clean passes. GPT-5.4 passed **31/56**, matched the human patch on **35/56**, and produced **11** clean passes.

### GPT-5.5 vs Opus 4.7: which should we use?

GPT-5.5 was the better default when review quality and complete integration mattered. Opus 4.7 wrote smaller patches and had lower footprint risk, so it may be preferable for narrow changes where compact diffs matter most.

### What is a clean pass?

A clean pass means the candidate patch passed the task tests and was accepted by the code-review grader. It is stricter than test-pass rate because it asks whether the patch is maintainable.

[Methodology →](https://www.stet.sh/methodology)

---

[Read this page’s summary](https://www.stet.sh/agents/api/public/v1/page-tldr?url=https%3A%2F%2Fwww.stet.sh%2Fblog%2Fgpt-55-vs-opus-47&trace=t_5da45423e484)

## Ask a follow-up about this page

Need a more specific answer? Ask the exact question you have. It returns a fresh, cited answer from Stet’s knowledge base. No account or API key required.

Open this URL with your own question, URL-encoded. Include your question in q. All research inputs are optional; no account or setup.

`GET https://www.stet.sh/agents/agent/faq?q={question}&trace=t_5da45423e484`

[Tool guide and examples](https://www.stet.sh/agents/agent-access) · [OpenAPI](https://www.stet.sh/agents/api/public/v1/openapi.json) · [Page TLDR form](https://www.stet.sh/agents/agent-tools/page-tldr?url=https%3A%2F%2Fwww.stet.sh%2Fblog%2Fgpt-55-vs-opus-47). Opening a form does not execute a tool.