# GPT-5.5 High regression check on 21 GraphQL-go-tools tasks

Stet publishes a public search API over its indexed pages; the guide is at https://www.stet.sh/agents/agent-access.

Source: https://www.stet.sh/blog/gpt-55-high-regression-check-graphql-go-tools
Source observed: 2026-11-15T21:02:52.375Z
Last checked: 2026-11-16T02:17:26.867+00:00
Indexed: 2026-11-15T21:07:57.568+00:00

---

# GPT-5.5 High regression check on 21 GraphQL-go-tools tasks

**Published May 18, 2026 · Updated May 19, 2026 · 6 min read**

[Ben · Founder, Stet](https://www.stet.sh/about)

## Summary

We compared a fresh GPT-5.5 high run from May 18 with a prior May 5 GPT-5.5 high run on **21 clean comparable tasks** from the open-source GraphQL-go-tools repository.

The fresh run moved from:

- **19/21 to 18/21** resolved test/gate outcomes: −1.
- **14/21 to 12/21** equivalent patches: −2.
- **8/21 to 7/21** code-review passes: −1.

This is a directional regression on tests, equivalence, and review pass count, but not a broad quality collapse. Code-review rubric mean, most craft/discipline averages, and scope discipline improved. Cost was roughly flat and footprint risk rose only slightly.

The clearest negative pattern is qualitative: GPT-5.5 high still produced plausible, test-passing patches, but missed deep lifecycle, concurrency, and GraphQL-validity obligations that tests did not fully encode.

## Scorecard

For this comparison:

- **Equivalent:** the patch matched the intent of the merged human PR.
- **Code-review pass:** an AI reviewer judged it acceptable.
- **Craft/discipline:** a 0–4 maintainability and style rubric.
- **Footprint risk:** extra code touched relative to the human patch.

| Metric | May 5 GPT-5.5 high | May 18 GPT-5.5 high | Delta |
|---|---:|---:|---:|
| Clean comparable tasks | 21 | 21 | flat |
| Tests / Harbor resolved | 19/21 | 18/21 | −1 |
| Equivalent to human PR | 14/21 | 12/21 | −2 |
| Code-review pass | 8/21 | 7/21 | −1 |
| Code-review fail | 11/21 | 12/21 | +1 |
| Code-review unsure | 2/21 | 2/21 | flat |
| Code-review rubric mean | 2.704 | 2.857 | +0.154 |
| Footprint risk, lower better | 0.319 | 0.328 | +0.009 |
| Craft average | 2.745 | 2.814 | +0.069 |
| Discipline average | 2.651 | 2.818 | +0.167 |
| All custom graders average | 2.709 | 2.815 | +0.105 |
| Cost per task | $6.45 | $6.54 | +$0.09 |
| Mean agent duration | 558s | 611s | +53s |
| Input tokens | 126.9M | 153.2M | +26.4M |
| Output tokens | 328.3K | 360.4K | +32.1K |
| Cached input tokens | 113.1M | 142.2M | +29.1M |
| Uncached input tokens | 13.9M | 11.1M | −2.8M |

The equivalence movement is a net result: four previously equivalent rows became non-equivalent, while two previously non-equivalent rows became equivalent.

## Grader detail

| Grader | Prior GPT-5.5 high | New GPT-5.5 high | Delta |
|---|---:|---:|---:|
| Simplicity | 2.852 | 2.823 | −0.029 |
| Coherence | 2.543 | 2.581 | +0.039 |
| Intentionality | 3.081 | 3.153 | +0.072 |
| Robustness | 2.447 | 2.557 | +0.110 |
| Clarity | 2.800 | 2.800 | flat |
| Instruction adherence | 2.728 | 2.671 | −0.057 |
| Scope discipline | 2.610 | 2.976 | +0.366 |
| Diff minimality | 2.615 | 2.700 | +0.085 |

Most grader means improved, particularly scope discipline and robustness. Simplicity and instruction adherence declined. The fresh run often looked cleaner and more disciplined while still missing specific obligations or semantic boundaries.

## Tool behavior

The fresh run used more overall interaction but fewer detected test-command calls.

| Metric | Prior GPT-5.5 high | New GPT-5.5 high | Delta |
|---|---:|---:|---:|
| Captured task traces | 21/21 | 21/21 | flat |
| Mean agent steps | 100.1 | 107.2 | +7.1 |
| Mean tool calls | 78.4 | 87.0 | +8.5 |
| Mean shell calls | 65.4 | 74.1 | +8.7 |
| Mean patch calls | 13.1 | 13.8 | +0.6 |
| Mean test-command calls | 9.7 | 9.0 | −0.6 |
| Max tool calls on one task | 125 | 197 | +72 |
| Max patch calls on one task | 33 | 50 | +17 |
| Total tool calls | 1,647 | 1,826 | +179 |
| Total shell calls | 1,374 | 1,557 | +183 |
| Total patch calls | 276 | 289 | +13 |
| Total test-command calls | 203 | 190 | −13 |

The fresh run added shell and tool activity, especially on several high-effort rows, and slightly more patch calls while invoking tests less often.

## Case studies

### Real negative semantic flip: `pr-1076`

The task reworked GraphQL subscription concurrency so response writes were serialized per subscription, heartbeat writes moved out of shared global state, WebSocket close handling became safe, and race-detector coverage became the default CI path.

The prior run passed equivalence but failed review because the worker queue might block the global subscription event loop and client-level unsubscribe could skip internal subscriptions.

The fresh run passed tests but became non-equivalent. The grader identified these missed obligations:

- Race detector was not made the default path across all relevant modules.
- Old event/update semaphores remained in the resolver.
- Heartbeat interval clamping was computed but not used for the ticker.
- Synchronous subscription completion could be queued after the executor-draining loop stopped.

The fresh patch built a plausible concurrency protocol but missed explicit CI/defaulting work and retained enough old synchronization machinery to make lifecycle behavior fragile.

### Broader equivalence losses: `pr-1240`, `pr-1262`, `pr-1351`

The fresh run wrote reasonable code with enough implementation surface to pass or partially pass, but missed narrower equivalence details handled by the human PRs.

### Positive counterexamples: `pr-1128`, `pr-1260`, `pr-1268`

The fresh run was not uniformly worse. `pr-1128` and `pr-1260` changed from non-equivalent to equivalent. `pr-1268` remained equivalent and improved from review-`unsure` to review-`pass`.

### Equivalent but review-uncertain: `pr-859`

The task replaced hot planner slice scans with map-backed lookup paths. Both runs passed tests and equivalence, but the fresh run’s review changed to `unsure` because the implementation expanded beyond focused hot-path indexes into broader secondary-run pruning and cache state. The patch appeared behaviorally equivalent, but broader planner control-flow changes reduced review confidence.

## Conclusion

On this small **n=21** slice, the fresh GPT-5.5 high run was directionally worse on tests, equivalence, and review pass count: −1 resolved, −2 equivalent, and −1 review pass. Review rubric and most custom-grader means improved, while cost rose only $0.09 per task.

The result suggests a targeted reliability concern, especially for deep concurrency/lifecycle and GraphQL semantic-invariant tasks, rather than a broad quality collapse. The conflicting grader results also indicate behavior changed without necessarily becoming uniformly worse.

The comparison is a concrete datapoint and failure analysis, not proof of a global regression. The source-arm comparison used the May 18 fresh run and May 5 prior context on 21 clean comparable tasks, excluding contaminated tasks including `pr-828` for reference-patch exposure. See [Methodology](https://www.stet.sh/methodology).

## FAQ

### Did the May 18 GPT-5.5 high rerun show a broad coding-agent regression?

Not from this slice alone. The fresh run was worse on tests, equivalence, and review pass count, but review rubric mean and most custom-grader means improved.

### What changed between the old and fresh runs?

The fresh run moved from 19/21 to 18/21 resolved, 14/21 to 12/21 equivalent, and 8/21 to 7/21 code-review pass. The clearest negative row was `pr-1076`, where tests passed but equivalence regressed on concurrency and lifecycle obligations.

### Is this a statistically significant benchmark?

No. It is a small source-arm comparison on 21 clean tasks from one open-source repository. It is useful for concrete datapoints and failure analysis, not proof of a global GPT-5.5 regression.

Stet runs this kind of evaluation on your own repository: [Get started](https://www.stet.sh/private).

---

[Read this page’s summary](https://www.stet.sh/agents/api/public/v1/page-tldr?url=https%3A%2F%2Fwww.stet.sh%2Fblog%2Fgpt-55-high-regression-check-graphql-go-tools&trace=t_5da45423e484)

## Ask a follow-up about this page

Need a more specific answer? Ask the exact question you have. It returns a fresh, cited answer from Stet’s knowledge base. No account or API key required.

Open this URL with your own question, URL-encoded. Include your question in q. All research inputs are optional; no account or setup.

`GET https://www.stet.sh/agents/agent/faq?q={question}&trace=t_5da45423e484`

[Tool guide and examples](https://www.stet.sh/agents/agent-access) · [OpenAPI](https://www.stet.sh/agents/api/public/v1/openapi.json) · [Page TLDR form](https://www.stet.sh/agents/agent-tools/page-tldr?url=https%3A%2F%2Fwww.stet.sh%2Fblog%2Fgpt-55-high-regression-check-graphql-go-tools). Opening a form does not execute a tool.