# Stet research: coding models, instructions and eval methods

Stet publishes a public search API over its indexed pages; the guide is at https://www.stet.sh/agents/agent-access.

Source: https://www.stet.sh/blog
Source observed: 2026-11-15T21:02:52.459Z
Last checked: 2026-11-19T11:08:24.991+00:00
Indexed: 2026-11-15T21:05:35.912+00:00

---

# Analysis and notes from the Stet runs

We publish methodology notes, benchmark evidence, and model-comparison writeups from real AI coding-agent evaluations.

## Research entries

- [Running an agent fleet](https://www.stet.sh/blog/running-an-agent-fleet) — **November 12, 2026.** How one engineer steers a 1M line product with a fleet of coding agents: a Chief of Staff, Team Leads and leaves, hill climbing KPIs, and knowing when to parachute in.
- [Cost per successful task ranked the coding agent that lost every head to head 1st](https://www.stet.sh/blog/cost-per-successful-task) — **November 6, 2026.** On a 24-task repository board, GPT-6 Luna ranks first by cost per successful task despite losing every head-to-head comparison. Survivorship on easy tasks, cheap quitting, and binary gates explain the result; paired quality and cost on separate axes are the proposed fix.
- [A benchmark mined from my own repo agreed with FrontierCode](https://www.stet.sh/blog/benchmark-agreed-with-frontier-code) — **October 19, 2026.** Across 24 tasks mined from Git history, model rankings correlate with Cognition's FrontierCode at 0.96 Spearman over 10 shared configurations. The writeup covers qualifying tasks, running agents, grading patches, and ranking configurations.
- [I ran GPT 6.1 Sol against GPT 6 Sol on 24 tasks from my own repo, and found that it maintained quality while cutting cost in half](https://www.stet.sh/blog/gpt-6-1-sol-vs-gpt-6-sol) — **October 12, 2026.** On 24 merged tasks from Stet's own repository, GPT 6.1 Sol recorded 9 wins, 10 ties, and 5 losses head to head, at about half the cost per task, with bigger patches and more tests.
- [On 25 Stet Tasks, Opus 4.8 Wrote Smaller Patches. Opus 5 Searched Wider.](https://www.stet.sh/blog/opus-4-8-vs-opus-5-same-score-different-routes) — **August 25, 2026.** Opus 4.8 and Opus 5 tied on strict functional score across 25 matched Stet tasks. Opus 4.8 had the lower footprint on 20 tasks; Opus 5 ran more shell commands on 18 and more tests on 15.
- [I Tested Six Ways to Save $$$ on 5.6 Sol. The only one that worked was using Terra](https://www.stet.sh/blog/gpt-56-token-saving-modes) — **July 20, 2026.** Six token-saving interventions produced different results depending on the counting method. None cut the ten-task batch total twice. Caveman and the cheaper Terra model looked mildly cheaper per task, within run-to-run noise.
- [Sonnet 5 vs Opus 4.8: how they behave and when to use each](https://www.stet.sh/blog/sonnet-5-vs-opus-4-8-reasoning-dial) — **July 8, 2026.** On 24 real tasks, Sonnet scaled effort into more checking while Opus stayed flatter through high. Graders leaned Sonnet on clarity and Opus on diff minimality.
- [GLM 5.2 vs Composer 2.5 on 50 real Go and Rust PRs: last on quality, and not the cheapest](https://www.stet.sh/blog/glm-5-2-passes-tests-fails-review) — **June 19, 2026.** Across 50 real merged PRs from `graphql-go-tools` (Go) and `sqlparser-rs` (Rust), GLM 5.2 ranked last on craft and equivalence in both repositories, cost about twice Composer 2.5, and wrote more code than the human while missing the change.
- [Composer 2.5 was 6.5x cheaper on 50 real Rust and Go PRs. Here's where it's actually safe.](https://www.stet.sh/blog/composer-25-cheaper-where-its-safe) — **June 18, 2026.** Composer 2.5 was 6.5–7x cheaper than the compared models and tied the test gate, but finished last on craft in both repositories.
- [When Fable 5 Is Worth the Premium](https://www.stet.sh/blog/when-fable-5-is-worth-the-premium) — **June 14, 2026.** Fable 5 is compared with Opus 4.8, GPT-5.5, and two other models on 30 real GraphQL Go Tools and SQLParser Rust tasks, with taskwise W/D/L results by metric.
- [Composer 2.5 vs Opus 4.8 vs GPT-5.5 vs Opus 4.7 - 50 Real PRs in Go and Rust](https://www.stet.sh/blog/opus-48-vs-gpt-55-vs-opus-47-vs-composer-25) — **June 2, 2026.** Four frontier coding models were graded on 50 real merged Go and Rust PRs for tests, craft, equivalence, and cost. Opus 4.8 led on craft in both repositories.
- [I had Codex iterate on its own AGENTS.md 8 times and measured each version against real PRs. The best one still regressed on a clean holdout.](https://www.stet.sh/blog/how-i-used-codex-to-improve-its-own-agents-md) — **May 27, 2026.** Codex optimized its own `AGENTS.md` against real Stet repository tasks. The best candidate improved the training slice but regressed on a clean holdout and was not safe to ship.
- [GPT-5.5 High Regression Check on GraphQL-go-tools](https://www.stet.sh/blog/gpt-55-high-regression-check-graphql-go-tools) — **May 18, 2026.** A fresh GPT-5.5 Codex high rerun on 21 clean `GraphQL-go-tools` tasks was directionally worse than the May 5 GPT-5.5 high run on tests, equivalence, and review pass count; the evidence was mixed and did not show a broad quality collapse.
- [Opus 4.7 Low Vs Medium Vs High Vs Xhigh Vs Max: the Reasoning Curve on 29 Real Tasks from an Open Source Repo](https://www.stet.sh/blog/opus-47-graphql-reasoning-curve) — **May 12, 2026.** Claude Opus 4.7 was evaluated at low, medium, high, xhigh, and max reasoning effort on 29 matched `GraphQL-go-tools` tasks. Medium won the behavioral metrics; more reasoning did not reliably produce better patches.
- [GPT-5.5 Codex low vs medium vs high vs extra high (xhigh): the reasoning curve on 26 real tasks](https://www.stet.sh/blog/gpt-55-codex-graphql-reasoning-curve) — **May 7, 2026.** An interactive reasoning-effort comparison of GPT-5.5 Codex at low, medium, high, and xhigh on 26 matched `GraphQL-go-tools` tasks.
- [GPT-5.5 vs GPT-5.4 vs Opus 4.7 on 56 real coding tasks from 2 open source repos](https://www.stet.sh/blog/gpt-55-vs-opus-47) — **May 1, 2026.** Across 56 real coding tasks in two open-source repositories, Opus 4.7 wrote smaller patches while GPT-5.5 produced patches that more often survived review.
- [Opus 4.7 vs Old Opus 4.6 vs New Opus 4.6](https://www.stet.sh/blog/opus-4-7-zod) — **April 17, 2026.** Three Opus snapshots had the same 12/28 test pass rate. Above the gate, Opus 4.7 was directionally better and more disciplined, without being fundamentally smarter.
- [Your AGENTS.md is the highest-leverage code you're not testing](https://www.stet.sh/blog/agents-md-testing) — **April 8, 2026.** `AGENTS.md` is loaded into every turn; testing and monitoring changes to it is necessary when operating at organizational scale.
- [Your AI coding benchmark is hiding a 2x quality gap](https://www.stet.sh/blog/both-pass) — **March 14, 2026.** Three models had the same pass rate, but one matched the human patch twice as often. Test pass rate is the gate, not the source of truth.

[Get started](https://www.stet.sh/private) with Stet, or visit [How it works](https://www.stet.sh/how-it-works), [Why](https://www.stet.sh/why), [Leaderboard](https://www.stet.sh/leaderboard), [Model comparisons](https://www.stet.sh/leaderboard/compare), [Instruction evals](https://www.stet.sh/instruction-evals), and [Methodology](https://www.stet.sh/methodology).

---

[Read this page’s summary](https://www.stet.sh/agents/api/public/v1/page-tldr?url=https%3A%2F%2Fwww.stet.sh%2Fblog&trace=t_5da45423e484)

## Ask a follow-up about this page

Need a more specific answer? Ask the exact question you have. It returns a fresh, cited answer from Stet’s knowledge base. No account or API key required.

Open this URL with your own question, URL-encoded. Include your question in q. All research inputs are optional; no account or setup.

`GET https://www.stet.sh/agents/agent/faq?q={question}&trace=t_5da45423e484`

[Tool guide and examples](https://www.stet.sh/agents/agent-access) · [OpenAPI](https://www.stet.sh/agents/api/public/v1/openapi.json) · [Page TLDR form](https://www.stet.sh/agents/agent-tools/page-tldr?url=https%3A%2F%2Fwww.stet.sh%2Fblog). Opening a form does not execute a tool.