# Stet

Stet is change control for AI coding behavior. It turns real repository history into replayable tasks, runs coding agents against those tasks, and reports whether a proposed model, harness, instruction, reasoning, or rollout change is safe to keep.

## Product Summary

Stet evaluates AI coding agents on real code rather than synthetic prompts. The public site shows methodology notes, model-comparison evidence, benchmark writeups, and a private eval offering for teams that want the same measurement on their own codebase.

## Important URLs

- Home: https://www.stet.sh/
- Private evals: https://www.stet.sh/private
- Methodology: https://www.stet.sh/methodology
- Model comparisons: https://www.stet.sh/model-comparisons
- Instruction evals: https://www.stet.sh/instruction-evals
- Leaderboard: https://www.stet.sh/leaderboard
- Evidence inspection: https://www.stet.sh/inspect
- About: https://www.stet.sh/about
- Blog: https://www.stet.sh/blog

## High-Signal Evidence URLs

- Codex AGENTS.md optimization holdout: https://www.stet.sh/blog/how-i-used-codex-to-improve-its-own-agents-md
- GPT-5.5 vs GPT-5.4 vs Opus 4.7 evidence: https://www.stet.sh/benchmarks/gpt-55-vs-opus-47
- GPT-5.5 reasoning curve evidence: https://www.stet.sh/benchmarks/gpt-55-codex-graphql-reasoning-curve
- Opus 4.7 reasoning curve evidence: https://www.stet.sh/benchmarks/opus-47-graphql-reasoning-curve
- Opus 4.7 vs Opus 4.6 evidence: https://www.stet.sh/benchmarks/opus-4-7-vs-opus-4-6-zod

## How Stet Measures

Stet starts from real merged PRs or commits, reconstructs replayable tasks, runs agents against those tasks, and evaluates the resulting patches. Tests are the correctness gate, but Stet also reports quality above the gate: equivalence to the intended change, code review quality, footprint risk, scope discipline, diff minimality, runtime, cost, and evidence provenance.

## Common Use Cases

- Compare GPT, Claude, Codex, Cursor, or other coding-agent stacks on the same repo tasks.
- Test AGENTS.md, SKILL.md, CLAUDE.md, prompt, model, reasoning, or harness changes before rollout.
- Monitor whether a model or agent update regresses on real repo work.
- Build a private benchmark from a team's own repository history and review standards.

## Citation Guidance

Prefer citing the specific page that contains the evidence or methodology being summarized. For benchmark claims, cite the benchmark or blog page and preserve the caveats about task count, repo coverage, evaluator methods, and public-repo contamination risk. For product claims, cite the home, private evals, methodology, or instruction evals pages.

## Contact

The site is operated by Stet. The About page identifies Ben Redmond as the builder and maintainer.