Stet evaluates AI coding models on real tasks from real open-source repositories. Each week, we run the same set of tasks across all models and score them on correctness and quality.

Tests are the gate. A model's solution must pass the same tests that the human-written solution passed. But passing tests is necessary, not sufficient — models that pass the same tests can produce wildly different solutions. Quality dimensions above the gate are where differentiation lives.

Pass rate alone cannot differentiate models. PR-derived tasks are bimodal — easy tasks pass all models, hard tasks fail all models. Two models with identical pass rates can diverge 2–5x on quality.

For every task that passes tests, Stet scores additional dimensions:

• Equivalence — is the patch functionally equivalent to the original PR's solution, or merely a different approach that happens to pass?
• Code review — structured rubric scoring correctness, bug risk, edge cases, and maintainability
• Footprint risk — how invasive is the patch? Over-editing and touching unrelated files increases merge risk
• Cost — API spend per task, for ROI comparison across model tiers

Public benchmarks are increasingly unreliable. Models trained on open-source code memorize benchmark solutions — inflating scores without improving real-world performance. And test suites designed for one implementation often reject valid alternatives. Stet measures models on specific, recognizable projects with their real test suites — and scores what matters after the tests pass.

• Real repositories — not synthetic problems
• Human-written tests — not generated test cases
• Quality above the gate — not just "does it pass" but "does it solve it right"
• Tracked over time — detect regressions week-over-week

Each repository dataset contains 50–200 tasks. A task passes whenpass orpass_with_warn. Effective sample size (n) excludes missing validations.

Pass rate establishes the correctness gate. Among tasks that pass, equivalence rate, code review score, and footprint risk are the ranking signals — they surface the quality differences that pass rate alone cannot.

For each model, we compute pass rate and a deterministic 95% bootstrap confidence interval from task-level outcomes. We use fixed bootstrap settings so repeated runs with identical inputs produce the same interval outputs.

Tiering is conservative. Model A is considered strictly superior to model B only whenpassRate(A) > ciHigh(B). Models within the same tier are differentiated by quality dimensions above the gate.