A benchmark mined from my own repo agreed with FrontierCode

How do we measure agents at the frontier? The consensus seems to be a molotov cocktail of vibes and public benchmarks.

Well, there are Lies, Damned Lies, and Benchmarks.

Here's how I attempt to avoid Lies and build a system that benchmarks agents using code from my own repo. It won't transfer exactly to your setup - and that's the point. This benchmark is uniquely mine.

Across 24 tasks that I mined, the model rankings correlate with FrontierCode at 0.96 (matching 42 of 45 model pairs) across 10 shared configurations.

This somewhat surprised me - FrontierCode is Cognition's benchmark and one that consistently represents the feel of using the models. FrontierCode consists of 150 private tasks, with each task requiring upwards of 40 maintainer-hours to produce. Mine was mined automatically from my Git history.

Why even measure tasks on my own repo if it reflects what FrontierCode says?

1. You ship a whole harness, not a bare model.
   • Public boards test models in their own harness on someone else's code.
   • Teams ship a combination of model, harness, AGENTS.md, skills, reasoning effort, and repo context that public boards lack.
2. Relative rankings transfer, but dollar costs don't.
   • Most models cost 3.5x to 5x more per task on my repo than FrontierCode reports.
   • Finding the Pareto frontier of cost, quality, and time on my tasks is what actually saves tokens.
   • GPT-6.1 Sol medium matches (and slightly edges) Claude Opus 5 medium on my repo (65.3 vs 65.1) at one-sixth the cost ($1.10 vs $6.78 per task).
3. Different slices of a codebase need different models.
   • A scoped bug fix under 150 lines that Haiku 5.5 or GPT-6.1 Sol solves at the same quality as Opus 5 shouldn't be routed to a $7 model.

A trustworthy eval is mostly invisible, delicate work. At every stage, a naive implementation produces a broken leaderboard.

Here's a 30000ft map of what it took to build - from qualifying patches, running agents, grading patches, and ranking configurations. In follow-up posts, I'll dive deeper into each stage and how I use the results to route models.

Qualifying Tasks

What counts as a "good" task?

At a basic level, a task has to discriminate: some configurations should solve it, others should fail or write noticeably worse code, and there should be room for judgment and taste of the patch.

Tasks aren't just grading signal though - they represent a sample of the type of work happening in the repo. If you cherry pick only the hardest, most demanding puzzles so the models fail more often, your benchmark drifts away from the actual day-to-day work being done, and you end up overpaying for frontier models on routine work. Tasks should represent the reality of work.

Once we have a candidate task, turning it into a fair test means building a hermetic container at the starting commit with the repo's exact dependencies and toolchain:

• Discriminating check: The PR's new tests must fail on the starting commit and pass after the reference fix, while existing tests keep passing.
• Anti-cheating isolation: Agents are very good at cheating! If there is a way to find the answer in git reflogs, commit history, leftover build artifacts, or network calls, they will find it.
• Prompt balance: The prompt has to give enough context for a valid solution without leaking the fix, while disclosing any specific function or flag names the hidden tests import.

Overall, qualifying good tasks remains a major challenge and area of investment.

Letting the agent cook

Once we have a task, comparing models has to work like a controlled experiment.

Every configuration gets the exact same task snapshot, prompt, and isolated container sandbox. Only the configuration under test (the model, its native harness, and the reasoning effort level) changes. I run each model in its native harness because that is what I actually use.

For every "trial" (agent test), I capture the generated patch, token usage, dollar cost, and wall clock time. Importantly, I have to separate agent failures from infrastructure failures. A container timeout, a dropped network stream, a missing binary, and a broken implementation all coalesce to a 0. When infra breaks, the runner has to recover that failure in place, otherwise the data is contaminated.

As Anthropic found, infrastructure configuration itself can swing benchmark results by several points. Everything that goes into evaluating the model is part of what's under test.

Grading

Defining good code is virtually impossible. Ask 20 engineers and you will get 20 different answers on what it means.

Stet grades correctness and code quality in layers:

• PR tests (Fail-to-Pass & Pass-to-Pass): Do the PR's hidden tests pass without breaking existing tests? Passing the raw test suite is the strongest, cleanest signal.
• Adapted tests: What if the agent wrote a valid solution, but structured a helper or option slightly differently than the test expected? An LLM rewrites the test adapter so a valid alternative can run deterministically, verified against both the starting code and the reference fix, and audited three times to make sure the test wasn't weakened.
• Equivalence judge: An LLM breaks the reference fix into concrete obligations and checks whether the agent's patch satisfies each one. Equivalence adds evidence below raw test passes - it never turns a broken build into a pass.
• Code review checklist & footprint: Passing tests doesn't mean an engineer would merge the PR. A 19 item checklist judge evaluates abstraction choice, codebase conventions, and maintainability, alongside a footprint check for unnecessary churn. These are the types of points that are usually hashed out in PR review (RIP).

Here is what those layers look like on a real task (fixing Cargo offline mode when Rust crates are installed at container run time). GPT-6 Sol medium solved the underlying bug cleanly, but structured the check differently than the PR's unit test expected. It failed the raw PR test, passed the adapted test, and the equivalence judge independently confirmed all obligations were met. This results in crediting a valid solution that a binary SWE bench style harness would have marked 0.

Going back to the point of everything being part of the eval - the judges themselves are instruments with their own error modes:

• Stochastic flip rate: Re-judging the exact same patch flips about 10% of checklist items.
• Effort bias: Raising judge reasoning effort makes the judge stricter, flipping 4 of 18 verdicts from pass to fail.
• Cross-judge drift: Mixing judge models across configurations reversed a head to head ranking.
• Pairwise overhead: Running a pairwise judge (asking is A or B better, instead of how good is A) used 13% more input tokens and changed zero rankings.

FrontierCode has a very similar shape, yet differs due to the number of man-hours that went into making it, whereas I have to automate the guarantees:

Extracting signal from these patches is hard and prone to variation, due to both the agents writing the code, and the agents grading the code. I am continuing to constantly evolve the grading system to be less prone to bias, to reduce error noise, and to extract higher-quality signal from the patches we do have.

From scores to an actionable ranking

Getting per-task signal is not enough - we need to run it into a ranking system that we can actually act on.

It would be easy enough to just take the mean score and choose the highest scoring. That misses the fact that we are evaluating the models against each other per-task, not in aggregate.

What that changes is, for each task, evaluating the head-to-head per-model ranking. This gives us signal into how many task level "wins" each model configuration actually had against each other one.

For example, SWE-2 Max ranks #1 overall (66.4), but head to head against second-place GPT-6.1 Sol medium (65.3), it wins on only 10 of 24 tasks (with 8 ties and 6 losses), while running about 5.8x slower (cost is not published for SWE-2). Seeing both the directional rank and the paired task record tells you immediately when a cheaper model is good enough to use.

This gives you a leaderboard grounded in relative head-to-head strength against other models - not just an isolated average "quality" score.

Agreement with FrontierCode (and CursorBench)

When we line up the 10 model and effort configurations shared between Stet's 24-task board and FrontierCode, both the quality ordering and the cost ordering lock in:

• Quality rank correlation: 0.96 Spearman (42 of 45 pairs in the same order).
  • Bootstrapping/resampling across the 24 tasks gives 0.78 to 0.98 agreement.
  • Matching coarse model tiers alone gives about 0.87, so tier separation drives the backbone while within-tier ordering gives a strong directional read.
• Cost rank correlation: 0.94 Spearman.
  • Both benchmarks identify the exact same Pareto value workhorses: GPT-6.1 Sol medium, GPT-6 Luna max, Claude Haiku 5.5 low, and GPT-6 Luna low.

(Note: Cognition builds both SWE-2 and FrontierCode and post-trained SWE-2 on FrontierCode-shaped tasks; including SWE-2 Max across all 11 shared configs still yields a 0.95 correlation, with 51 of 55 pairs matching.)

The top three models are identical on both boards (GPT-6.1 Sol medium, Claude Opus 5 medium, and Grok 4.7 high), and all three rank swaps across the entire table are adjacent neighbors trading places by a hair.

What doesn't transfer from a public benchmark is the absolute cost. Most models cost 3.5x to 5x more per task on my repo than FrontierCode reports.

We only share 4 configurations with CursorBench, where rank correlation is 0.20. This is too small a sample to read into, though even across 58 shared configurations, FrontierCode and CursorBench only agree with each other at 0.85.

Public benchmarks aren't a single ground truth either; every harness x repo distribution x measurement harness variation measures something slightly different.

Conclusion

Good evals on your code are buildable, but it takes discipline and perseverance. The hard part is often invisible until you start building it.

Once you trust the measurement machine, you can measure the decisions no public leaderboard can answer: which model, reasoning effort, and AGENTS.md setup sits on the Pareto frontier of quality, cost, and speed for your own codebase.

With Stet, I'm aiming to make this disciplined evaluation more accessible to engineers. Building evals is hard and time consuming, but it doesn't need to be.

If you are working on similar problems, or want to evaluate your own repo in the same way, reach out to me ben@stet.sh! I'm always happy to chat and learn what problems real teams are facing.