Cost per successful task ranked the coding agent that lost every head to head 1st

Cost per successful task/PR is starting to become the default in how we measure coding agent impact, as cited by inference providers like Baseten as the “most important metric to track”.

On the surface level it’s easy enough to understand — it’s how much each PR that was successfully merged by an agent costs (total spend / # successes).

On my repo, optimizing for this crowns GPT-6 Luna low as #1 at $0.26 per success, even though it went 0-13 against all other configurations. Meanwhile, the #1 value workhorse on the board (GPT-6.1 Sol medium, $2.64 per success) gets buried at #7 when measuring cost per successful task.

In my last post, I explored how I built a 24 task benchmark from my own repo’s tasks that agreed with FrontierCode v1.1 on model rankings (0.96 correlation). Despite the agreement, my total dollar costs across the tasks ran 3.5× to 5× higher.

Because my costs were so much higher, the industry standard advice, from inference providers to software factories, is to track cost per successful task (or cost per merged PR) on your own repo. When I computed it on my board, it inverted the rankings.

Cost per successful task subtly shifts the denominator of what we’re measuring — it is an average over survivors, and doesn’t reflect the full spectrum of work that leads to the PR.

Here’s where and why it breaks down, and why I rely on paired relative quality and cost on separate axes instead.

What cost per successful task is trying to measure

Cost per successful task creates a single unit economic number ($ / merged PR) so leadership can compare frontier models against small / open source models, and decide which one to roll out across the org.

It’s good to have a topline metric — but this is the wrong one, for three reasons:

• Survivorship bias on task difficulty
• Cheap quitters beat thorough workers
• Binary gates flatten taste

Survivorship bias on task difficulty

We have two models: GPT-6.1 Sol medium (10/24, $2.64/success), and GPT-6 Luna low (5/24, $0.26/success). Dividing each model’s total spend by its own successes ($1.28 / 5 vs. $26.40 / 10) normalizes away the denominator, which ignores the fact that Sol solves twice as many tasks as Luna.

Software engineering tasks aren’t coin flips where you can reroll a cheap model 4x to solve a hard task. Sometimes the agent just can’t solve the task! All 5 tasks Luna low passed were small single commit fixes where 12 to 14 out of 14 models succeeded (and on 3 of the 5, all 14 models passed 100%). On the 14 multi-file feature tasks, Luna low went 0 for 14.

I’d rather pay more money for a model that actually does the work I need it to.

Cheap quitters beat thorough workers

A model that takes 2 turns, decides the problem is too hard, and quits will spend less than a model that takes 50 turns, attempts to solve the problem, and fails.

On my board, GPT-6 Luna low averaged 2.2 median minutes and 12.6k output tokens ($0.05/try). On 3 multi-file tasks, it quit without producing a patch, each in under 4 minutes for less than $0.06. On an example task (Windows runtime evidence), Luna low edited three stub files (117 diff lines) in about 4 minutes ($0.02) and stopped, whereas Luna max worked for nearly two hours ($0.79) across 21 files, building the CI workflow, the adapter, a PowerShell runner, and tests.

Under total spend / successes, quitting in 4 minutes for $0.02 keeps Luna low’s numerator near zero ($1.28 total across all 24 tasks!).

Which is more aligned with human intent — the model that writes a stub and quits, or the model that gives it a fair shot but falls short? Is that difference worth paying for? It’s at least worth knowing about.

Binary gates flatten taste

Binary pass/fail gates that determine “success” or “failure” likely depend on hidden “golden” tests. This is great for determining if the agent implemented the desired behavior, yet it fails to capture some subtler parts of software engineering, such as taste and maintainability. Just because a PR passes CI obviously doesn’t mean that we should merge it.

With the firehose of slop code coming out of coding agents, humans need to be arbiters of what’s “good enough” to merge into the codebase. Review cycles waste scarce human attention. Finding models that are highly aligned with your codebase standards is therefore crucial to maintaining, and even improving, the quality of the codebase.

Failures aren’t free

A counter argument (and how most model routing works!) is essentially “if GPT-6 Luna low is $0.05 per attempt, and GPT-6.1 Sol medium is $1.10 per attempt, let’s just run Luna and escalate to Sol when it fails”.

A naive implementation misses the fact that failures are often not immediately obvious, and in production systems, can manifest in bugs for the customer.

This is an exercise of de-risking, both on the part of the engineer, and on the part of the company. Neither can use the excuse of a dumb LLM if breaking a customer’s experience. Using more expensive tokens is (usually) worth it to ship fewer bugs, especially if you have big customers paying you big bucks. For most people currently, due to the difficulty of evals, it’s then worth it to accept some slippage, and route most reasonably difficult requests to the frontier.

But, your CFO wants to save money, and you only get $200 of tokens per month. Thus, the optimal policy is to only route GPT-6 Luna low the tasks we actually know it can handle reasonably well. The problem is that it’s hard to know ahead of time which tasks Luna can do! The solution is to “slice” tasks, and perform routing to cheaper models based on shown capabilities. And to find those slices, you first need a ranking method that can compare models head to head on the same task.

How I rank instead

Instead of cost per successful task, I measure quality relative to all other agents being measured. Why?

I’m interested in knowing which agent to use for a specific task — not how good the agent is in a vacuum. These sound similar, but are subtly different things.

Usually, benchmarks measure whether or not the agent solves a task with held-out tests, producing a binary score. Some benchmarks include qualitative grading on top of this (would a maintainer accept this PR?). Either way — the end result is a score that is: # passes / total tasks, avg(quality score as determined by LLM), or a combination of both.

Stet grades tasks using a combination of hidden tests, behavioral checklists, code quality, and equivalence to the merged task. This is fundamentally similar to how other benchmarks score agent patches.

The difference is in how we aggregate the scores to answer the question of “which agent is better”.

For every task in the dataset, we compare every agent configuration (harness x model x effort x context) against every other configuration, and compute the wins, ties, and losses each one accumulates. An agent that passes tests will always beat one that fails, and when agents both pass (or fail), we use code review and behavioral similarity to the original patch to break the tie (only counting a win when the measured gap exceeds the judge noise).

We then create Elo for the models (Bradley-Terry, same math behind chess rankings and chatbot arena) to turn the head to head matrix into each configuration’s expected win share from 0-100, where 50 is average.

On the 5 easy tasks that GPT-6 Luna low passed, it mostly tied the other models, and on the other 19 tasks, it lost 198 of 234 decided matchups. Its aggregate pair record is 0-13, putting it at the bottom of the board with a score of 21.2.

(There is a lot of very interesting math here, and I will likely do a further post going deeper into it.)

Slicing

If topline cost / success is broken, then how do we use benchmarks to drive the cost of coding agents down?

The approach I take is to slice the tasks by workload shape and constraints, and look for slices where a cheaper configuration can do as well as more expensive configurations. In other words, figure out where Luna is about as good as Sol, and use Luna in those cases.

Here are some example slices from my work:

Slice 1: Small, scoped fixes vs. multi-file features (where effort & model tier diverge).

• 10 small tasks (median 2 files, a few dozen lines):
  • Haiku 5.5 low ($0.25, 4.7 min median) and Luna low ($0.05, 2.2 min) get most of their passes here.
  • Raising effort to Haiku max ($3.85, 46.7 min) is net negative on small tasks — the agent over-edits, loses diff focus, and scores lower on our code review checklist.
• 14 large multi-file tasks:
  • Cheap low-effort models collapse, failing to perform the work.
    • Luna low median 3.5 files / 144 lines vs Luna max 15 files / 1,430 lines.
    • Luna max beats low on the checklist on 10 of 13 tasks.

Slice 2: Interactive agents vs. one shot background agents (the time axis).

• Cost per task ignores wall clock.
• GPT-6 Luna max ($0.37) is the strongest cheap candidate for asynchronous background tasks, if you can wait 54.6 median minutes (vs 11.0 minutes for Sol 6.1), though GPT-6.1 Sol medium ($1.10) still beat it 13–7–4 head to head.
• For interactive agent usage or fast turnaround scoped work, Haiku 5.5 low ($0.25, 4.7 min median) is the cheap candidate worth testing, while Sol 6.1 medium ($1.10, 11.0 min) remains the default when correctness matters more (13–10–1 over Haiku low).

Conclusion

Metrics are semantic and encode implicit assumptions. “Cost per successful task” assumes failures are free/cheap, success is binary, and every ticket in your backlog is equally hard.

If you’re building these sorts of evals: be thoughtful, and measure what you actually care about. For me that looks like “how good is this agent configuration compared to every other agent configuration” and “can I do this same type of task for cheaper”.