Skip to the comparison
STET
Join the waitlist

DeepSeek V4.1 Flash max vs GPT 6 Luna low

No clear quality winner: DeepSeek V4.1 Flash max won 17, GPT 6 Luna low won 3, 3 tied, 1 undecided; GPT 6 Luna low leads on cost and agent time: 9.2× cheaper and 7.2× faster.

24 tasksData from

Stet score

DeepSeek V4.1 Flash maxdeepseek-harness · max reasoningRank 11 of 16

45.995% range 35.5–56.126.0 points higher

GPT 6 Luna lowcodex · low reasoningRank 16 of 16

19.995% range 11.6–29.5

Their 95% score ranges don’t overlap.

Cost / task

DeepSeek V4.1 Flash max$0.49from 23/24 tasks
GPT 6 Luna low$0.059.2× cheaper

Agent time / task

DeepSeek V4.1 Flash max49.3 min
GPT 6 Luna low6.9 min7.2× faster

Quality

Head-to-head matchups

DeepSeek V4.1 Flash max won17

Tied31 undecided

GPT 6 Luna low won3

DeepSeek V4.1 Flash max’s win share 79%, 95% range 64–91%Ties are within judge noise.

Task by task

DeepSeek V4.1 Flash max solved 7 tasks that GPT 6 Luna low didn’t, and GPT 6 Luna low solved 0 that DeepSeek V4.1 Flash max didn’t. Both solved 5; neither solved 11.

DeepSeek V4.1 Flash max only 7

  • stet-5276aa3fPassed tests
  • stet-commit-bbbbae09Passed tests
  • stet-66762b4cPassed tests
  • stet-commit-5d766ddePassed tests
  • stet-43493c52Passed tests
  • stet-commit-f0caded9Passed tests
  • stet-commit-342d0ac0Equivalent to the merged fix

GPT 6 Luna low only 0

A task counts as solved when its tests passed or a judge found the fix equivalent to the merged change, as in the leaderboard’s Solved column. Stet score also weighs code quality, so it can order the two differently. 1 of the 24 tasks is left out because one of the two has no scored result on it.

All 23 compared tasks
Task by task
TaskDeepSeek V4.1 Flash maxGPT 6 Luna lowSolved by
stet-3169aefaPassed testsPassed testsBoth
stet-89dfbc27Passed testsPassed testsBoth
stet-b28f4fe7Passed testsPassed testsBoth
stet-232b50baPassed testsPassed testsBoth
stet-5276aa3fPassed testsFailedDeepSeek V4.1 Flash max only
stet-commit-bbbbae09Passed testsNo change madeDeepSeek V4.1 Flash max only
stet-66762b4cPassed testsFailedDeepSeek V4.1 Flash max only
stet-15439c21Passed testsPassed testsBoth
stet-commit-5d766ddePassed testsNo change madeDeepSeek V4.1 Flash max only
stet-43493c52Passed testsFailedDeepSeek V4.1 Flash max only
stet-commit-f0caded9Passed testsFailedDeepSeek V4.1 Flash max only
stet-6f84e978FailedFailedNeither
stet-2450ca2dFailedFailedNeither
stet-commit-342d0ac0Equivalent to the merged fixFailedDeepSeek V4.1 Flash max only
stet-commit-f8ae1177FailedNo change madeNeither
stet-commit-5e075641FailedFailedNeither
stet-commit-2826765eFailedFailedNeither
stet-commit-a8716b88FailedFailedNeither
stet-commit-e928166fFailedFailedNeither
stet-commit-30506f8eFailedFailedNeither
stet-commit-87814583FailedFailedNeither
stet-commit-d2ab0c25FailedFailedNeither
stet-commit-e41c2e2dFailedFailedNeither

Side by side

Side by side
MetricDeepSeek V4.1 Flash maxGPT 6 Luna low
Harnessdeepseek-harnesscodex
Reasoning effortmaxlow
Rank11 of 1616 of 16
Stet score45.995% range 35.5–56.119.995% range 11.6–29.5
Solved12/2411 passed, 1 judged equivalent5/245 passed, 0 judged equivalent
Cost / task$0.49from 23/24 tasks$0.05
Agent time / task49.3 min6.9 min

See which setup wins on your repo

Stet replays your merged PRs, scores quality above pass/fail, and delivers comparison reports through your agent.

Run it on your repo

About this data

  • 24 tasks, each a merged change replayed from the repo as it was just before the merge.
  • Data from .
  • Both setups come from the same leaderboard run of 16 setups: bmk_d4ba310fdf50db4ccb2092e3.

See all 16 setups on the leaderboardHow Stet measures