Recorded setup pairings
- Quality
- Even
- No clear difference
- 1–9 head to head, 14 tied
- Speed
- 1.1× faster
- GPT 6 Luna max
- 59.7 min vs 68.3 min per task
- Cost
- 7.5× cheaper
- GPT 6 Luna max
- $0.37 vs $2.76 per task; $2.76 from 23/24 tasks
- Quality
- Even
- No clear difference
- 0–8 head to head, 12 tied, 4 undecided
- Speed
- 9.9× faster
- GPT 6 Luna low
- 6.9 min vs 68.3 min per task
- Cost
- 52× cheaper
- GPT 6 Luna low
- $0.05 vs $2.76 per task; $2.76 from 23/24 tasks
Questions about these results
- Which is better for coding, GPT 6 Luna or SWE-2?
- GPT 6 Luna cheaper and faster in both pairings. No clear quality difference in any pairing. 2 pairings; 24 merged changes in Stet-AI/stet.
- Where do these coding results come from?
- Both coding agents worked the same 24 merged changes from the private Stet-AI/stet repository, each in the real repo as it was before the change. The repo’s tests and judges graded every patch against the merged change; no public benchmark scores are used.
- Are reasoning settings combined into one model score?
- No. Each effort pairing keeps its own recorded quality call, head-to-head record, cost and agent time. Incompatible scores are never averaged.