Recorded setup pairings
- Quality
- Even
- No clear difference
- 1–9 head to head, 14 tied
- Speed
- 2.4× faster
- GPT 5.6 Luna max
- 28.5 min vs 68.3 min per task
- Cost
- 3.1× cheaper
- GPT 5.6 Luna max
- $0.90 vs $2.76 per task; $2.76 from 23/24 tasks
GPT 5.6 Luna low leads on speed and cost; no clear quality difference.
23/24 tasks run by both setups
- Quality
- Even
- No clear difference
- 1–9 head to head, 13 tied, 1 undecided
- Speed
- 10× faster
- GPT 5.6 Luna low
- 6.7 min vs 68.3 min per task
- Cost
- 54× cheaper
- GPT 5.6 Luna low
- $0.05 vs $2.76 per task; $2.76 from 23/24 tasks
Questions about these results
- Which is better for coding, GPT 5.6 Luna or SWE-2?
- GPT 5.6 Luna cheaper and faster in both pairings. No clear quality difference in any pairing. 2 pairings; 24 merged changes in Stet-AI/stet.
- Where do these coding results come from?
- Both coding agents worked the same 24 merged changes from the private Stet-AI/stet repository, each in the real repo as it was before the change. The repo’s tests and judges graded every patch against the merged change; no public benchmark scores are used.
- Are reasoning settings combined into one model score?
- No. Each effort pairing keeps its own recorded quality call, head-to-head record, cost and agent time. Incompatible scores are never averaged.