Recorded setup pairings
Haiku 5.5 max leads on speed; SWE-2 max on cost; no clear quality difference.
23/24 tasks run by both setups
- Quality
- Even
- No clear difference
- 1–6 head to head, 16 tied, 1 undecided
- Speed
- 1.2× faster
- Haiku 5.5 max
- 58.4 min vs 68.3 min per task
- Cost
- 1.4× cheaper
- SWE-2 max
- $2.76 vs $3.83 per task; $2.76 from 23/24 tasks
- Quality
- Even
- No clear difference
- 1–11 head to head, 12 tied
- Speed
- 6.9× faster
- Haiku 5.5 low
- 9.9 min vs 68.3 min per task
- Cost
- 11× cheaper
- Haiku 5.5 low
- $0.25 vs $2.76 per task; $2.76 from 23/24 tasks
Questions about these results
- Which is better for coding, Haiku 5.5 or SWE-2?
- Haiku 5.5 faster in both pairings. No clear quality difference in any pairing. 2 pairings; 24 merged changes in Stet-AI/stet.
- Where do these coding results come from?
- Both coding agents worked the same 24 merged changes from the private Stet-AI/stet repository, each in the real repo as it was before the change. The repo’s tests and judges graded every patch against the merged change; no public benchmark scores are used.
- Are reasoning settings combined into one model score?
- No. Each effort pairing keeps its own recorded quality call, head-to-head record, cost and agent time. Incompatible scores are never averaged.