GLM 5.2 on 50 real Go and Rust PRs: last on quality, and not the cheapest

GLM 5.2 has been getting a lot of hype for being a "frontier killer". We can evaluate it 2 ways: how cheap is it, and is good enough for most work?

On tasks from these 2 open source repos, it is neither. It costs roughly twice Composer 2.5 for lower quality on both dimensions that matter.

Route GLM to supervised first-draft work with a smarter model supervising. Don't run it unattended against production-grade repositories.

The setup: fifty real merged pull requests across two repos — graphql-go-tools in Go and sqlparser-rs in Rust — replayed against frozen snapshots so nothing leaks, one attempt per task. Every patch is graded beyond pass/fail: a craft score (0–4, code quality and idiom) and equivalence (0–1, how closely the patch reproduces the merged human PR's actual behavior). Stet, the local eval harness I build, runs and grades them; a blinded gpt-5.4 judge scores independently of the runner. GLM ran at medium reasoning. Does GLM belong at this table, and if not, where does it actually belong?

It passes tests – so does everyone

GLM passed 38 of 50 tasks. That sounds usable. Composer 2.5 passed 44/50. GPT-5.5 also 44/50. Opus 4.8 hit 47/50.

The test gate has saturated as a discriminator — all four models cluster above 75%. A model can write a structurally plausible patch, leave the core behavior incomplete, and still go green, because, on this dataset, test suites rarely cover every edge the human PR was fixing. GLM does exactly that, repeatedly, as the task contrasts below show.

Routing decisions built on pass/fail won't separate GLM from Opus 4.8. The axes that diverge are equivalence and craft.

Last on quality, in both repos

GLM trails the entire field on craft and equivalence in both repos. Against the premium arms that gap is decision-grade — large enough to survive the statistics at n=50. Against Composer the picture splits by repo. On Go the craft and equivalence gaps are −0.10 and −0.14: noise-band, too close to call. On Rust craft stays close but equivalence falls 0.17 short, which is decision-grade. On the Rust repo every other arm clears 0.95 equivalence; GLM sits at 0.78, the clear outlier.

GLM is last on quality. Composer is also cheap, and it's better.

How GLM actually works

The quality gap has a mechanism:

Turns and tokens

GLM grinds. It ran a median of roughly 135 agent turns per task on Go and 122 on Rust. Opus 4.8 ran a median of ~113 turns on Rust, GPT-5.5 ~94. GLM takes more turns than the premium models and produces worse output; its cost edge comes from cheap per-token pricing, not from efficiency. Median token spend ran ~3.9M per Go task and ~2.5M on Rust.

It writes more than the human, then misses the change

The patch shape is the tell. GLM writes roughly 1.8x the human's churn. On Go the median GLM patch added 222 lines and deleted 16 across 4 files; the human PR was +111 / −47. On Rust, GLM +284 / −12 across 6 files against the human's +110 / −17. GLM bolts new code alongside the existing path and rarely edits or deletes what's there. The premium arms write add-heavy patches too, though, so diff shape alone isn't the gap — the difference is in the content. The same-task contrasts below show GLM adding around the change instead of making it.

Wildly variable

The variance is wide, and the diffs show why. Go #1308 (expose @oneOf input objects through introspection and validation) is the tail-risk case: GLM found the right surfaces — the @oneOf directive, the isOneOf introspection field, the validation rule — and the patch mostly works. But it ran its new check ahead of the existing one, so an undefined variable used in a one-of selection comes back with the wrong error, and it ground 326 turns and 14.1M tokens — only ~34K of them output, the rest re-reading the same files — to get there, at $4.07. Go #1034 (canonicalize variable names while preserving the originals for validation) is the over-write case: the human added a small variables_mapper layer; GLM wrote +522 / −6 — twice the human — pulling in a third-party JSON parser and re-serializing the variables byte-by-byte, then dropped the original-variable path anyway. More code, same miss, craft 1.94.

It can also land cleanly. On Rust #2174 (a derive_dialect! macro for custom dialects, 734 / 285 in the human's PR) GLM shipped an equivalent patch in roughly half the churn — it built the derive path but hand-wrote the dialect method table instead of generating it. Rust #1538 it got right too, but ground through 11.3M tokens and $3.21 to do it. Same model, same settings, wildly different outcomes — which is the routing problem.

Same-task contrasts with Composer

The sharpest window is running both models on identical tasks. sqlparser-rs #1472 makes ! dialect-specific — Hive reads !a as logical NOT, PostgreSQL reads a! as factorial, and a dialect that supports neither must reject both. The human added two opt-in predicates to the Dialect trait (supports_factorial_operator, supports_bang_not_operator, both defaulting to false) so each dialect declares what it allows and everything else rejects ! for free. GLM instead hard-coded dialect_of!(self is HiveDialect | GenericDialect) branches in the parser and let the permissive GenericDialect accept both forms. It passed the happy-path tests — it even added one asserting MsSql rejects ! — but it's non-equivalent: it keys on dialect identity instead of capability, so it accepts syntax the human's design rejects. Composer shipped the equivalent, review-clean patch, craft 3.68. sqlparser-rs #1493 (JSON_TABLE FOR ORDINALITY) is the same shape — Composer equivalent and review-clean, craft 3.53; GLM non-equivalent because it parses a data type before checking for FOR ORDINALITY, so the real, type-free n FOR ORDINALITY won't parse and only the non-standard n INT FOR ORDINALITY does — the right code in the wrong order. Two more in the same vein: on Go #859 ($0.36) GLM made the planner's path lookups O(1) with three parallel caches but never touched the DataSourceMetadata membership scans the task also called out — half the optimization. On Rust #1398 ($0.73) it changed how the interval value parses but never added the error that rejects a missing unit, and it deleted the INTERVAL 1 + 1 DAY regression test rather than make it pass — the suite went green by dropping the coverage that would have failed.

The pattern is a plausible partial: tests go green, behavior stays incomplete — which is exactly the failure a test-only gate cannot see.

Cheaper than frontier, pricier than Composer

GLM at $1.40/task on Go and $1.04 on Rust runs about 0.3x what the premium arms cost: Opus 4.8 $3.02–$3.98, GPT-5.5 $3.41–$4.69, Opus 4.7 $3.55–$5.93. Measured upward, the cost position is real.

Measured against the model directly below it, it isn't. Composer costs $0.71 on Go and $0.53 on Rust — GLM is roughly 2x Composer in both repos, for worse quality. The blended local score makes GLM look closer than it is: Opus 4.8 80.2, GPT-5.5 75.6, Composer 71.6, GLM 64.6. GLM sits only ~7 behind Composer because code footprint is the one component where it doesn't trail, and it props the composite up. The per-axis read is harsher and more accurate: decision-grade behind the entire premium field on craft and equivalence, decision-grade behind Composer on Rust equivalence.

The "cheap" story only holds if you ignore the model sitting right below GLM in the standings.

On lived cost: running the same fifty-task eval consumed 100% of a week's usage on GLM's $60/week plan. The same tasks on Composer used about 30% of a month's usage on its $20/month plan — and GLM's tighter caps fragment a long run into pieces the way Composer's don't.

Route by behavior, not the test gate

If you already run Composer 2.5, GLM buys you nothing on these workloads — more turns, more cost, lower quality. The routing comparison isn't close.

GLM's honest slot is supervised first-draft generation: work where a human or frontier model is tightly guiding it. The deletion-aversion and the plausible-partial pattern are manageable with eyes on the output; they are not manageable at scale without them. Tighter turn and cost caps contain the flail cases like #1308's 326-turn, $4.07 grind, but caps limit the damage — they don't fix the incompleteness.

GLM is a draft generator with a reviewer attached, not a merge trigger.

What this is and isn't

Fifty tasks per model, two repos — graphql-go-tools in Go and sqlparser-rs in Rust — June 2026. Grader: blinded gpt-5.4, independent of the runner. Single seed, one attempt per task. Contamination audited clean on both repos. Costs are cache-aware. GLM ran at medium reasoning.

Per-repo replication is the confidence mechanism: when a finding holds in Go and in Rust, that's a stronger signal than one repo alone. Pooled numbers are directional, not definitive.

This is not a general benchmark — two repos, one domain cluster (parsers and query planning), one judge model, one agent scaffold. Other code domains, task distributions, and harness configurations may rank these models differently. I can't build a routing rule for your codebase out of these numbers.

Measure your own harness on your own code. GLM's behavior on sqlparser-rs says something specific about dialect-sensitive parser work; whether that carries to your service layer, your migrations, your infrastructure code is an open question. What would these standings look like on the three repos you actually maintain?