STET■

flux-pr-5409

Zod (TypeScript) · W2 · GPT-5.4

graphql-go-tools (Go)sqlparser-rs (Rust)Zod (TypeScript)
W2W1
|
GPT-5.3 CodexGPT-5.4GPT-5.1 Codex Mini
fail_high_conf

Tests failed. 1/3 commands passed. Strength: strong.

69.2% run pass rate
Tier 1
primary testsfailednon equivalentcommand source drift
pnpm build
gold passagent pass
npx vitest run packages/zod/src/v4/core/tests/locales/he.test.ts -t "string\ type\ \(feminine\ \-\ צריכה\)|number\ type\ \(masculine\ \-\ צריך\)|array\ type\ \(masculine\ \-\ צריך\)|set\ type\ \(feminine\ \-\ צריכה\)|array\ max|string\ expected\ \(feminine\),\ number\ received|number\ expected\ \(masculine\),\ string\ received|boolean\ expected\ \(masculine\),\ null\ received|array\ expected\ \(masculine\),\ object\ received|object\ expected\ \(masculine\),\ array\ received|function\ expected\ \(feminine\),\ string\ received|feminine\ types\ use\ צריכה|masculine\ types\ use\ צריך|single\ value|two\ values|multiple\ values|not_multiple_of|unrecognized_keys\ \-\ single\ key|unrecognized_keys\ \-\ multiple\ keys|invalid_union|invalid_key\ in\ object|startsWith|endsWith|includes|regex|email|url|uuid|invalid\ element\ type\ in\ tuple\ shows\ full\ error\ message|inclusive\ minimum\ \(>=\)|exclusive\ minimum\ \(>\)|inclusive\ maximum\ \(<=\)|exclusive\ maximum\ \(<\)|verifies\ all\ type\ translations\ are\ correct"
gold passagent fail
pytest -q tests/behavior/he_locale_behavior_test.py
gold passagent fail

Partial score: 1/3

Publishable: noCache: miss

Trajectory

unknown · partial order only

Canonical trajectory missing; showing coarse derived order only.

patch written
Patch captured
#1

Stet captured agent.patch for this trial.

agent.patch
validation
Tests failed
#2
validation
equivalence
Equivalence judgment
#3

non_equivalent

validation
code review
Code review judgment
#4

fail

validation
decision
Final decision
#5

fail_high_conf

validation

Quality

equivalence
non_equivalent
97% confidence
code review
fail
3 findings
footprint
high (1.00)
behavioral
33.3%
cost
$0.73 · 959K

Equivalence Reasoning

behavioral

The agent patch (as shown) adds generated `node_modules/.bin/*` wrapper scripts and does not implement the intended Hebrew locale behavior changes in `packages/zod/src/v4/locales/he.ts` (gender-aware phrasing, localized type labels, improved wording for the specified issue codes). This misses the core functional intent.

Code Review

correctness: 0/4introduced bug risk: 0/4edge case handling: 0/4maintainability idioms: 0/4

The agent patch likely does not satisfy the task: it appears unrelated to Hebrew locale message updates and instead adds generated node_modules binary shims with high risk and low maintainability.

3 findings
Patch does not implement the requested Hebrew locale changes
major

The task requires edits to Zod Hebrew localization behavior, but the patch shown adds shell wrappers in node_modules/.bin instead of locale logic updates.

app/node_modules/.bin/attw:1
Environment-specific hardcoded paths in added binaries
major

New scripts export NODE_PATH with absolute /app-prefixed package paths, which can fail outside that layout and create runtime inconsistencies.

app/node_modules/.bin/prettier:13
Generated dependency artifacts committed instead of source changes
major

Adding many node_modules/.bin files is repository noise and not maintainable for source-controlled fixes; it obscures meaningful code changes.

app/node_modules/.bin/biome:1
Evidencevalidation (204.4 KB)results (75.3 KB)run_metadata (1.6 KB)agent_patch (256.0 KB)summary (257.4 KB)manifest (695 B)