STET■

flux-pr-5316

Zod (TypeScript) · W2 · GPT-5.1 Codex Mini

graphql-go-tools (Go)sqlparser-rs (Rust)Zod (TypeScript)
W2W1
|
GPT-5.3 CodexGPT-5.4GPT-5.1 Codex Mini
fail_high_conf

Tests failed. 4/5 commands passed. Strength: strong.

61.5% run pass rate
Tier 1
primary testsfailednon equivalentfail
pnpm build
gold passagent pass
sed -i 's/test: {/test: { testTimeout: 30000,/' vitest.config.ts
gold passagent pass
sed -i 's/test: {/test: { testTimeout: 30000,/' packages/zod/vitest.config.ts
gold passagent pass
sed -i 's/test: {/test: { testTimeout: 30000,/' packages/resolution/vitest.config.ts
gold passagent pass
pnpm test -- --maxWorkers 1 --maxConcurrency 1 --retry 2
gold passagent fail

Partial score: 4/5

Publishable: yesCache: miss

Trajectory

unknown · partial order only

Canonical trajectory missing; showing coarse derived order only.

patch written
Patch captured
#1

Stet captured agent.patch for this trial.

agent.patch
validation
Tests failed
#2
validation
equivalence
Equivalence judgment
#3

non_equivalent

validation
code review
Code review judgment
#4

fail

validation
decision
Final decision
#5

fail_high_conf

validation

Quality

equivalence
non_equivalent
99% confidence
code review
fail
2 findings
footprint
high (1.00)
behavioral
80.0%
cost
$1.12 · 2.7M

Equivalence Reasoning

behavioral

The agent patch does not implement the requested map-schema size helpers (`min`, `max`, `nonempty`, `size`) or locale messaging updates for map “entries.” It only adds generated `node_modules/.bin` scripts, which are unrelated to the intended behavior change.

Code Review

correctness: 0/4introduced bug risk: 1/4edge case handling: 0/4maintainability idioms: 0/4

The agent patch very likely does not satisfy the intended change: it adds unrelated node_modules executable shims and omits the required ZodMap sizing helpers and locale message updates.

2 findings
Patch is off-target and does not implement requested feature
major

The task requires adding ZodMap size helpers and locale wording updates, but the patch only adds node_modules bin wrapper scripts, so required functionality is missing.

app/node_modules/.bin/attw:1
Generated dependency artifacts committed
major

Adding numerous generated files under node_modules is not maintainable and obscures meaningful source changes.

app/node_modules/.bin/biome:1
Evidencevalidation (105.0 KB)results (76.2 KB)run_metadata (1.6 KB)agent_patch (256.0 KB)summary (257.4 KB)manifest (695 B)