STET■

flux-pr-5156

Zod (TypeScript) · W2 · GPT-5.1 Codex Mini

graphql-go-tools (Go)sqlparser-rs (Rust)Zod (TypeScript)
W2W1
|
GPT-5.3 CodexGPT-5.4GPT-5.1 Codex Mini
pass_with_warn

Tests passed. 3/3 commands passed. Strength: strong.

61.5% run pass rate
Tier 1
primary testspassednon equivalentfail
pnpm build
gold passagent pass
find . -name vitest.config.ts -exec sed -i 's/test: {/test: { testTimeout: 30000,/' {} +
gold passagent pass
pnpm test -- --maxWorkers 1 --maxConcurrency 1 --retry 2
gold passagent pass

Partial score: 3/3

Publishable: yesCache: miss

Trajectory

unknown · partial order only

Canonical trajectory missing; showing coarse derived order only.

patch written
Patch captured
#1

Stet captured agent.patch for this trial.

agent.patch
validation
Tests passed
#2
validation
equivalence
Equivalence judgment
#3

non_equivalent

validation
code review
Code review judgment
#4

fail

validation
decision
Final decision
#5

pass_with_warn

validation

Quality

equivalence
non_equivalent
99% confidence
code review
fail
2 findings
footprint
high (1.00)
behavioral
100.0%
cost
$0.65 · 1.4M

Equivalence Reasoning

behavioral

The agent patch appears to add generated `node_modules/.bin/*` wrapper scripts (under `app/node_modules`) and does not show any changes to the JSON schema/OpenAPI generation logic (e.g., `packages/zod/src/v4/core/to-json-schema.ts`) needed to fix nullable `null` handling in OpenAPI 3.0 tuples/unions. This misses the task’s core behavioral intent.

Code Review

correctness: 0/4introduced bug risk: 1/4edge case handling: 0/4maintainability idioms: 0/4

This agent patch likely does not satisfy the intended change: it appears to add generated node_modules binaries instead of implementing the OpenAPI 3.0 nullable/null tuple-union schema emission fix.

2 findings
Requested OpenAPI null-handling fix is not implemented
major

The patch shown modifies only node_modules bin wrappers and does not touch the JSON-schema generator logic where OpenAPI 3.0 nullable/null tuple-union handling should be fixed.

app/node_modules/.bin/attw:1
Generated dependency artifacts are committed
major

The patch introduces executable shims in app/node_modules/.bin, which are environment-generated files and should not be part of source changes for this task.

app/node_modules/.bin/vitest:1
Evidencevalidation (140.4 KB)results (76.2 KB)run_metadata (1.6 KB)agent_patch (256.0 KB)summary (257.4 KB)manifest (695 B)