STET■

flux-pr-5222

Zod (TypeScript) · W2 · GPT-5.3 Codex

graphql-go-tools (Go)sqlparser-rs (Rust)Zod (TypeScript)
W2W1
|
GPT-5.3 CodexGPT-5.4GPT-5.1 Codex Mini
pass_with_warn

Tests passed. 1/1 commands passed. Strength: strong.

69.2% run pass rate
Tier 1
primary testspassednon equivalentfail
pnpm test
gold passagent pass

Partial score: 1/1

Publishable: yesCache: miss

Trajectory

unknown · partial order only

Canonical trajectory missing; showing coarse derived order only.

patch written
Patch captured
#1

Stet captured agent.patch for this trial.

agent.patch
validation
Tests passed
#2
validation
equivalence
Equivalence judgment
#3

non_equivalent

validation
code review
Code review judgment
#4

fail

validation
decision
Final decision
#5

pass_with_warn

validation

Quality

equivalence
non_equivalent
89% confidence
code review
fail
5 findings
footprint
low (0.19)
behavioral
100.0%
cost
$2.54 · 991K

Equivalence Reasoning

behavioral

The agent patch only partially implements the intended behavior. It updates codec callback types in `codec(...)` and `$ZodCodecDef`, but misses the broader pipe typing update (`$ZodPipeDef` transform/reverseTransform), so async codec pipeline typing is not fully propagated. It also omits key build/versioning parts present in the intended change: no update to `packages/zod/src/v4/core/versions.ts` and no `**/package.json` inclusion in `packages/zod/package.json` `files`, which can break stub manifest availability in published artifacts.

Code Review

correctness: 1/4introduced bug risk: 1/4edge case handling: 1/4maintainability idioms: 2/4

The agent patch only partially satisfies the intended change and likely leaves material gaps, especially in stub package generation behavior and metadata shape; it also introduces extra API tightening not clearly required.

5 findings
Stub manifest discovery is keyed to `index.js` instead of declaration entrypoints
major

The generator only emits stubs for directories containing `index.js`, but the task requires scaffolding based on declaration/module typing outputs (`index.d.cts`). This can miss directories that need type metadata for TS resolution.

scripts/write-stub-package-jsons.ts:24
Generated stub `package.json` content is missing expected module type metadata
major

The stub content omits `"type": "module"`, reducing parity with the required manifest scaffolding and potentially affecting module interpretation/resolution in downstream tooling.

scripts/write-stub-package-jsons.ts:5
Directory exclusions can skip needed outputs
major

The scan excludes `src` entirely. If generated artifacts or declaration entrypoints relevant to stub generation are under paths affected by this rule, required package manifests will not be written.

scripts/write-stub-package-jsons.ts:14
Prefault API was tightened beyond requested scope
major

The patch changes `prefault` and its internal default type to disallow `undefined`, while the requirement only called out default-value enforcement. This broadens API impact and may reject previously accepted inputs.

packages/zod/src/v4/classic/schemas.ts:89
Async codec typing changes appear only partially propagated
major

The codec def signatures were widened to `MaybeAsync`, but the broader pipe-level contract is not shown as updated in this patch. Partial propagation risks type inconsistencies in pipeline composition.

packages/zod/src/v4/core/schemas.ts:3653
Evidencevalidation (129.3 KB)results (70.9 KB)run_metadata (1.6 KB)agent_patch (7.8 KB)summary (257.4 KB)manifest (695 B)