STET■

Changelog

All notable changes to Stet are documented here. The format follows Keep a Changelog and Stet adheres to Semantic Versioning.

Unreleased

[v0.14.4] - 2026-08-26

This patch release unblocks Bazel fail-to-pass proof on repositories with a warm or shared Bazel cache. v0.14.3 cleared the two reported selection blockers, but running the customer-shaped journey end to end surfaced a third, independent defect: the broad-command parent proof abstained on every Bazel repository whose cache was not cold, because Bazel console attestation refused the stream that a cached run produces. Any operator passing --bazel-cache-root, or simply running a second build, hit it.

Fixed

  • Accept (cached) rows in Bazel console attestation. A cached row is an explicit per-target result Bazel attests for that action key, but the result regex matched only //label PASSED in Ns, so every cached row fell through to the unsupported-status branch and refused the whole stream. On a warm cache this refused the base and gold receipts for the unnarrowed command, and the declared-broad parent proof abstained with "base trial lacks exact command-bound output".
  • Enforce Bazel closed-world coverage against the selected count rather than the executed count. Bazel reports Executed 3 out of 60 tests when 57 targets hit the cache; requiring executed == selected made the parent proof structurally impossible under caching. Coverage is now proven by one unique explicit result row per selected target, so a stream carrying fewer rows than selected targets is still refused.
  • Match Bazel's plural footer grammar. The footer pattern accepted only passes/fails, but Bazel conjugates with the count and writes 57 tests pass and 3 fail locally for multi-test runs. Every footer above one test failed to parse, independent of caching.
  • Read durable receipt copies when binding raw command-bound output. The raw fallback in the declared-broad admission path consulted only the receipt path the trial reported, which lives under a verifier scratch root that is already released by the time the binding runs, so the fallback could never rescue a stream no parser recognizes. It now consults the task-local copies under build_logs/worktree_verifier/ as well, still bound to the exact command.
  • Restore targeted Bazel narrowing for repositories that configure a selector runner. A prior change upgraded command_intent to declared_broad for any explicitly loaded config carrying a test_selector block with no explicit intent, which made the broad command an immutable parent and disabled targeted fail-to-pass selection entirely: the customer-shaped Bazel journey abstained and ran bazel test //.... Only a route whose fallback is keep_broad now declares the broad parent; skip_task and every other fallback keep narrowing authority. The live tripwire again proves narrowed_proven_dynamic_f2p with exact labels.

Internal

  • Move the Bazel cached result case from the attestation refusal table into accept coverage, and add regression tests for a partially cached run, a fully cached run, and raw binding after scratch release.

v0.14.3 - 2026-08-25

This patch release removes the two remaining v0.14.1 customer blockers in large Bazel monorepos: the broad-pattern enumeration gate now honors the resolved candidate budget, and external-repository labels no longer abort source-coverage queries. Raising the enumeration ceiling requires both --bazel-candidate-budget and --bazel-dynamic-attempt-budget. The pinned Chuck/Bazel release tripwire now reproduces both failing customer shapes so release evidence proves the enforced limits, not the requested ones. (v0.14.2 was tagged but never published: its release run was killed by the CI step timeout corrected here, so no v0.14.2 artifacts exist.)

Fixed

  • Honor the resolved candidate budget on the broad-pattern Bazel enumeration gate: when source-rdeps narrowing declines with a per-pattern limit error and the observed cardinality fits the admitted budget, admit the complete enumeration instead of failing closed at the 16-label default. Selection limit errors now report the enforced limit, and canonical selector receipts gain an additive effective_package_enumeration_limit field so receipts reflect the limit that was actually applied. (00e261c3)
  • Skip external-repository labels (@repo//... and canonical @@repo+version//...) during Bazel source-coverage validation instead of failing the whole query as malformed; hermetic-toolchain labels surfaced by dependency-walking queries no longer abort selection, while shell-hostile labels still fail closed. (00e261c3)
  • Reshape the pinned Chuck/Bazel release tripwire to reproduce the failing customer shapes: an overflow package whose source-rdeps rescue declines, an external toolchain label reached through data, and receipt assertions on the enforced enumeration limit with a negative assertion that no per-pattern limit error occurs. (850b06d5)
  • Record per-target Bazel query provenance without aliasing the run-wide aggregate attempt trail: every target in a package serialized every per-label coverage attempt, producing O(targets x attempts) duplicate records (55 MB receipts on a 302-target package, scaling toward GB on customer monorepos). Per-target provenance now carries the enumeration attempts plus that target's own coverage attempt; the union across targets preserves every distinct attempt. Measured on the live tripwire fixture: 55 MB -> 2.2 MB (~96%), with every distinct query attempt retained. (7acb1555)

Internal

  • Recalibrate the lv2 Chuck/Bazel tripwire bounds to the reshaped fixture's measured cold runtime, then reduce the fixture to mechanism scale. The reshaped 302-label fixture measured ~4,312 s cold (dominated by sequential dynamic verification proofs), making a 3-success gate cost 2.5-5.5 h. Its 302-label package is the control the source-rdeps hatch narrows, and its magnitude was never what it proved, so it is now 41 labels - still decisively above the pinned 16. The packages carrying the customer mechanisms are unchanged: the 18-label package whose hatch declines, and the external-repository toolchain package. Measured cold runtime is now 1,198 s with every enforced-behavior assertion intact, and the contract digest is re-pinned. (ed6bb526)
  • Order the release tripwire's runtime bounds against the CI step budget that actually binds them. The gate's runtime is mirrored in five places, and the outermost - the GitHub Actions timeout-minutes on the tripwire step - was never reconciled with the others: the per-attempt bound (55 min) exceeded the whole step budget (50 min), so a single hung attempt could outlive the step that runs it and die without emitting a verdict. On GitHub's runner the gate measures ~25.5 min per attempt (1.7x the local host), so a 3-success gate needs ~77 min. Bounds are now ordered inner to outer: per-command timeout 1500 s, per-attempt max_runtime_ms 2,400,000 (40 min), step timeout 130 min. The per-attempt bound now fires before the step ceiling, so a genuine hang produces a receipt instead of an unexplained step kill.

v0.14.1 - 2026-08-19

This patch release unblocks Bazel task selection in large, multi-platform monorepos and closes the validation gap that allowed lower-level selector tests to pass without proving the public build path.

Fixed

  • Wire --bazel-candidate-budget and --bazel-dynamic-attempt-budget through pre-materialization selection, retain complete overflow-aware candidate sets instead of enforcing the legacy 16-label gate, and keep first-N sampling forbidden with explicit no-sampling authority in canonical selector receipts. (398f22c1, 4cbef53c)
  • Treat exhausted dependency-query retries as task-local selector infrastructure failures, so an unhealthy canary is rejected without ending healthy sibling or fanout work; shared authentication, toolchain, resource, and timeout failures remain run-blocking. (398f22c1)
  • Execute explicit manual fixed-manifest keep_broad fallback through real base/gold verification, while automatic and omitted fallback policies remain fail-closed with durable selector rejection receipts. (398f22c1)
  • Strengthen the Chuck/Bazel release guard with closed-world scenario and receipt validation, mutation coverage, and CI ownership triggers so release evidence must prove the public high-cardinality selector boundary. (398f22c1)
  • Persist an explicit empty termination error for successful live-scenario verification results, so the release guard can distinguish verified success from an omitted execution state and fail closed on incomplete receipts. (2c9b08e0)

v0.14.0 - 2026-08-10

This release makes onboarding recovery and Dataset Factory decisions evidence-bound, expands Bazel selection safely for large packages, and hardens grader execution and CLI recovery. A candidate-bound Chuck/Bazel tripwire now blocks publication unless three pinned customer-shaped runs satisfy the release identity, evidence, and resource checks.

Added

  • Add a sealed onboarding recovery policy and ledger: bind the target floor, recovery pool, cleanup receipt, retryability, and producer action to exact operation, candidate, and attempt identities; persist decisions and return typed next actions for resume, retry, replacement, or human intervention. (206fc2e7, 4c43c2bb, ec727a76, 599d371e)
  • Add no-spend stet suite factory universe source admission. Validate candidate-blind source manifests and keep invalid states as typed needs_action without discovery, qualification, provider calls, tests, or evaluation. (52becc8a)
  • Add source- and ledger-bound Dataset Factory readiness and materialization planning. Keep prospective facts no-spend and return typed new-request actions without claiming qualification, selection, provider execution, tests, or corpus authority. (de35bbf9)
  • Add stet suite factory universe seal to persist a source-bound prospective-universe membership ledger; shortfalls remain typed needs_action and do not authorize selection, qualification, provider calls, tests, or evaluation. (a174d420)
  • Add --provider-cost-cap-usd to grader regrade admission. Require a provider-issued, non-overridable all-in reservation and per-call/total cost observability; unavailable authority blocks before any provider invocation. (b24c79d2)
  • Add a native Codex v2.alpha_rewardkit judge path with macOS Seatbelt admission, isolated HOME and runtime state, read-only/no-delegation policy, strict structured-output and usage validation, and raw-free fixed failure receipts. (303e63d6)
  • Add bounded full-set Bazel candidate and dynamic-attempt budgets (--bazel-candidate-budget / build.bazel_candidate_budget and --bazel-dynamic-attempt-budget / build.bazel_dynamic_attempt_budget), each capped at 4096 and rejecting overflow instead of sampling or truncating the first N labels. (0bf88c1c, b26ee13e)
  • Add Docker-free local worktree controls to stet eval run and persist source-bound agent-test replay worktrees for durable recovery. (0c783f5d, 1bbb0fe8)
  • Require a clean-candidate Chuck/Bazel release tripwire in CI and release workflows: run the pinned journey three times, bind source/scenario/candidate identity, and block publication on missing dynamic proof, manual rescue, unresolved release-blocking incidents, or resource-bound evidence. (84e6e5e4, 3eeb8e9f)

Changed

  • Make Bazel overflow recovery source-scoped: after every changed test source is exactly query-proven, scoped keep-going rdeps may retain only safe, in-pattern actual test labels. A typed transient ExternalDepsException / exit 37 receives one same-identity retry; partial query evidence stays non-authoritative, and dynamic base-fail/gold-pass remains the only F2P/READY authority. (0bf88c1c)
  • Keep onboarding readiness tied to the requested launch floor; under-floor receipts now recommend expansion instead of presenting partial setup as ready. (b8133bf6)
  • Bind RewardKit contract synthesis, provider usage, and per-task grader applicability to task and contract identity; distinguish missing from not-applicable grading and retain typed unavailable outcomes instead of fabricating scores. (303e63d6)
  • Verify manifest ai_task attestation against the retained enrichment response (llm-diagnostics/<task_id>/enrichment.raw.txt): recorded digests must match the file and the re-parsed <ai-task> body must equal the manifest ai_task. A manifest separated from its llm-diagnostics/ directory is no longer certifiable; --allow-unattested-ai-task remains a recorded degradation. (c36e64b9)
  • Keep repository context read-only, expose lifecycle readiness in baseline status, and preserve machine-readable context and strict-manifest errors. (48c5c3f0, cd42ed99, 35ad4de7, 0637a8b8)
  • Preserve retained failed compare-arm evidence and make mixed-vintage combine, multi-arm repair revalidation, retest repair, and P2P adequacy honor authoritative proof state. (a20f6686, ed39bb2c, ae85df95, ee9d77c3, b3dd0591)

Fixed

  • Separate release readiness from post-release customer acceptance in the live guard. Three clean candidate-bound journey receipts can authorize publication, while customer-only incidents, arbitrary-repository assertions, and external acceptance remain fail-closed under the customer profile. (3eeb8e9f)
  • Preserve the one same-identity retry for a transient Bazel ExternalDepsException even when Bazel also prints its generic internal-crash banner, while continuing to fail closed on authentication, Java/toolchain, sysctl, timeout, and negated-marker diagnostics. (01a0e731)
  • Classify Bazel internal-query, sysctl, and ProcessHandle crashes as typed toolchain infrastructure failures instead of semantic F2P failures. (dbef7740)
  • Admit deletion only for UUID-scoped Bazel command-<UUID>.profile.gz files under verifier output bases; other cache mutations and deletions remain blocked. (9d1eacc8)
  • Distinguish logically removed scratch roots awaiting physical reclamation from failed or retained cleanup in verifier warnings. (308727b0)
  • Disable Git-LFS filters at every git invocation against the disposable worktree object store — agent patch capture, worktree file overlay commits, and the verifier status snapshot — so a global gitconfig declaring a required LFS filter with no reachable endpoint no longer loses agent patches with exit 128 or degrades verifier parser evidence. (db14130c)
  • Skip Bazel config-variant pseudo-labels (.asan/.dbg/.tsan siblings returned by the gold test query) after the primary label proves, recording a redundant_variant_coverage receipt instead of burning one gold trial each; record a gold-trial Bazel exit 4 ("no test targets were found") on a query-derived candidate as a benign no_test_targets skip instead of a failed trial. Explicit test commands and the no-proof failure path are unchanged. (cd3cf6dc)
  • Let --retry-rejected recover an interrupted manifest root missing build-summary.json when prior task or rejection evidence exists, while still refusing empty or mistyped output roots; recovery makes no completed- root claim. Persist direct-build failure summaries and bounded aggregate rejection evidence, and classify selector/executor infrastructure failures as "task not judged" rather than F2P verdicts. (43bada49, f0915f61, cf082ba0)
  • Reject manifest-mode --limit and --skip, fail closed on runstore source or request drift, and propagate cancellation through eval rules plan replay preflight. (38bffbab, bf2dd472, 1cc54a1d)
  • Accept the canonical Bazel <version> label for exact Bazel/Bazelisk runtime probes while rejecting mismatched or free-form labels. (eaba5c61)
  • Classify oracle test.patch conflicts separately from behavioral failures and bound adapted-reference AI work to a 10-minute call with one timeout retry. (a6787a7d)
  • Harden Windows live-scenario executable resolution and native-runtime boundaries for hosted toolcache paths. (8095c3fb, 2b0fb253)

v0.13.0 - 2026-08-05

This release makes onboarding and builds work on real customer repositories: forked Bazel toolchains, LFS-tracked paths, network-restricted hosts, and fixed-HOME credential helpers are now handled in trusted phases, manifest ai_task instructions require attested provenance, and subprocess, cache, and Harbor execution layers are hardened against parallel-load races and hung cleanup.

Added

  • --credential-home-file <src>[=<relative-dest>] (and the build.credential_home_file / runner.worktree.credential_home_file config keys) materialize operator-approved credential files into the replaced HOME of trusted phases, for tooling that reads fixed HOME paths such as a Bazel remote-cache credential helper. File bytes are snapshotted at authorization, written 0700/0600 behind a symlink-refusing join, redacted from Stet-owned diagnostics, and bound into the run fingerprint; destinations that collide with Stet-owned agent directories (.claude, .codex, .config/cursor) are rejected. (30506f8e)

Changed

  • Manifest-carried ai_task instructions now require a generation receipt (ai_task_provenance, or a clean llm_diagnostics.enrichment reference for older manifests). Unattested tasks are skipped as unattested_ai_task_not_allowed; --allow-unattested-ai-task admits them as a recorded degradation (prompt_regime: unattested-provenance, not certified READY). (342d0ac0)
  • Enrichment now emits the manifest ai_task as a single outcome-focused instruction instead of a multi-part template. (5c53988a)

Fixed

  • Runtime probes whose output is recognized but unparsable — for example a fork-suffixed Build label: 9.1.1-glyd1 from a patched Bazel — no longer abstain with a missing observed-environment fingerprint: the fingerprint falls back to hashing the raw normalized probe output, and every empty-fingerprint branch now records an explicit unavailable reason in parser evidence and the gate message. Genuinely ambiguous output (two or more distinct version records) still fails closed. (30506f8e)
  • Materialized Python tasks now probe python3 --version instead of python --version, matching hosts that ship no bare python. (30506f8e)
  • Git-LFS smudge is disabled in the disposable object store used for worktree materialization and when the verifier resets held-out test targets to base, so repos with LFS-tracked paths build and validate without a reachable LFS remote even when a global gitconfig declares a required LFS filter. (30506f8e, 4f387368)
  • Successful --source-mode reference base/head builds now write build-summary.json; previously the build exited 1 after materializing the task ("read build summary: … no such file or directory"), which failed CI wrappers and left eval tooling treating the dataset as incomplete. (4f387368)
  • Share the worktree Bazel selector's Bazelisk and repository download caches across environment-group cohorts under --bazel-cache-root (both are content-addressed, so cross-cohort reuse is byte-safe). Changing the test command no longer starts a cohort with a cold repository cache, which on network-restricted hosts failed the selector as abstained_query_failed. Repository caches under existing .stet-bazel-verifier-cache-v1/cohorts/<fingerprint>/trusted/ directories are no longer read, so the new .stet-bazel-verifier-cache-v1/shared/ directory starts cold: on a network-restricted host, warm it once with network access or copy an existing cohort's trusted/repository contents into .stet-bazel-verifier-cache-v1/shared/repository before the first post-upgrade run. Stet never deletes an operator-owned cache root, so reclaim the old per-cohort directories manually if disk space matters. (f0caded9)
  • Serialize the worktree Bazel selector's binary-readiness cache and retry preflights that failed for non-cacheable reasons, so concurrent cohorts no longer inherit a neighbor's transient failure. (dd8cd346)
  • Cancel subprocess process groups SIGTERM-first with a bounded force-kill fallback so buffered agent output is flushed before termination, and bound Harbor agent execution, Docker cleanup, and Claude command timeouts explicitly. (fafd5496, e928166f, 904a4a58)
  • Tolerate never-materialized output roots during completion registration instead of failing with a raw lstat error. (ded8db24)
  • Honor declared test-command overrides for opaque commands and bind the worktree selector to its final harness authority in stet suite build. (1d33be65, 5c684fdf, b3fbf154)
  • Make onboarding completion replay idempotent and route target selection through a deterministic authority. (9222d23b, 60a0e6c2)
  • Void candidate test verdicts when the gold baseline itself fails, instead of scoring against a broken baseline. (6b1cb716)

Internal

  • Harden R7 head-to-head decision and report authority to fail closed on stale, contradictory, or asymmetric evidence, and centralize decision projection (STET-743, plus related eval-report correctness fixes). (d609fffc, eb2e5da3)
  • Migrate durable build evidence to a single revisioned writer core with secured provenance and path reads (STET-729). (96b5be9c, cf28669a)
  • Add deterministic live-scenario validation lanes that prove worktree, Docker, Harbor, and Windows execution boundaries before release (STET-732–STET-767). (c919a690, 207089c5)

v0.12.2 - 2026-08-03

Fixed

  • Let trusted .stet/stet.yaml configuration forward explicitly named environment and file-backed credentials into worktree coding-agent and candidate-verifier execution. Stet keeps a replacement HOME, withholds unlisted ambient variables, exposes disposable file snapshots, redacts exact credential material from owned output, and refuses credential-bearing candidate patches. (e8f0e686)
  • Prevent automatic update checks from replacing a newer running Stet binary with an older cached or published release, and reject malformed release versions rather than guessing their precedence. (861f453f, 04b47f31)

v0.12.1 - 2026-08-03

Fixed

  • Let operators explicitly forward names-only GitHub and AWS file-backed credential inputs from trusted .stet/stet.yaml configuration or CLI flags into selector and base/gold worktree phases. This prevents private dependency wrappers from falling into interactive login under Stet's replaced HOME, while candidate execution remains credential-free and durable output redacts credential values, paths, and file contents. (274421f0)

v0.12.1-rc.1 - 2026-08-03

Fixed

  • Let operators explicitly forward names-only GitHub and AWS file-backed credential inputs from trusted .stet/stet.yaml configuration or CLI flags into selector and base/gold worktree phases. This prevents private dependency wrappers from falling into interactive login under Stet's replaced HOME, while candidate execution remains credential-free and durable output redacts credential values, paths, and file contents. (274421f0)

v0.12.0 - 2026-08-01

This release makes the agent-first build and onboarding loop more trustworthy on unfamiliar repositories: Stet records the exact scoring and execution authority it observed, keeps retries bound to that authority, and preserves clear infrastructure outcomes instead of inferring success. Onboarding now uses the requested ready-task floor by default; representative selection is available only as an explicit temporary opt-in while it is validated on real repositories.

Added

  • Attest the observed named scoring targets that prove fail-to-pass behavior and project only complete, supported evidence into the scoring surface. (545f2251)
  • Add an evidence-only blind cheat challenger as a third fail-to-pass proof leg in stet suite build, so a cheap solution to the selected tests is recorded without changing the build's proof strength. (70b4ce18)
  • Deduplicate exact-match authority executions while retaining canonical receipts, and resume completed authority plans without rerunning proof. (4a2a94f2, 80812435)

Changed

  • Keep stet suite onboard --target-ready N at its requested ready-task floor by default. Set STET_ONBOARD_REPRESENTATIVE_V1=1 before a fresh onboarding to validate a bounded 2N reservoir and select a static-diversity panel. (ad426e73, b23d5b2b)
  • Give candidate agents a consistent fair-internet-use policy: documentation and general references are allowed, while upstream solutions and reference patches remain blocked. (eebd6911)

Fixed

  • Retry one gold verification only when the first attempt failed before any test executed while acquiring dependencies. Both attempt receipts are retained across direct, batched, Harbor, and worktree validation; network activity during repository tests remains a normal test failure. (f033b557)
  • Let CMake/CTest plans ignore non-test support paths in a test patch while still requiring exactly one declared test source to select a target. (9636aed6)
  • Bind validation replay, authority receipts, and R3 artifacts to their exact certified content identity; ambiguous or incomplete authority fails closed. (1f3b9a2e, 4208c894, c79a79dc)
  • Preserve selected head-to-head validation evidence across revalidation and distinguish absent smoke failure kinds rather than silently borrowing or misclassifying them. (23df59ab, 27f5e406)
  • Isolate Codex evaluator runtime state under the owning output root and forward the selected AI command through onboarding, avoiding shared runtime leakage. (dcf4e35e, 46435f1d)
  • Recognize CMake CTest, PHPUnit, and Swift test plans while keeping incomplete source history and verifier runtime blockers explicit and fail-closed. (fee2e89d, 13f4fe88, 7c43c491, e5584e3e, 06d79c3c)
  • Publish Harbor companion agents as version 0.9.1, matching the version the Stet Harbor invocation pins at runtime. (b23d5b2b)

v0.12.0-rc.2 - 2026-07-31

Fixed

  • Retry one gold verification only when the first attempt failed before any test executed while acquiring dependencies. Both attempt receipts are retained across direct, batched, Harbor, and worktree validation; network activity during repository tests remains a normal test failure. (f033b557)
  • Let CMake/CTest plans ignore non-test support paths in a test patch while still requiring exactly one declared test source to select a target. (9636aed6)

v0.12.0-rc.1 - 2026-07-31

This release candidate makes the agent-first build and onboarding loop more trustworthy on unfamiliar repositories: Stet records the exact scoring and execution authority it observed, keeps retries bound to that authority, and preserves clear infrastructure outcomes instead of inferring success. Onboarding now uses the requested ready-task floor by default; representative selection is available only as an explicit temporary opt-in while it is validated on real repositories.

Added

  • Attest the observed named scoring targets that prove fail-to-pass behavior and project only complete, supported evidence into the scoring surface. (545f2251)
  • Add an evidence-only blind cheat challenger as a third fail-to-pass proof leg in stet suite build, so a cheap solution to the selected tests is recorded without changing the build's proof strength. (70b4ce18)
  • Deduplicate exact-match authority executions while retaining canonical receipts, and resume completed authority plans without rerunning proof. (4a2a94f2, 80812435)

Changed

  • Keep stet suite onboard --target-ready N at its requested ready-task floor by default. Set STET_ONBOARD_REPRESENTATIVE_V1=1 before a fresh onboarding to validate a bounded 2N reservoir and select a static-diversity panel. (ad426e73, b23d5b2b)
  • Give candidate agents a consistent fair-internet-use policy: documentation and general references are allowed, while upstream solutions and reference patches remain blocked. (eebd6911)

Fixed

  • Bind validation replay, authority receipts, and R3 artifacts to their exact certified content identity; ambiguous or incomplete authority fails closed. (1f3b9a2e, 4208c894, c79a79dc)
  • Preserve selected head-to-head validation evidence across revalidation and distinguish absent smoke failure kinds rather than silently borrowing or misclassifying them. (23df59ab, 27f5e406)
  • Isolate Codex evaluator runtime state under the owning output root and forward the selected AI command through onboarding, avoiding shared runtime leakage. (dcf4e35e, 46435f1d)
  • Recognize CMake CTest, PHPUnit, and Swift test plans while keeping incomplete source history and verifier runtime blockers explicit and fail-closed. (fee2e89d, 13f4fe88, 7c43c491, e5584e3e, 06d79c3c)
  • Publish Harbor companion agents as version 0.9.1, matching the version the Stet Harbor invocation pins at runtime. (b23d5b2b)

v0.11.3 - 2026-07-24

Makes a stalled Bazel fail-to-pass selector diagnosable instead of silent: query failures now carry the elapsed time, the argv, and Bazel's own output, the toolchain download is resolved on its own clock so it can no longer masquerade as a query timeout, and trusted selector and verification phases reach the operator's proxy and TLS trust store. Candidate agents also gain a shared fair-internet-use policy across both backends, which changes the exported task instructions.

Changed

  • Give candidate agents the same fair-internet-use policy on both backends: general lookups (documentation, API references, error and language semantics) are permitted, while anything that could reveal the task's reference solution or upstream patch is not. The Harbor path previously carried no policy and the worktree path a blanket ban, so the two backends measured different harnesses and the ban understated capability. This rewrites the exported agent instructions and moves the export cache generation and hash salt v12 to v13, so recorded runs from before this release are no longer prompt-comparable with runs after it. (84b04e4c)
  • Forward proxy routing and the TLS trust store to the trusted selector query and to base/gold verification. On a proxied or TLS-inspected network, stripping those turned a fetch failure into a silent hang. Candidate verification is deliberately excluded and keeps full agent containment; no credentials, tokens, netrc, or agent sockets are forwarded in any phase. (2da09e80)

Fixed

  • Report evidence on every Bazel selector query failure, timeouts included: elapsed time, the exact argv, the names of the forwarded connectivity variables, and a 4 KB tail of Bazel's stdout and stderr. Progress reporting is no longer suppressed, so a stalled selector names the fetch it is waiting on, and capture writes to real files because the process-group kill sets a WaitDelay and os/exec abandons its pipe copiers exactly when the output matters. Credential flag values and URL userinfo are redacted. (2da09e80)
  • Resolve the Bazel toolchain in a separate timeout budget before the first query, which is where Bazelisk downloads the pinned Bazel. An unresolvable toolchain now fails as a toolchain blocker carrying Bazelisk's own error instead of impersonating a query timeout. The outcome is memoized per binary, BAZELISK_HOME, and .bazelversion, so a cohort with a durable cache root pays the download once; preflight timeouts are deliberately not memoized, so one transient blip cannot condemn every remaining task. (2da09e80)
  • Keep the Bazel selector query workspace and output base under the per-cohort trusted root when --bazel-cache-root / build.bazel_cache_root is set, so external repository rules are extracted and executed once per cohort rather than once per task. The extract, patch, overlay, and query span is serialized per durable root by an in-process mutex plus an OS file lock, so concurrent invocations sharing a cohort root cannot corrupt each other. With no cache root the previous ephemeral behavior is byte-identical. (b1110eb3)

Internal

  • Answer the toolchain preflight in the selector cancellation fixtures so they exercise the query they were written to cover rather than the preflight, and correct the BazelQueryRuntime doc comment that still described the query output root as task-scoped. (a3aa591a)
  • Fix a stale BazelQueryCachePaths call site left behind when the signature widened, which stopped internal/cli from compiling, and assert that the query output root also lands under the durable cache root. (1e1ee7a6)

v0.11.2 - 2026-07-24

Makes cold Bazel fail-to-pass selection survivable by giving operators a durable repository cache root, so a first-task timeout warms later runs instead of throwing the downloads away. The rest of the release unblocks valid head-to-head resumes and recovers fail-to-pass tasks whose runner has no function extractor.

Fixed

  • Let operators point the Bazel selector repository cache at a durable path via build.bazel_cache_root / --bazel-cache-root on stet suite build and stet dataset regenerate-f2p. The previous per-command MkdirTemp root was deleted on exit, so a cold-cache selector timeout never warmed later runs. An operator-supplied root is external and unmanaged: Stet creates it if missing and never GC-deletes it; unset keeps the ephemeral default. (4cf34c3a)
  • Restore head-to-head resume for legacy measuring-device digests and experiment-requested custom graders. Recovery stays fail-closed for genuinely unavailable graders, but legacy digests are grandfathered and custom grader specs are threaded into resume pre-flight so valid resumes are no longer blocked. (188be3f7)
  • Recover fail-to-pass tasks on runners with no function extractor by deriving proposal-free file-level candidates from test.patch, stamping narrowing_unverified on the resulting coarse proofs. Terminal abstains now name the concrete recovery routes that were tried. (ec4c5000)

Internal

  • Rework the GPT-5.6 token-saving-modes comparison visualization on the leaderboard, moving its dataset into a dedicated module and refreshing the post's chart rendering. (cc09aeb1)

v0.11.1 - 2026-07-23

Restores fail-to-pass selection on large Bazel repositories, where a hard-coded query deadline rejected every task before evaluation could start. The rest of the release hardens worktree containment, manifest-backed builds, and head-to-head evidence handling so partial, superseded, or untrustworthy results fail closed instead of being silently substituted.

Fixed

  • Make the Bazel fail-to-pass selector query timeout configurable and raise the default from 2m to 10m. Large Bazel repos pay a cold startup in the selector's isolated output base, which silently rejected every task at stage: selector with runtime_classification: timeout. Set build.bazel_query_timeout in .stet/stet.yaml or pass --bazel-query-timeout to stet suite build; stet dataset regenerate-f2p accepts the same flag. The timeout error now names the knob. (a8161f5f)
  • Query-prove every changed source label before large-package reverse-dependency recovery in the selector; a missing or ambiguous source now abstains instead of running a reverse-dependency query against unverified owners. (c05a90e8)
  • Recognize syntactically valid Bazel and Bazelisk target patterns in parser-proven target positions within direct or simply composed commands, via a dedicated command parser. Path-valued flags, redirect operands, post--- arguments, and unsupported shell grammar remain fail-closed in the integrity scanner. (b7a7f93f)
  • Recognize Bazel labels wrapped in transparent command context joined by &&; all other shell grammar keeps normal absolute-path scrutiny so labels in opaque command text are not trusted. (f4331bb9)
  • Accept Bazel root package labels such as //:target in integrity scanning, and reject labels whose path segments are empty, ., or ... (2ab96dba)
  • Retire each task's isolated worktrees, object store, homes, and owned build caches once its artifacts persist, leaving a lifecycle receipt in the task artifact directory. Use --worktree-keep to protect them instead. (f468da1f)
  • Retain worktree replay evidence for validation overrides under its own output root, so non-kept verifier evidence survives while the scratch root stays disposable. (a605b750)
  • Freeze the exact input manifest and every PASS-task patch before planning manifest-backed builds, and retain a sanitized authority copy under the dataset. Reuse, resume, and rejected-task retry now fail closed when the caller manifest or retained authority no longer matches; use --restart only when intentionally replacing that dataset. (66df39f1)
  • Preflight charged experiment arm models while planning stet eval rules, resolving them through the same provider, config, and command semantics execution uses, so an unresolvable model fails before model spend rather than mid-run. (4a3d0116)
  • Restore the persisted variant treatment when resuming stet eval rules, taking the baseline/candidate role from the persisted treatment rather than the arm key, and rejecting missing, ambiguous, or mismatched arm manifests and unsupported persisted harness settings. (04cd8d10)
  • Validate head-to-head recovery against the full persisted grader profile, so repairing a subset of graders cannot silently redefine the persisted profile's rubric digest; unavailable specs fail closed. (d754ddfa)
  • Exclude terminal cells whose patch or execution environment is untrustworthy grader input from grader coverage accounting, so intentionally skipped graders no longer create phantom coverage gaps or repair work while the cells are retained as infrastructure evidence. (5dab833f)
  • Contain worktree integrity violations as fail-closed task-cell outcomes with a terminal receipt, instead of treating them as benign empty patches or escalating them to arm-wide harness failures. (64b8c955)
  • Stop marking an automatically resolved provider command as an explicit user command, so worktree execution can distinguish operator-supplied agent commands from resolved ones. (616cc071)
  • Publish provider-native session evidence once during Harbor normalization, refusing symlinked or irregular sources and detecting conflicting logs, so mutable raw exports cannot replace a selected canonical trajectory after its source trial is superseded. (75632fec)
  • Prefer canonical provider-native trajectory evidence when stitching, rejecting superseded trials and empty or irregular evidence files rather than accepting them as usable. (4ae979c7)
  • Migrate the one legacy case where Harbor normalization retained the only provider-native agent log in a superseded flat trial, binding every field of the retained log to the exact selected cell against a pinned source descriptor before copying its bytes into scratch. No source path is mutated. (1167fe08)

Internal

  • Add the GPT-5.6 token-saving-modes leaderboard study post, covering six treatments against a GPT-5.6 Sol baseline with an interactive comparison view. ([660d97a8])

v0.11.0 - 2026-07-21

Promotes the verified v0.11.0-rc.2 feature set to stable and makes native Codex worktree execution settings explicit and auditable before model spend.

Fixed

  • Enforce supported native Codex worktree execution settings: validate and isolate requested credentials before launch, apply the requested reasoning level, reject unsupported kwargs, and persist a bounded per-task execution-request receipt without claiming provider-side acceptance. (1ebeb2b2)

v0.11.0-rc.2 - 2026-07-21

This follow-up 0.11 candidate makes build and evaluation evidence more durable: Stet can recover partial-clone source authority, preserve only test coverage that remains trustworthy, and keep stitched or repaired comparisons bound to the exact selected trial and measurement function.

Fixed

  • Make portable source packs reliable from Git partial clones: hydrate the pinned base's history and objects before publishing, and report incomplete source history with a concrete recovery path. (fbe0de9a, b42b28c8)
  • Preserve selector-approved terminal test coverage when dynamic F2P finds a focused witness; require the retained command to pass on gold, and abstain when a verifier fails before executing the intended test. (450f4daa, dddb0e98)
  • Reuse safe, command-scoped Bazelisk and repository-download caches for Bazel test selection while keeping mutable query and output state isolated per task. (124d3d6e)
  • Keep stet eval rules corpus-floor checks fail-closed for genuinely diverse task selections; skip subsystem deduplication only when every qualifying task has the same subsystem key. (5276aa3f)
  • Prevent stitched evaluation cells from replacing canonical evidence unless the exact selected trial supplies a usable patch, fresh validation, and provider-native session or trajectory evidence; leave incomplete replacements pending rather than borrowing sibling or superseded evidence. (66053c52)
  • Mark quality-only comparisons with mixed retained grader measurement-function fingerprints as inspect, preventing promotion or superiority claims from directionally incompatible evidence. (8082f5c9)
  • Preserve plugin-overlay identity when repairing retained-study provenance, including manifest-declared overlays. (a07938ce)

v0.11.0-rc.1 - 2026-07-20

This first 0.11 release candidate makes the onboarding and verifier loop more durable: Stet can carry bounded setup and source authority forward, prove proposal-selected tests more honestly, and preserve typed infrastructure causes instead of misreporting them as model or dataset quality signals.

Added

  • Add a durable stet suite onboard ready-task pipeline with cohort ledger, setup-contract reuse, candidate evidence, and operator feedback; add stet suite bootstrap-authority to publish a bounded exact-tree Git authority for enforced reference builds (fac9ed1b, bf4b13d5, 1c31be89)
  • Add schema-constrained proposal test selection, dynamic F2P attestation, bounded repair feedback, and receipt-safe named-test evidence for supported and unsupported test runners (00f99b40, 6b510d68, 76706695, d4e2bbe2)

Changed

  • Preserve selected-command verifier artifacts, authoritative runtime and plugin-overlay activation, and grader/provider provenance across retries and recovery (ad7d7bfe, 31103391, a150b4d6, 9d97146b)
  • Synchronize the shipped public Stet skill bundle with the current source guidance (bbf6f040)

Fixed

  • Classify Bazel and selector toolchain failures as executor infrastructure errors, cancel timed-out Bazel query descendants, and keep fallback receipts from being misreported as F2P or flake outcomes (cc2963a2, 764dd1d1, 89052086)
  • Preserve typed provider failures when rebuilding built-in h2h graders and allow provenance repair without commercial entitlement or telemetry gating (e64567bc, 9d97146b, 15439c21)
  • Sanitize operator and onboarding diagnostics, keep root Node test commands at repository root, and bind proposal-derived F2P proofs to durable command receipts (a58f9248, 122c6ed3, b8d74923)

v0.10.0 - 2026-07-14

Promotes the fully verified v0.10.0-rc.4 candidate to stable with no product behavior changes. Stet 0.10 hardens source authority, portable and reference source handling, bounded artifact and cache lifecycles, worktree and Bazel isolation, runtime fingerprints, GPT-5.6 pricing, and headless setup while reconfirming the existing --max-snapshot-bytes behavior: config and CLI flag support, flag precedence over config, and the built-in 500 MiB default without implying a cap on worktree object stores or Bazel caches.

v0.10.0-rc.4 - 2026-07-14

This candidate makes dataset builds safer to operate by bounding temporary work and cache lifecycles to Stet-owned roots while preserving intentional Go module cache reuse. It also makes runtime fingerprints authoritative and fail closed on ambiguous standard probes so trial evidence stays trustworthy across platforms.

Fixed

  • Bound worktree-agent, verifier, and scratch/cache lifecycles to Stet-owned roots with fail-closed admission and cleanup receipts that distinguish logical deletion, allocated blocks, and observed filesystem recovery while preserving GOMODCACHE reuse (f5411acf, 696cfbd5)
  • Stabilize runtime fingerprints with authoritative versions and normalized Windows executable paths; fail closed on missing or ambiguous standard probes while preserving custom diagnostics (c78bfa22)

v0.10.0-rc.3 - 2026-07-14

Internal

  • Provision pinned pytest in fresh GitHub Actions release runners so release-surface checks execute before publication (d1ced74c)

v0.10.0-rc.2 - 2026-07-14

This candidate adds cost-aware GPT-5.6 model support, headless setup sessions, and reusable reference-build inputs. It also makes artifact storage and execution authority more explicit so operators can bound disk use and keep task evidence tied to the correct sources.

Added

Changed

  • Share Git authorities, portable source packs, and Harbor base images across manifest tasks to reduce repeated setup work (bc84292c, 0de53704, 9c2f4109, 4905d484)
  • Account for managed bytes across roots and report unique-base storage exposure for admission planning (942f2996, c46bee0a)
  • Reconcile storage budgets before retiring replay artifacts and apply dependency-aware retirement semantics (c7c07574, 74a222fb, 27767fbe)

Fixed

v0.10.0-rc.1 - 2026-07-11

This candidate makes large Bazel-backed task corpora faster and more reliable by reusing per-task caches, failing fast on non-progressing flake loops, and preserving durable verifier evidence for repair. It also hardens task selection and gate parsing, isolates worktree agents from source-repo rules, and moves Stet's source and public CLI distribution to their org-owned repositories.

Added

  • Add Bazel hygiene checks, per-task action caches, install-generation caching, and cached h2h verification for worktree dataset builds (f67f5815, f3c05669, c5b10717)
  • Add manifest-mode --target-ready N as an exact ready-floor cohort control with fail-closed shortfall receipts, plus first-class Claude rules treatments with activation evidence and provider preflights (daceffc7)

Changed

  • Route active source and CLI distribution references through org-owned Stet-AI/stet and Stet-AI/stet-cli repositories while preserving installer and updater overrides (12a22c0e)

Fixed

  • Preserve durable verifier receipts and structural selector evidence, clean Bazel scratch state robustly, and stabilize dataset regeneration and scenario execution (c310b406, 79dff515, bf789d0e)
  • Bind output-root registration and compare cleanup to pinned, symlink-resistant filesystem authority (daceffc7)
  • Isolate worktree agent execution from source-repository rules so the evaluated harness surface stays explicit (1e07f20e)
  • Repair three Bazel selector false-denial paths and accept Markdown-emphasized gate verdicts and scores (9b563215, 3ea45a08)

Internal

v0.9.1 - 2026-07-10

Makes the fixed 500 MiB compressed repo-snapshot cap operator-configurable so larger monorepos can build without patching Stet, and records per-base snapshot evidence in the build summary.

Added

  • Add --max-snapshot-bytes N and build.max_snapshot_bytes (flag over config over the built-in 500 MiB default) to raise the per-base compressed repo-snapshot cap, plus a snapshot block in build-summary.json reporting the cap, its source, and per-base accepted/oversize measurements. Oversize-snapshot attrition now carries --max-snapshot-bytes-specific remediation guidance in the build summary and rules-skill next actions.

v0.9.0 - 2026-07-10

Promotes the fully verified v0.9.0-rc.2 candidate to stable with no product behavior changes. Stet 0.9 makes agentic grading decision-authoritative across verification contracts, confined binary and RewardKit judges, task decisions, and trial reports; expands Docker-free worktree execution and repairable build receipts; and ships the same asset set, source traceability, and installer behavior proven by RC2.

v0.9.0-rc.2 - 2026-07-10

This candidate makes Stet's agentic grading path substantially more decision-authoritative: verification contracts now drive confined binary, pointwise, and pairwise judges into task decisions and trial reports, while build and validation receipts preserve more of the evidence needed to repair failed runs. It also expands Docker-free worktree execution and operator-facing model-comparison surfaces, while deliberately excluding the incomplete WS4 Bazel cache work before tagging.

Added

Changed

  • Reduce default build flake reruns from three to two and polish the leaderboard landing and comparison guidance (97492a8f, 174bd7cb)

Fixed

  • Keep RewardKit grading fail-closed and reproducible by invalidating stale pairwise evidence, binding workspace criteria, wiring synthesis and seeds, preserving legacy runner provenance, routing Claude through the agent judge, enforcing judge timeouts, reliably applying agent patches, and blinding the judge (0b40caab, f6dcc5a5, ee09ad1e, f9b1e802, d67395d3, 110522dd, 2d2fed14, c6aae863)
  • Preserve evidence-bearing PASS tasks and discover manifests, accept natural grader phrasing, retry exhausted provider-schema calls, surface repairable invalid patches and multi-arm direction, and correct mechanical-cleanliness, parser, citation, truncation, and persisted-review recovery (6296340a, 8c10eb0c, 33a1b5f6, 662ba5b6, 58ac7492, 948ef38a, 66762b4c, e1adf9d7, d877a754)
  • Harden worktree, build, and harness evidence by completing integrity contracts, bounding discover work, proving Bazel F2P labels, forwarding batch context exclusions, and applying Codex overlay instructions and skills (65a4bab1, 0503633f, 1a1fc15d, ed99b575, 5bbb3dd5, 6e8f4eab)
  • Make Harbor and agent startup more reliable by aligning the published agent pin, raising agent timeouts, failing fast on Cursor bootstrap no-signal runs, and tolerating concurrent Docker-cleanup races (05396474, 78646513, 2ba1f51b, c70ead21)
  • Restore first-run quality-grader guidance and correct the dogfood brief boundary (618e8c70, 4f0b0a09)

Removed

Internal

v0.9.0-rc.1 - 2026-07-06

Adds an interpretation layer to Stet receipts so coding agents get a trusted, performance-first read of an eval instead of relaying a bare posture token, calibrated confidence so strong evidence can promote without over-hedging, and a pinned grader identity so decision-grade evidence is reproducible. Operators also get per-arm harness levers (Claude Code plugin overlays), native Windows builds, and unified disk reclaim.

Added

  • Surface a machine-readable interpretation block on eval receipts: a preflight brief for eval rules plan that forbids performance claims pre-run and frames inspect/invalid/blocked as fixable states with the charged next action, plus a performance read for eval report that leads with how the candidate made the agent perform on real tasks — per-grader quality tallies, gold-test 3-state flips, credited rescue lanes, and token direction — with evidence confidence demoted to a secondary honesty rail (3476fe71, 079cab46)
  • Calibrate confidence on eval compares: direction-aware P(superiority) on every bootstrap metric, gold-correctness Beta-Binomial on task flips, a calibrated strong-correctness bypass that lifts inspect to promote (P≥0.95 + 0 regressions, or quality-dim P≥0.95 with majority support) while structural blockers still force inspect, and a promote-with-caveat path allowing one isolated regression at P≥0.90; honesty floor preserved on evidence_quality and claim entries (079cab46)
  • Pin grader identity for decision-grade evidence: compile-time grader bundle sha256, judge prompt-template digest, and reducer version freezable via stet graders promote, with stet graders status|show classifying the running binary against the pin and compare execution refusing under drift before arm spend; candidate/scratch/drifted builds always force inspect so they can never yield decision-grade claims; with no pin, behavior is byte-identical to today (e073a019)
  • Add a per-arm plugin_overlay run-config lever that activates a pre-baked Claude Code plugin/hook/MCP HOME snapshot (e.g. Caveman, Ponytail, RTK, Context Mode) with a fail-fast activation assertion, in both the worktree and Harbor/Docker backends, via stet eval run --plugin-overlay <dir>; a missing or failed activation records a typed plugin_not_active failure and drops the patch without suppressing real agent failures (b75026c1)
  • Add Windows build support: cross-compile and package stet.exe as a zip asset, an install.ps1 for Windows install, and a self-update swap helper since Windows cannot overwrite a running executable in place, plus a windows-latest CI job that builds, installs, and runs stet.exe (26e8c9dd)
  • Surface trajectory/behavioral metrics in multi-arm compare: per-arm standings, pairwise deltas, and win/loss/tie for tool_calls, shell_calls, trajectory_length, and patch_rewrite_ratio, plus new bounded time_to_first_edit and new_files_added metrics, with eval_report.behavior now populated (55d13685)
  • Extend variance-reduction estimator blocks (per-pair raw-delta + Wilcoxon + CUPED with gold-patch-size covariate) to the continuous quality dimension graders on stet eval compare --multi-arm, matching the two-arm path so dimension comparisons are statistically tighter; strictly additive and categorical graders stay on McNemar (163a7e3a)
  • Render a per-arm "Behavioral recall" panel in stet eval output and eval_report.v1.json, showing the strict precise-gold-fn pass-rate as a high-precision lower bound against the recall-supported rate, with a credited / inconclusive / non_solve breakdown; stays silent on clean test-backed corpora (9f059a31)
  • Add claude-sonnet-5 to the model-name registry (alias "sonnet 5") and the h2h pricing table (3eb46f7c)
  • Show grader-level and per-task interpretation summaries in the HTML report, sourced from each grader's interpretation object (9d0531cd)
  • Add --rebuild-builtin-prompts regrade mode that re-renders built-in equivalence/code-review grader prompts from the current embedded template instead of re-grading the stored prompt, with a retained-patch fallback for locating agent patches (8fa249a8)
  • Add a graded_equivalence derived 0-4 score grader projected from the binary equivalence payload, surfacing per-grader scores and code-review rubric scores on task detail, plus stet runs regrade-graders --rederive-only to backfill derived artifacts from validation.json with zero LLM calls; back-compat-preserving (excluded from grader-profile fingerprints, compares admit it only when a persisted decision metric exists on both arms) (358d3592)
  • Add stet graders surface alongside the grader-identity pinning: stet graders promote|status|show for pinning and classifying the live grader profile (e073a019)
  • Surface selector-proof coverage in eval output and named eval-study receipts for tracked comparisons (75bf293a, 40f939dc)
  • Add explicit quality posture selection to stet init (6cc90580)

Changed

  • Interpretation rewrite: confidence tier (strong/likely/flat) with odds-language ("19-in-20"), a composed VP-ready one-liner, a headline that leads with effect-size and always names any regression, and must_convey/must_not_claim obligations that enforce plain, jargon-free wording; all agent-facing strings de-jargoned (079cab46)
  • Unify disk reclaim behind one Reclaimer engine: root scratch/patch, the Harbor export cache, and Docker/Harbor daemon objects now share one dry-run/JSON/byte-accounting contract via stet artifacts compact --include-docker, with no change to what any existing engine deletes or when it deletes automatically (ad9c9a50)
  • Auto-resolve the evaluator provider and pin the requested model on the evaluator subprocess via --ai-model-id, and inject Claude OAuth env into the evaluator and preflight shell commands so the evaluator invokes the intended model rather than the provider default (9c061817, e180a03c)
  • Require non-interactive stet init (--yes or non-TTY) to fail fast and write no config when a runnable provider is detected, requiring an explicit --ai-provider; eval rules plan now reports the full enforced quality grader panel instead of letting a legacy evaluator-only profile suppress it (a8926d3e)
  • Switch the Harbor eval image to python:3.12-slim + uv, replacing the hand-rolled terminal-bench Ubuntu image (53d4746b)
  • Pin the harbor CLI bundled by Stet to 0.17.1 (was 0.8.0) (069da7ac)
  • Tighten review and equivalence grading: add a ship-decision rubric and verifier-contradiction guard to the review grader, require strict obligation-aware equivalence parsing with a malformed-output retry loop, and tighten the craft/discipline grader rubrics for scoped-ownership and near-equal-footprint judgments (63a66a3b, 28382292, 3c78e2d2)

Fixed

  • Make stet dataset regenerate-f2p idempotent and align rules fixtures with the digest gate so repeat runs don't re-derive clean artifacts (888c69dd)
  • Detect subscription/rolling usage caps (e.g. z.ai 5-hour "Usage limit reached") in the quota classifier so quota-killed tasks are recorded as hard-runtime rate-limit failures and engage the resume path, instead of being recorded as honest no-patch results that bias validity and skip on --resume (e55376c7)
  • Recover usage-limit-masked cells and partial-arm Harbor resume so interrupted compares don't silently drop arms (e4166afd)
  • Resolve --grader-ai-model-id independently on the experiment validate path and forward the Claude setup-token to remaining evaluator/grader clients so invoked-model provenance is correct (0857afb5, 92a54d6a)
  • Recover rules reports across manifest aliases and derive repair-patches evaluator from the suite grader_ai_cmd so repaired runs grade with the intended model (49b11bee, 80aeded2)
  • Write repair-tests validation.json flags as a canonical object and accept legacy string-array flags in MatrixFlags decode (e977e40c, d6a55300)
  • Decouple --stitch-rerun merge-back from full validate+grade so real agent patches land in canonical runs per-cell best-effort, with a pending-stitch.v1.json breadcrumb and recovery command on unrecoverable merges (f969e137)
  • Generalize the run-root integrity guard so partial roots cannot read as "complete" and regrade-graders summary/report writes are atomic and stale-safe on partial failure (a492bacc, 8fc611d6)
  • Resolve agent.patch across run-root layouts in repair-patches (0bf5fb2e)
  • Stop forcing cargo offline when the install runs at RUN time, fixing Rust task builds (6f84e978)
  • Tolerate prose preamble in quality-lane assessor JSON and clamp over-deep rev-ranges on short-history repos during discover (bd8d425c, cef0254b)
  • Allow dataset-backed rules suites to omit selection.mode and give an honest weak-signal off-ramp for zero-yield onboarding instead of a confusing failure (57a9ada4, ff86a6f1)
  • Reuse build gold/F2P proof under identity match so equivalent rebuilds don't re-prove the feature (4fffea4d)
  • Converge human_patch_guard.py to a class-based source-fetch denial for the Harbor verifier (98dfd9e1)
  • Reuse proved selectors for rules replay validity and credit repaired targeted-F2P cells as discriminating test verdicts (f7955bdd, 2cf4c352)
  • Block partial instruction datasets, tiny config-diff slices, and instruction bootstrap shortcuts during onboarding, and enforce the onboarding task floor for instruction rules so low-quality datasets can't slip through (c6529d4c, c805053c, f80c7d9e, cd9693f3)
  • Make onboarding honest and directional: surface directional inspect signal, actionable partial-dataset repairs, onboarding receipt blockers, test-relevance-mismatch warnings with lifecycle guidance, and guide wrapper repos to their implementation roots (8a264f91, 0310b58f, 6e79eeb3, 637a6e3c, 2cdffdd8)
  • Recognize locked uv pytest selectors, dependency groups, transparent test wrappers, pest test selection, and valueless flags, and retain F2P selector/rejection evidence during build (5832b008, 6fcfcf81, 8283b7dd, 8af2c25d, 75ec9322, 43493c52, f965a9e5, 2a3e352f, 0d0df83d)
  • Fail closed on unbuildable install-config synthesis and on partial suite roots; classify targeted runtime unknowns as infra and separate executor/infra runtime failures from dataset failures (44b5da1d, 9e96eb78, 59c7dacc, 82280491)
  • Resolve test commands into inspectable command plans for targeted F2P (npm/pnpm/yarn/bun aliases, safe && chains), with provenance recorded, so new language onboarding yields ready tasks instead of zero-yield builds (d8b82caf)
  • Preserve configured test commands in suite build, bootstrap uv starter harnesses and task environments, and improve python uv onboarding defaults (5c8f60ad, 918bd7d0, 1a0841a8, 214d31ba)
  • Make --stitch-rerun merge-back honor recipe tests for onboarding builds and keep tests.failure_mode coherent after test repair (978876f2, d81ef68c)
  • Stop pinning node and toolchains in the generated harness, and install Go in the generated onboarding harness (a6e27868, 07261287, c1bfc18d)
  • Surface grader claim readiness and missing-equivalence repair guidance in status; treat post-write grader gaps as inspect, clear stale no_test_signal exclusions on repaired cells, and auto-repair fresh grader coverage gaps while keeping recommended quality on instruction graders (bbb9c839, 0d01a422, 59296b07, 2009491d, e0bc10b0, f2f10108)
  • Bundle history from shallow clones, report dev-source commits, and loosen the default install allowlist (638ed9be, 6a92b85c, 3e319224)

Removed

  • Delete the pre-product Python F2P selector scripts (fix_f2p_test_selectors.py, write_f2p_targeted.py, verify_f2p_targeted.py) now that function-level F2P selection lives entirely in the product via stet dataset regenerate-f2p (9c472648)

Internal

  • Add build-time BuildKit task-dependency cache mounts for Harbor image bakes, replacing the broken run-time bind-mount cache that never populated; on by default, opt out with STET_TASK_DEP_CACHE=off, with reclaim via stet harbor cleanup --prune-buildkit ([07d69145], 7ef559f2, 5c5fb536)
  • Dedup the Harbor export cache via a content-addressed blob store (files ≥1 MiB stored once and hardlinked) and cut eval-run I/O churn via snapshot excludes and copy-on-write export copy (f48b0a6c, b99f3cc6)
  • Land grader-discrimination calibration internals and evaluator OAuth env plumbing (e180a03c)
  • Collapse six repair/rerun verbs (--stitch-rerun, repair-patches, revalidate-tests, repair-tests, repair-ai-coverage, regrade-graders) onto one invariant-preserving run-mutation substrate so atomicity, breadcrumbing, derived-artifact regeneration, scratch GC, and the integrity check are defined once behind a commit chokepoint; behavior-preserving with no change to verb flags or behavior (715dbbdd)
  • Refresh onboarding prompts, agent docs, and rules-skill-loop guidance, and rename craft/discipline graders to quality graders in skill docs (99d45afa, 9874c882, 133bcfab, 0967129c, 8395ebca)

v0.8.0 - 2026-06-24

Adds a worktree-native Harbor backend for stet eval runs, a deterministic targeted-F2P dataset build path (per-language fail-to-pass extractors, install-config recipes, Rust onboarding), receipt-first resumable eval combine, native multi-arm compare, and stet eval calibrate for grader-discrimination policy. Operators get more isolated and reproducible eval execution, recoverable combine/relaunch flows, and validity gates that close equivalence fail-open and contamination bypasses so Trial Results stay trustworthy.

Added

  • Build targeted fail-to-pass datasets deterministically: stet dataset regenerate-f2p, stet suite build --install-config consuming committed install-config recipes, per-language F2P extractors (Go/Rust/JVM and more) with a candidate ladder plus proof-aware P2P narrowing, dynamic proof of the targeted feature, and end-to-end F2P verification against the committed corpus (d78641e7, 29cc7206, d6be690b, 9b47a131, 5c00adac, 9cc8cef6, 86d76217, 80c35e2a, 3bfedb68)
  • Onboard Rust toolchains and pre-fill workspace test selection in stet init, with a test_selection config override and workspace auto-detect (29cc7206, 333b93f7)
  • Add a worktree-native Harbor backend for eval execution: a worktree lifecycle primitive, Harbor-shaped agent runner and verifier roles, worktree-native validation staging, runtime provenance, and execution-integrity guardrails, selectable via --harbor-backend worktree on stet eval rules and stet eval smoke (cd75e79d, 1a816a71, b8d701d4, f09ddbda, d4280b0c, b86994b5, b8111584, 4f90703a, 4533afcb, 10813c32, 04626fc1, f0cd6413)
  • Make stet eval combine receipt-first, incremental, and resumable via --finalize (68ff1607)
  • Add --relaunch-arm to stet eval rules resume so operators can recover a single wiped-out arm without rerunning the whole comparison (2a5e8756)
  • Report native multi-arm compares with per-arm quality-metric statistics (4c47d851, 5517f6d3)
  • Add stet eval calibrate to measure grader discrimination and gate eval decisions on the calibrated policy (0ae3bbae, 0ab2122e)
  • Add a stet runs repair-patches flow to repair runs from stranded or missing agent patches (6ec65fc2)

Changed

  • Make per-cell verdicts tests-first so an equivalence judge no longer fails open on test-bearing corpora, and gate equivalence-only test-verdict corpora explicitly (ab36df6c, 66945125)
  • Flag agent-answer contamination on a graded cell instead of blocking grading outright, keeping the Trial Result while marking the provenance risk (125c4a38)
  • Surface stranded agent patches in the run instead of silently dropping them, so recoverable work is visible to repair flows (c4b030b4)
  • Mark superseded trial directories after a repair so the canonical outcome is unambiguous (a3fb0fc6)
  • Update shipped skill guidance for dataset onboarding and for the worktree and repair rules workflows (daaadc01, eaa8f6f5)

Fixed

  • Make targeted-F2P build robust: scope gradle valueless-flag handling to the JVM/gradle path, force-add ignored files into the synthetic base so the test-file revert is faithful, install a no-op solve for corpus tasks lacking solution.sh, ensure verdicts measure the gold feature, and fix guarded toolchain-root inference (f4208d11, 89dfbc27, 23c8dc81, 7cc1f98c, 90405960)
  • Harden worktree and Harbor eval execution: fix worktree eval parity and Composer evidence, harden Harbor export migration and the Go toolchain environment, and allow current worktree paths through integrity scans (ce0a3636, ff731ad8, 99ada10e, 5a4bcc80)
  • Repair grader-coverage recovery, verifier retest repair-outcome projection, H2H repair-outcome surfaces, and the Harbor non-verdict projection so repaired and verdict-less outcomes report correctly (682771d6, bb3318f6, 5e017239, ffd4e4d9)
  • Exclude infra-verifier failures from quality scoring so toolchain/infra noise does not score against the model (3cb809f0)
  • Fix provider-schema classification of unsupported flags so unsupported levers are reported rather than silently mishandled (4247d5d3)
  • Improve stet artifacts doctor repo-cleanup UX (4bdc5b0f)
  • Serialize concurrent H2H progress emits to avoid interleaved progress output (75b75ac6)

v0.7.0 - 2026-06-15

Promotes Stet's optimize loop from evidence sidecar to public release workflow. Operators can now plan task slices, launch guarded optimization branches, inspect loop/frontier/decision receipts, narrow existing eval roots, and reclaim scratch artifacts safely while Stet keeps promotion, Harbor isolation, and Trial Results fail-closed.

Added

  • Add native optimize-loop artifacts, workbench views, trajectory scans, rejected-lever tracking, and uncertainty-aware decisions so operators can inspect what changed, why a candidate advanced or stopped, and whether evidence is ready (ff48d44f, 199384eb, 7ec24c18, e6f63feb, 230b9e10, 99a70928)
  • Add guarded optimize launch and frontier selection receipts so promotion evidence records candidate identity, selected harness surface, and the launch basis (9d134184, 820b7e4c, d7cd1911)
  • Add objective policy profiles and measuring-device grader profiles so optimization decisions are scoped to the intended quality target and required grader coverage (0051c5dd, f836938c)
  • Add native task-slice planning and explicit lane suites so operators can split evals into named, replayable task slices (dfe9bfec, 8cec3ec3)
  • Add default garbage collection for regeneratable per-run scratch, plus opt-in stet artifacts compact --include-datasets reclaim for materialized datasets (3cc2930c, 5355fecd)
  • Register Claude Fable 5 pricing and model aliases for h2h runs (99a7ee8a)

Changed

  • Support existing-root task selectors so compare/report workflows can narrow durable eval roots without rerunning or mutating source roots (95e1e5ff)
  • Harden dataset onboarding around selector evidence, Bazel proof, LLM diagnostics, generalized readiness, and descriptive validation modes (2d38b7f8, 943a42ed, 8ca9d3e6, 6457e2d9, 4604ec75, 25e23b2b)
  • Require replay tasks to carry repo.repository_slug metadata, with dataset builds deriving it from canonical change-request remotes (96ab61ea)
  • Show the eval status reportability tree so operators can see why a run is or is not reportable (1d292a08)

Fixed

  • Keep optimize promotion fail-closed across holdout readiness, lane awareness, suite identity, candidate identity, decision-subject validity, evidence handling, fixed candidate context, launch status, and child-lane control-plane paths (df58c11b, e3c538ee, 46be6415, 14e7fb6a, 62052fbc, f2f984d7, 5b91124b, fba1208d, 6b64909e, c6e36f10)
  • Preserve Trial Result and report integrity across runtime token evidence, profile policy projections, provider-schema judging, repair profiles, grader stat refreshes, eval-rules terminal state, summary-only compare status, and hardened orchestration (4cf9878e, ec86be5c, c4e881cc, b63bf95d, 06f2d8aa, 7ef85bd8, 4ef77abb, 4aded968, d203bce6)
  • Fail closed on invalid eval inputs and incompatible roots, including empty eval patches, frozen-baseline compatibility, overlapping combine compare sources, explicit lane-suite replay gates, and claim readiness (a1311943, 996bb7b3, 313d2e42, 2711dd78, 47d63508)
  • Close Harbor and contamination bypasses by gating network-contamination waivers, rendering prompt templates across auth agents, defaulting exports to runtime internet, isolating Go runtime behavior, closing guard source bypasses, and hardening Fable answer-contamination guards (a4035e87, e9f26502, a6a0bc38, 1a40ebbf, 389d7f46, d9bfd909)
  • Improve artifact reclaim safety, retired Harbor cache cleanup, and trial-result read caching during artifact regeneration (8a214153, b2f0022a, 156b6e4f)

v0.6.0 - 2026-06-02

Prepares the next release around decision-grade eval evidence, safer task materialization, and more complete operator workflows. This release adds decision-grade receipts, richer h2h report diagnostics, Cursor-backed Harbor evals, matched A/A/B workbench planning, self-serve trial paths, and stricter Harbor/build behavior so operators can understand why an eval promoted, held, or failed closed.

Added

  • Add Cursor CLI support for Harbor-backed evals, including model resolution, pricing metadata, auth handling, behavior telemetry, and post-run hook support (47786a51, ed9eb49f, 24bf67f4)
  • Add Opus 4.8 model resolution and pricing metadata so new-model eval reports can price runs accurately (43e1127f)
  • Add provider-native structured grader runtimes with explicit evaluator config, runtime provenance, and fail-closed custom-grader validation (13495052, edae64fd)
  • Add smoke policy receipts and persisted skill-loop proposed edits so probe/gate outcomes carry more actionable trial-result evidence (57159cee, cd7f9bf5)
  • Add h2h report diagnostics for benchmark economics, process behavior, missing equivalence evidence, patch reliability, and default compare statistics (c346edb7, 86a7cf86, 94d5086d, f21f268c, cec492f5, bb98fc7a, 77b75968)
  • Add multi-arm standing and variance-reduction analysis for leaderboard and model-comparison reads (c6e7fa87, 81aca262)
  • Add compare cost-attribution diagnostics and a no-spend matched A/A/B workbench planner for optimizer decisions over existing Trial Results (ada9a178, eac3419b)
  • Support combining split eval roots, including repaired-verifier evidence via stet runs repair-tests, so operators can finish n=20+n=5 benchmark runs without losing provenance (b1d85108, 576c4868, f3893aef)
  • Narrow eligible broad verifier commands during build, with deterministic coverage checks and LLM abstain/retry handling so generated task suites avoid overly broad test runs (112838fa, dd1054e7, a0f39cd1, de656c3f, 36a840dc, 5c9c5317, 1bc5f290)
  • Add the self-serve CLI signup flow and extend the default commercial trial window to 21 days (4b5843a4, 8e50d415)
  • Publish the Codex agents iteration post, composer-comparison visualization, and Opus 4.8 launch comparison visualization for the leaderboard site (c60f6188, 71b6fb81, eac3419b)

Changed

  • Pin the Harbor invocation Stet uses and refresh the shipped operator docs around invocation provenance, release lifecycle, compare/check-in, rubric authoring, and dist install behavior (bac3f2ec)
  • Update shipped Stet guidance so operator-facing next steps, review-agent waiting, and dataset-build instructions match the current CLI surface (ca1c6e84, d3cd9477)
  • Clarify build-time verifier narrowing and dataset guidance, including the GraphQL Go Tools rev range used by leaderboard dogfood runs (cd4c66fe, 41abe228)

Fixed

  • Fail closed for incomparable cost deltas, noise-aware recommendations, missing instruction evidence, repaired grader decisions, grader timeouts, asymmetric grader coverage, unknown equivalence, active-run status, frozen-baseline staleness, and decision-grade cost evidence (b81016be, cd76bbb6, 9a9bf65f, 9bd742a1, 04b5bcfc, e1268e43, 0e03a876, bc6b23ea, d7b2e8c8, 1f1f03f2)
  • Preserve cache artifact cost provenance and canonical patch-presence semantics, attribute skill-activation evidence to the candidate arm, normalize behavior telemetry across reports, exclude scratch files from compare staging, clean up partial arms, and invalidate no-patch trial evidence (c26fc24a, d6504750, c7de2548, 68fdade2, 37567d9d, dab771c8)
  • Disable Harbor task network access by default, freeze Go toolchains from runtime_version, require Go runtime metadata for toolchain injection, and preserve test.patch trees during gold/F2P validation (5e4667b1, 17ebb4fa, cd85a5e4, 53410be4)
  • Make eval dataset discovery explicit for Harbor-backed runs, drop harness preambles from assembled instructions, and route rules validation through stet eval agent so candidate and validation flows bind the intended task corpus and canonical command surface (9d922b16, b549f55f, b4c9182a)
  • Add an explicit operator-reviewed waiver path for known-benign agent network contamination while preserving fail-closed default behavior and waiver audit trails (eac3419b)
  • Increase Harbor agent timeout budgets for long-running eval tasks (fc537baf)
  • Remove the blocked self-install CTA from private trial surfaces and improve leaderboard SEO/AI crawl assets (cc1a0005, a7b36f43)

Internal

v0.5.0 - 2026-05-23

Expands Stet's operator surface for customer trials and multi-provider change requests. This release adds GitLab change-request support, auto-update plumbing for installed CLIs, replay-valid suite selection, declarative frozen-baseline reuse in change manifests, and tighter report/plan diagnostics so eval outcomes fail closed instead of drifting into ambiguous states.

Added

  • Add GitLab change-request providers and thread provider identity through dataset and discovery flows (9de6d051, 8c0f1fac, a80b63c0)
  • Add CLI auto-update support for installed Stet binaries, including runtime identity and dist-update routing (380fd72c)
  • Generate ai_task for rev-range and base-head build modes so generated task corpora carry executable task prose (36e5b5f5)
  • Add replay-valid suite selection for rules evals so operators can select tasks whose replay evidence is trustworthy (8bb784d0)
  • Add declarative frozen-baseline reuse in change manifests for rules evals (e9ab7984)
  • Add posture-aware h2h evidence framing for capability-release reads (STET-397) (73ce7573)
  • Add leaderboard AI crawl and private-eval FAQ schema surfaces for search and answer-engine ingestion (e3fd6c91, 8a3055e1)

Changed

  • Publish beta/customer Stet guide updates and tighten shipped Stet agent guidance for report interpretation and post-run learning capture (721ebd3f, 50fa0884, 95325017)
  • Clarify independent grader-model cost controls, completed compare adequacy caching, disk cleanup boundaries, and GitLab support closeout docs (4aa8485d, b6e280b6, 2ab39e06, c084fa79)

Fixed

  • Detect repaired compare evidence in manifest-backed status and report flows (6a0bd5a5)
  • Add mixed grader-profile diagnostics so compare/report output exposes profile disagreements instead of hiding them in aggregate state (26cde721)
  • Preserve smoke-reuse behavior telemetry and staged-skill activation during skill_diff evals (4c600bd3, 6040da19)
  • Tune validation patch guardrails without weakening release-gate semantics (7ef15ca3)
  • Stabilize custom grader metric ordering and preserve custom graders when refreshing regrade decisions (a44f7883, f354b569)
  • Retry transient replay-validity failures so temporary verifier noise does not become durable eval state (c879f140)
  • Block GitLab merge-request human-patch access in Harbor candidate runs (cc6e81d8)
  • Preserve rules compare report binding and surface Harbor validator disagreement provenance (b2e3e382, 3a1ec6ca)
  • Fail closed when stet eval rules plan sees unresolved baseline context (1d382037)
  • Reconcile terminal compare state after finalized eval runs (5b563f26)

v0.5.0-rc.1 - 2026-05-23

Expands Stet's operator surface for customer trials and multi-provider change requests. This release candidate adds GitLab change-request support, auto-update plumbing for installed CLIs, replay-valid suite selection, declarative frozen-baseline reuse in change manifests, and tighter report/plan diagnostics so eval outcomes fail closed instead of drifting into ambiguous states.

Added

  • Add GitLab change-request providers and thread provider identity through dataset and discovery flows (9de6d051, 8c0f1fac, a80b63c0)
  • Add CLI auto-update support for installed Stet binaries, including runtime identity and dist-update routing (380fd72c)
  • Generate ai_task for rev-range and base-head build modes so generated task corpora carry executable task prose (36e5b5f5)
  • Add replay-valid suite selection for rules evals so operators can select tasks whose replay evidence is trustworthy (8bb784d0)
  • Add declarative frozen-baseline reuse in change manifests for rules evals (e9ab7984)
  • Add posture-aware h2h evidence framing for capability-release reads (STET-397) (73ce7573)
  • Add leaderboard AI crawl and private-eval FAQ schema surfaces for search and answer-engine ingestion (e3fd6c91, 8a3055e1)

Changed

  • Publish beta/customer Stet guide updates and tighten shipped Stet agent guidance for report interpretation and post-run learning capture (721ebd3f, 50fa0884, 95325017)
  • Clarify independent grader-model cost controls, completed compare adequacy caching, disk cleanup boundaries, and GitLab support closeout docs (4aa8485d, b6e280b6, 2ab39e06, c084fa79)

Fixed

  • Detect repaired compare evidence in manifest-backed status and report flows (6a0bd5a5)
  • Add mixed grader-profile diagnostics so compare/report output exposes profile disagreements instead of hiding them in aggregate state (26cde721)
  • Preserve smoke-reuse behavior telemetry and staged-skill activation during skill_diff evals (4c600bd3, 6040da19)
  • Tune validation patch guardrails without weakening release-gate semantics (7ef15ca3)
  • Stabilize custom grader metric ordering and preserve custom graders when refreshing regrade decisions (a44f7883, f354b569)
  • Retry transient replay-validity failures so temporary verifier noise does not become durable eval state (c879f140)
  • Block GitLab merge-request human-patch access in Harbor candidate runs (cc6e81d8)
  • Preserve rules compare report binding and surface Harbor validator disagreement provenance (b2e3e382, 3a1ec6ca)
  • Fail closed when stet eval rules plan sees unresolved baseline context (1d382037)
  • Reconcile terminal compare state after finalized eval runs (5b563f26)

v0.4.3 - 2026-05-19

Hardens Harbor eval isolation for candidate agents. Exported Harbor datasets no longer keep candidate-visible gold patches in fresh or cached task bundles, and the human-patch guard now blocks GitHub pull refs passed through direct git commands, config environment, and included config files.

Fixed

  • Prevent Harbor dataset exports and cache hits from leaving gold/reference patches visible to candidate agents (f05f41c9)
  • Block GitHub pull refs in the human-patch guard so candidate agents cannot fetch target PR heads or merges directly (3dad80bf)
  • Inspect Git config, config-env, and include paths for hidden GitHub pull refs before allowing guarded git operations (693709b1)

v0.4.2 - 2026-05-18

Blocks Harbor candidate agents from reading public human-patch artifacts during evals. Codex and Claude Code runs now install a guard that prevents direct PR diff/patch, GitHub API, raw head-commit, and downloaded PR-patch access while leaving ordinary package and source lookup available.

Fixed

  • Block public GitHub PR and head-commit artifact access in Harbor Codex and Claude Code agents so candidate evals cannot inspect the target human solution (7774bac6)

v0.4.1 - 2026-05-18

Prevents Harbor patch capture from retaining harness and gold artifacts as agent output. Operators get cleaner trial results: .stet/gold.patch, guidance files, generated directories, and lockfile-only churn no longer leak into captured agent patches, while real source edits and deletions remain visible.

Fixed

  • Prevent gold patch and other harness-generated paths from leaking into captured agent patches; rewrite legacy Harbor artifact sources to the sanitized canonical patch while preserving real source edits and deletions (4d68d6a)

v0.4.0 - 2026-05-18

Hardens the correctness of cached evidence with a frozen-baseline harness-surface digest gate, adds a stet eval rules repair recovery command for interrupted compares, attaches paired-bootstrap confidence intervals and a headline-uncertainty envelope to compare receipts, and reports impl-vs-test-fixture patch surface composition on footprint_risk. The CLI gains --grader on stet eval run, deterministic --task-order-seed propagation through stet eval rules and monitor reruns, and judge-noise regrade seeding; replay-validity output surfaces typed gold-failure summaries. Adds a Claude Code hook harness surface so hook treatments are first-class compare variants, a validation_failure.kind subtype taxonomy that prevents setup blockers from reading as model no-patch behavior, and Linux ARM64 release assets. Returns stet eval status in ~2s on finalized compares, restores --out dataset reuse in the stet eval rules skill wrapper, and corrects the shipped skill docs around decision_receipt.recommendation and the --grader-ai-cmd / --grader-ai-model-id fallback.

Added

  • Publish Linux ARM64 (linux/arm64, including aarch64 hosts) CLI release assets and support them in install/update (9866fe4)
  • Accept repeatable --grader <id|bundle|rubric.yaml> on stet eval run with explicit-wins-merge over repo quality config, mirroring the flag on stet eval compare and stet runs regrade-graders (de6c0bc)
  • Add stet eval rules repair to reuse validation artifacts and resume an interrupted compare or rerun missing/partial arms; stet eval rules resume remains accepted as a compatibility alias (c64393c)
  • Plumb --task-order-seed through stet eval rules and the rules skill wrapper so dispatch order replays deterministically against a sorted task selection (f3e9ee5)
  • Honor suite-manifest eval.task_order_seed end-to-end through the stet eval rules compare path (6870494)
  • Persist task_order_seed in monitor rerun config so stet monitor reproduces the original dispatch order (6e07411)
  • Add a paired-bootstrap post-pass to stet eval compare with --bootstrap-iterations, --bootstrap-seed, --ci-level, and --no-bootstrap; receipts gain aggregate.<metric>.uncertainty blocks (baseline_ci, candidate_ci, delta_ci, win_loss_tie, bootstrap) and an Uncertainty: text section (3449973)
  • Carry per-metric uncertainty intervals into decision_receipt, including decision_receipt.headline_uncertainty for the headline metric's CI envelope (f72da01)
  • Report patch surface composition on footprint_risk results with a new surface_breakdown block (agent vs gold, implementation vs test_fixture sides, test/fixture/expected_output subkinds, and test_fixture_added_share); per-task summaries expose footprint_surface_breakdown (5c5a181)
  • Add --seeds N to stet runs regrade-graders so the operator can bound judge-noise variance during regrades (ece1165)
  • Add a durable arm identity contract so frozen-baseline reuse and arm-level evidence stay bound to a stable identifier across replays (5f671b3)
  • Print typed gold-failure summaries (category/reason plus harbor_log path and scrubbed excerpt) in replay-validity terminal output so it matches the JSON diagnostic (a731873)
  • Add a Claude Code hook harness surface — propagate hook-derived signals end-to-end through the rules-runtime artifact, eval-rules check-in, resume, status, runner runtime, and experiment spec so hook treatments are first-class h2h compare variants (174a94e6)
  • Add a no-patch validation_failure.kind subtype taxonomy (empty_patch, setup, pre_agent, verifier, sanitized_patch); propagate counts through h2h task summaries, reports, eval status, smoke preflight, and run validity; classify Harbor no-agent-start artifacts as setup blockers; prefer invalidating subtypes on smoke-preflight tie-breaks while preserving legacy matrix_status values for existing consumers (4939dd3f)

Changed

  • Update the shipped Stet skill docs to point at decision_receipt.recommendation as the verdict field (mirrored by lifecycle.decision); decision_receipt has no top-level decision field (d1374a76)
  • Document --grader-ai-cmd / --grader-ai-model-id as the read-only fixture fallback for LLM-backed graders on stet eval rules plan / launch / skill, and warn that --no-quality only drops auto-bundled craft/discipline graders — the default equivalence and code_review graders remain LLM-backed and still require an evaluator (d1374a76)

Fixed

  • Gate frozen-baseline reuse on a harness_surface.baseline_digest; cache hits whose surface no longer matches the active harness fall back to cache_status=unknown rather than replaying stale evidence (9776d82)
  • Include the implicit task list in the stet eval rules cache key so cache hits/misses match the realized task set (36ec2c3)
  • Majority-vote non-scored regrade samples when computing aggregate regrade outcomes (cb77959)
  • Harden replay-validity task identity so per-task gold-replay records bind to a stable identity (4028f8a)
  • Close arm identity QA gaps surfaced against the durable-identity contract (c5caca6)
  • Make frozen-baseline trial materialization selective and per-task, preserving trajectory artifacts and avoiding unnecessary copies (e0d20b9, f29a6db, 5f0a2fc)
  • Document the new compare bootstrap flags in stet eval compare --help (246ee48)
  • Honor --out dataset reuse in the stet eval rules skill wrapper by short-circuiting the rev_range_buildability preflight when dataset/build-summary.json already exists and --restart is not set; the reuse decision is logged to stderr so --plan does not silently mask a divergent --rev-range (d1374a76)
  • Return stet eval status in ~2s on finalized compares by reading the persisted eval_report.v1.json sample-adequacy instead of walking .stet/{eval-rules,leaderboard,archive,baselines}; the cache binds to the requested compare root and rechecks adequacy inputs for freshness so fail-closed behavior is preserved (STET-387) (8f439d49)

v0.3.1 - 2026-05-15

Enrichment runs natively in Go end-to-end; the Python enrich_dataset.py scaffolding is gone and prompts ship without XML fences.

Changed

  • Port enrich_dataset.py to Go and drop XML scaffolding from enrichment prompts (5bb132b)

v0.3.0 - 2026-05-15

Adds operator launch receipts, opus reasoning-curve evidence on the leaderboard, and tightens prompt-shape provenance with fail-closed enforcement when ai_task is missing. Smoke preflight is now bypassable for fast iteration, eval status terses on completion, and h2h gains typed grader-failure counters and small-sample directional reads.

Added

  • Surface opus reasoning-curve evidence on the leaderboard (5e691ba)
  • Shift ai_task prose to imperative goal-first phrasing during enrichment (8812b1f)
  • Record prompt-shape provenance during build and fail closed when ai_task is missing (7c941bd)
  • Add --prompt-shape to stet build with self-contained-natural as the default (2ef2130)
  • Expose --skip-smoke-preflight on stet eval run for fast candidate iteration (84214b2)
  • Terse stet eval status on complete and add eval report --paths (4ae50c7)
  • Recognize smoke-preflight runs in the frozen-baseline compare flow (98df234)
  • Polish plan JSON shape and emit an rc!=0 next-step hint from stet eval rules (b7d3eed)
  • Persist smoke-seeded task provenance on the runner runtime artifact (64653d1)
  • Add rules study holdout lanes (cbccb27)
  • Add operator launch receipts to the CLI (210ee15)
  • Add Mandarin blog translations to the leaderboard (dc16c7d)
  • Publish the opus reasoning-curve writeup on the leaderboard (16f961d)
  • Surface typed grader-failure counters on stet eval report, status, and the decision receipt (STET-312) (6e03355)

Changed

  • Reframe the stet-dist README as agent-first and add an onboarding quickstart (2c02fa7)
  • Collapse TaskMetaConfig literals and unify prompt-shape resolution across build paths (b77f10c)
  • Unify task-instruction assembly across all build paths (4b1dcb6)

Fixed

  • Hoist grader-evaluator preflight and gate activity_state on resolved backend (bebf8e1)
  • Account for smoke preflight provenance in compare math (c9711e7)
  • Preserve tracked .stet contents after repo-bundle bootstrap (5c8499c)
  • Expose limited directional reads for small-sample h2h comparisons (f217cb0)
  • Preflight rules replay validity (9ef099c)

Internal

  • Refresh dist collateral for v0.3.0 (cbcab16)
  • Drop the tessl MCP server config from codex (5645ee8)
  • Polish ONBOARDING.md voice and content for stet-dist (20751fa)
  • Include ONBOARDING.md in the dist sync CI step (7cd553c)
  • Add ONBOARDING.md explainer for human readers in stet-dist (1e3eab9)
  • Document why terse eval status points at the decision report instead of --paths (8bbd214)
  • Reframe the AGENTS guide and refresh design-stet plus QA scenario coverage (a1e5557)
  • Polish opus post presentation and sync copy from the vault (e49a181, 445d8e7)
  • Explain replay-invalid rules slices in the Stet skill docs (8c624e7)

v0.2.0 - 2026-05-11

Hardens the stet eval rules flow end-to-end with grader-profile persistence, preflight checks, and surfaced provenance; introduces the GPT-5.5 reasoning-curve leaderboard post and pairwise order-swap judging for custom-grader compares; adds activity-state disambiguation, sample-adequacy reporting, and a stet-qa skill harness for black-box testing the shipped Stet docs. Many small fixes tighten artifact integrity (non-regular file rejection, symlink-escape guards) across the harness.

Added

  • Add --grader-ai-cmd / --grader-ai-model-id wrapper flags to the eval-rules skill (STET-338) (9b8c370)
  • Surface agent-side exceptions in the last_error receipt (STET-332) (f96da10)
  • Improve Stet SEO surfaces on the leaderboard (7a18f7f)
  • Add sticky TOC to the GPT-5.5 reasoning-curve post (540dcc4)
  • Publish the GPT-5.5 reasoning-curve post and add it to model comparisons (5f890e5, 91c8fa4)
  • Preserve explicit --mode through stitch and repair (9cda0af)
  • Support optional requested graders and explicit repair evaluators (9ee0208)
  • Centralize structured custom-grader calls (b9ced48)
  • Persist and reuse eval-rules grader profiles, with planning and reported provenance (893a8df, dec49c3, 8b377af, ac48552)
  • Add safe stale Docker cleanup to harbor (fdfc2f5)
  • Add eval-rules grading controls (2a5ee65)
  • Persist task outcome history across runs (8fc26e1)
  • Surface eval sample adequacy (ecd2b60)
  • Add order-swap pairwise per-task judging for custom-grader compares (1d6cbdb)
  • Add the stet-linear-tpm skill for Symphony-ready ticket shaping (d31901f)
  • Add max as a first-class reasoning-effort tier (7d47898)
  • Make evals quota-aware (7309948)
  • Accept an explicit agent for stet eval smoke (37a7c19)
  • Add a default randomized H2H task order (f9e8320)
  • Bake claude-code and codex into the harbor image (e766928)
  • Route Harbor Codex through codex-lb (498a132)
  • Register gpt-5.5 with pricing and alias (97bbfbf)
  • Add stet-qa scenarios for claude_md, docs_glob, model_update, and skill_diff (98bd7cb)
  • Migrate leaderboard datasets to Stet v1 and rewrite ingestion (1a06b2a)
  • Bake install_config into per-task Dockerfile and skip node lifecycle scripts during baked install (1af7972, 8d86967)
  • Surface trajectory-derived behavior metrics in eval reports (0b08059)
  • Add artifact-retention compaction (6d2cb0e)
  • Add --skip-smoke-preflight to bypass the candidate smoke gate (eval-rules) (0d446c8)
  • Add Opus 4.7 vs 4.6 zod blog item to the leaderboard RSS feed (b8478f8)
  • Add the agents-md-preflight QA scenario and shared 1-task fixture (fdc0f61, 4819136)
  • Add the validate-change-manifest QA scenario and fixture (b6c477d, daaddce)
  • Add stet-qa SKILL.md scaffolding (skeleton, preamble, report template, fixtures reference, design spec) (20f7980, 83d0517, 4314d57, 7b29eb8, 8ebc50d)
  • Add the stet-qa implementation plan (ccea8fb)
  • Register eval_rules_plan as a commercial command (1e7f44e)
  • Publish the Opus 4.7 vs Opus 4.6 Zod writeup with cost and token detail (cf26998, 77fc7d0)
  • Add the GPT-5.5 vs Opus 4.7 blog post and blog SEO scaffolding (2d9e7da)
  • Add an agent setup snippet to /private (cf925a1)

Changed

  • Surface artifact-discrimination diagnostic for any custom-grader compare (36b923f)
  • Honor model-specific AI agents (d4f33a7)
  • Refine GitHub doc-only path classification (277508b)
  • Bound GRADE_CUSTOM rubric grader calls with a per-call timeout (0471faa)
  • Stabilize custom-grader no-patch scoring (8276a9d)
  • Update site navigation links and refresh the homepage funnel (67d401a, a9ab39c)
  • Update the GPT-5.5 vs Opus blog post and methodology data (8b8d276, dbe4cf4)
  • Overlay current conventions in materialized tasks (33d8150)
  • Improve SEO pages for model comparisons (a6c36e8)
  • Separate grader AI config in eval flows (a45858f)
  • Sort custom-rubric prompt criteria deterministically (3900957)
  • Convert cramped Opus 4.7 tables to bar charts and merge review/discipline into one rubric chart (1bab40e, 54d58ed)
  • Refresh validate-change-manifest dogfood with post-fix run (8a40b88)
  • Stop default Claude Keychain auth lookup (36eaadc)
  • Reduce Vercel compute for inspect routes (b50c773)

Fixed

  • Add activity_state to disambiguate in-flight versus terminal eval status (STET-333) (83b7ba9)
  • Tighten plan/manifest receipts and skill text (rules) (STET-329) (709295f)
  • Reject non-regular pinned dependencies, repo tarballs, validation agent patches, config files, test inference files, score patches, and unsafe cost-usage artifacts and output-root manifests; block git-internals context paths (ef30ea8, 4c35c65, bedad3a, 7202b8a, c812fc2, a59e904, e79efe0, 186e25a, 160263b)
  • Refresh stale grader coverage from arm summaries (3611c8c)
  • Repair short/non-hex citations, lift nested findings, and add skip-reason-specific refusal text (eval-rules) (f0e01bc)
  • Repair rules QA follow-up flows (d756cb5)
  • Treat nested smoke evidence as pending-compare arm evidence (d5781c8)
  • Share Claude envelope unwrap and refuse empty skill datasets (eval-rules) (826668f)
  • Unwrap Claude --output-format json envelope before grader parse (165f5eb)
  • Classify wrong-repo as repo_not_a_git_repository and fix QA fixture repo paths (ed1d4fc)
  • Wire grader-ai flags, classify default-branch launch_error, and document --tasks coupling (3ae2acb)
  • Surface pre-arm failures via launch_error and fix baseline path lookup (400959e)
  • Preflight rev-range buildability and document --rev-range (STET-331) (4b9611c)
  • Preflight grader provenance in eval-rules-plan (STET-330) (20a1c9c)
  • Preflight --out dataset and surface build errors in the eval-rules skill (ab85acf)
  • Preflight repo-managed skills root symlinks (STET-327) (b90aac8)
  • Refuse eval rules launch when LLM grader credentials are missing (STET-324) (99fc652)
  • Stop recommending resume on terminal rules-arm failure (STET-325) (5c37fb4)
  • Resume resolves treatment paths against the suite repo root (01433d1)
  • Preserve grader and cost evidence on stitch (55c0cbb)
  • Recover rules resume evidence (9370f05)
  • Preserve regrade economics in summaries (25f5edb)
  • Bound frozen-baseline materialization (cc44270)
  • Expose eval report reasoning effort (896df69)
  • Recover trailing custom-grader JSON (8785d0e)
  • Resolve nested sample-adequacy history roots (dab87b6)
  • Detect mixed arm provenance (c813057)
  • Surface rules compare arm status (73cfd44)
  • Preserve frozen-baseline custom graders (ab5c0b7)
  • Reclaim artifacts/agent.patch duplicates left by harbor (fc8cba9)
  • Avoid bare 429 quota classification (9b9b467)
  • Copied-patch path classification (d6688b0)
  • Surface repairable custom-grader parse failures (778891f)
  • Enforce grader-evaluator provenance (db0d690)
  • Classify executor failure tail output (c32f3f8)
  • Enforce clean guidance overlays (c3c8de7)
  • Target-pass commit overscan (0759526)
  • Improve blog text contrast (ee1772e)
  • Remove visible SEO duplication from blog posts (512e9b7)
  • Harden review-retry prompt parse errors (62045c5)
  • Repair embedded prompt source loading (577ecea)
  • Reject weekly doctor reports symlink escapes (966ea35)
  • Fix unsafe run-ID artifact lookup (5fc50b8)
  • Fix markdown-bold kv value parsing (38bd4e3)
  • Reject non-executable generated test commands (08737b5)
  • Fix Codex assistant trajectory capture (ab00275)
  • Reject symlinked validation roots (a19a4a1)
  • Reject non-regular gate prompt sources (6419f9a)
  • Validate manifest build numeric flags (1410ac1)
  • Ignore external diff for split patches (c2f2beb)
  • Exclude failed tasks from the weekly denominator gate (d25e61a)
  • Preserve UTF-8 in review-retry prompt (6e4a324)
  • Preserve eval-rules plan launch flags (f6db705)
  • Use per-task cost and time in the GPT-5.5 post (00b470c)
  • Fix frontend audit and Turbopack warnings (00164b2)
  • Prevent convention overlays from polluting agent patches (96d4090)
  • Fix eval-rules runtime evidence locator (f87d834)
  • Fix grader coverage reporting (53079cd)
  • Recover cache tokens when rollup omits them and sum Claude assistant turns (489ab31)
  • Merge experiment report when arms record different dataset paths but identical task slices (423d581)
  • Smoke-preflight liveness classification (0ba1f41)
  • Smoke-gate canonical harbor pass (9cd6a4b)
  • Honor gitignore in harbor patch capture (edc58f0)
  • Preserve evaluator model provenance in grader artifacts (8a0d747)
  • Avoid copying run artifacts during compare staging (e12bcfb)
  • Improve rules-skill wrapper recovery (a43d462)
  • Gate Vercel tracing config so local dev resolves tailwindcss (5f9f977)

Internal

  • Forbid bespoke leaderboard eval shortcuts in agent docs (942ac71)
  • Refine smoke-preflight provenance and terminal no-patch coverage (5f07a75)
  • Assert h2h repo root by source marker, not basename (f56a5a6)
  • Stabilize stet qa grader preflight (5006a18)
  • Migrate agents-md-preflight from sonnet-4.6 to gpt-5.4 (stet-qa) (32131fe)
  • Pin rev-range-buildability seeded_from to STET-331 fix SHA (316efae)
  • Sync dist skill snapshot (718a961, 2520373, 54c1383)
  • Roll up failure-path skill drift findings (STET-337) (8180be2)
  • Sync GPT-5.5 reasoning-curve post copy with vault edits (5aa49ab)
  • Ignore claude worktrees, generated leaderboard typings, local launchers, and scheduler lock (da09121, fcf22a1)
  • Reuse command normalization for task defaults (e9a1c19)
  • Share config value resolution (7cb5f5d)
  • Centralize relevance sorted unique strings (068040c)
  • Centralize Claude auth env lists (b3952ac)
  • Share workbench baseline record loading (7de8135)
  • Centralize path containment checks (8017b74)
  • Share safe path joining (f0537a0)
  • Filter weekly denominator membership (6d5f68c)
  • Preserve task grader rubric scores (608a249)
  • Share local-inspect file traversal (547133e)
  • Clarify leaderboard dogfood eval policy (5200eb4)
  • Refactor leaderboard navigation links (219ac3e)
  • Refactor aicmd explicit option handling (79809a0)
  • Simplify diff path marker parsing (81d33ac)
  • Cover Next config env behavior (9c5c168)
  • Trigger Vercel deployment (823b32c)
  • Record stet-qa dogfood-03 gap-closure verification (7d133fe)
  • Address stet-qa dogfood-02 unclear-receipt gap (c7eadd9)
  • Link the stet-qa skill from the Progressive Disclosure Index (22dbd82)
  • Record stet-qa dogfood: parallel dispatch synthesis (f338599)
  • Record stet-qa dogfood: validate-change-manifest (61bf64f)
  • Rename stet-qa report file to qa-report.md (033f03d)
  • Cover --pairwise cap and document flags in compare help (9b5991d)
  • Update archived zod evidence paths (274f59c)
  • Add the flu82 graphql xhigh launcher (778f3b9)
  • Merge origin/main and merge the GPT-5.5 reasoning-curve blog (18b7614, 903f126, 2c12471)

v0.1.0 - 2026-04-17

Initial productized release of the Stet CLI. Stet measures whether an AI coding change is safe to ship, covering the full capability lifecycle: probe and workbench for iterative improvement, gate for promote/hold/rollback, and monitor for scheduled regression detection. v0.1.0 lands the public command surface (stet build, stet eval, stet eval rules, stet baseline, stet monitor, stet workbench, stet auth), the canonical eval_report.v1.json trial result, a working harbor-backed Docker harness for replayable real-repo task corpora, the Next.js leaderboard frontend, and the stet-cli distribution channel for pip-style install.

Added

  • Public, productized CLI surface (stet build, stet eval, stet eval rules, stet eval workbench, stet eval batch-grade, stet eval calibrate, stet baseline, stet monitor, stet workbench, stet auth) with versioned model names, gauge-style help output, and a public manifest contract (e7eef57, 8047f55, 7f7dcef, 11c8b74, c3a26a4, 8fbd054, 251f33d, 086b59b, 932d07f, bbe1764, 066e700, f3e243b)
  • Replayable task corpus pipeline: rev-range discovery, materialization, schema-aligned task bundles, repo-managed install configs, and harbor-backed per-task Docker harness for real-repo evaluation (30d26fe, 0d3cdd4, 5a8763e, 29f4bea, 0ff45d1, 5ef34df)
  • stet eval validation pipeline with AI-powered offline scoring, equivalence obligations, gate pipeline, and footprint-risk classification (ab4471a, 3cc728e, d4b65b0)
  • Canonical eval_report.v1.json trial result and HTML report rendered alongside it as a sibling artifact (STET-244) (7685e31, c59037f)
  • Custom YAML rubric graders, scored rubrics on a 0-4 scale (and float-scored 0.0-4.0), compare-gate lifecycle, and stet eval calibrate for adversarial rubric calibration (786cbdd, b742af7, d4d0c4d, 1feb8ee, 066e700)
  • Head-to-head (h2h) flow with native cutover, instruction treatments, frozen-baseline compare, suite-driven runs, eval-rules skill loop, and capability release tracking via stet eval batch-grade (STET-187, STET-166) (d4b65b0, 2389f34, c057f69, 1525697, 79773c1, f3e243b)
  • Stet skill packaged in dist and made install-first-class, with hypothesis-driven iteration guidance, comparison workflows, baseline-freeze teaching, and stet-cli (benredmond/stet-cli) as the public distribution channel (ce5840a, 9e004ae, 15d6d7a, 4fa2394, 723a8f6, a7706a4)
  • Leaderboard Next.js frontend with model comparisons, blog posts (Opus 4.7 vs 4.6 Zod), inspect evidence preview, and editorial copy (280efb6, d7eb93c, 4033439)
  • Reasoning-effort eval arms and Claude Opus 4.7 support with priced model registration (7ed6f77, 3daf2cb, feb6691)
  • stet monitor status and stet monitor run for scheduled regression detection (STET-176, STET-178) (086b59b, 932d07f)
  • Workbench mutation-command gating and risk surfacing for the candidate iteration loop (STET-171) (b14df5f, bbe1764)

Changed

  • Rename project from Flux to Stet across the codebase, CLI surface, dist artifacts, and public copy (160c91b)
  • Make decision-quality graders the default for stet eval rules (77cde43)
  • Gate commercial Stet workflows by entitlement (9bcf523)
  • Reorient the Stet skill to be optimizer-facing (agent-first) (4e317f6)

Fixed

  • Numerous harness, gate-parsing, grader, leaderboard, baseline, and credential fixes accumulated over the v0.1.0 rcs (auth scoping, OAuth, Harbor harness install skew, smoke preflight, grader repair retry, baseline subset filtering, h2h credential env leaks, decision report wiring)

Internal

  • Migrate task plans into apex/tasks/ domain layout, add the dist skill mirror sync flow, repo-managed pre-commit hooks, and the QMD-backed task corpus (af6b5d9, 7c7b197)
  • Cache Harbor harness CLI setup, expire stale caches, and rename tb args to harbor args during the harbor cutover (915d87a, e2dfc3a, af91737)