Tools for evaluating coding agents on your own repo

Products for testing coding agents on your own code

Checked 2026-10-20. “Not published” means the product's own pages don't say.

Comparison

Product	Tasks from	Grading	Runs on	Start	Pricing
Stet	Your merged PRs	Tests, checklist, review	Hosted or your machine	Invite only today	Not published
Details: Stet Tasks About 40 of your merged PRs How it grades Correctness (original tests, or equivalence to the merged change); a checklist of required behaviors written without seeing any agent's patch; a code review; cost and time per task What it can vary Model, reasoning level and instruction files, on Claude Code, Codex or Cursor Who runs it Stet, in a hosted sandbox (Hosted); you, on your machine (Local) Your code Hosted copies one repo into an isolated sandbox. Local keeps code, tests and logins on your machine. No SOC 2 report or DPA yet. How you start Invite only today. Free first trial, set up by Stet; runs on your Claude, Codex or Cursor login or API keys Pricing Not published; first trial free Sourceshow it works security
Managed services
Superconductor Benchmark	PRs you pick	LLM judges	Vendor cloud	Free sign-up	Published
Details: Superconductor Benchmark Tasks PRs you pick; an AI infers the spec from each How it grades Several LLM judges from different providers score correctness, completeness and code quality (1–5); runtime and cost tracked; optional QA agent What it can vary Agent and model (Claude Code, Codex, Amp, Factory Droid, Grok Build, OpenCode, Pi, Cursor), 1–5 implementations per agent; example results list reasoning levels; instruction files not stated Who runs it Superconductor; each agent in its own cloud dev environment Your code Cloud sandboxes (Modal or Runloop, AWS US); Enterprise can run sandboxes in your own cloud. Working toward SOC 2 Type 2. How you start Sign up free, or a free benchmark set up on a 30-minute call (limited spots); your own subscriptions or API keys Pricing Free $0; Pro $100/month includes Custom SWE-Bench; Enterprise custom Sourcesbenchmark docs pricing security
Vals-Smith	Your merged PRs	Hidden tests	Vendor-run	Free credits to start	Credits; not published
Details: Vals-Smith Tasks Merged PRs, each with an issue statement, hidden tests and a reproducible setup; hidden tests must fail before the fix and pass after How it grades Hidden tests pass with no regressions; score is the % of tasks resolved What it can vary Models, coding agents and the harness; reasoning level and instruction files not stated Who runs it Vals; sign in with GitHub and pick a public or private repo Your code Whether code can stay on your machine: not stated. The page says your code never leaves your control. Trust center lists SOC 2 and GDPR. How you start Sign in with GitHub and pick a repo; 120 free credits to get started (launch post); or book a call, or apply as a design partner Pricing Free credits to start (120 at launch), then paid credits under the terms' paid-credit model; credit prices not published Sourcesproduct public benchmarks launch post terms trust center
Fullbeam	Your merged PRs	PR's held-back tests	Vendor cloud (EU)	Private preview	Not published
Details: Fullbeam Tasks Merged PRs, from the commit before the change; an AI-written spec you approve; must fail before and pass after How it grades The PR's own held-back tests, in a separate sandbox with no model keys; pass rate, cost and regressions; “Unknown” instead of zero What it can vary Codex runs: model, instructions, skills or Codex config, via a PR. Codex vs Claude Code: instructions and skills only, not models. Reasoning level not stated. Who runs it Fullbeam runs the benchmarks Your code No; short-lived sandboxes on Daytona's EU target. Not SOC 2 or ISO 27001 certified; DPA agreed for Enterprise. How you start Private preview, by invitation; request access Pricing No public price yet Sourcesproduct docs pricing
Faros Time Machine	Merged PRs, work items	AI judge, checklist	Vendor workspaces	Book a demo	Not published
Details: Faros Time Machine Tasks Merged pull requests and work items, replayed from the codebase as it was before the change How it grades An AI evaluator scores each result blind against a weighted checklist of the task's requirements; cost, quality and runtime What it can vary Model, harness, reasoning level, available skills, routing rules, context and configuration Who runs it Faros, in isolated workspaces away from production Your code The page says your code never leaves your boundary How you start Book a demo; bring a public or private repo Pricing Not published Sourcesproduct pricing
VulcanBench Teams	Your merged PRs	Hidden tests	Vendor engagement	Email	From mid four figures
Details: VulcanBench Teams Tasks Your merged PRs, or matched equivalents where a PR doesn't convert; hidden tests fail before and pass after How it grades Hidden tests in a network-isolated Docker sandbox, no LLM judge; pass@1, pass^k, tokens, cost and wall-clock time; delivered as a routing policy What it can vary Models and effort levels; runs through Claude Code, Cursor or an internal agent Who runs it VulcanBench builds and runs it as an engagement; the suite, tests and harness are yours to rerun Your code Built against a copy of your repos under your security team's access arrangement; never published or reused; usually runs on your API keys How you start Email them about your stack Pricing A focused engagement starts in the mid four figures Sourcesservice site
Warp Factory Benchmarks	Past runs, your tasks	LLM judge	Warp	Early access	Not published
Details: Warp Factory Benchmarks Tasks Past Factory runs replayed, or tasks you write, each with a prompt and correctness criteria How it grades An LLM judge on correctness, quality, efficiency, verbosity, cost and your own criteria; repeated runs; recommendations and routing rules What it can vary Model (frontier and open-weight), harness (Warp, Claude Code, Codex), factory configuration such as skills and prompts, runners Who runs it Warp Your code Not published How you start Request early access; needs a Warp factory Pricing Not published Sourcesproduct docs
Tessl	Your PRs or commits	LLM judge, rubric	Vendor sandbox	Free sign-up	Published
Details: Tessl Tasks Scenarios generated from your PRs or commits, each a task brief plus a scoring rubric; also from skills How it grades An LLM judge scores each solution against the scenario's weighted rubric, with and without your context; cost, time and tokens per solve What it can vary Agent (Claude, Codex or Tessl's own), model, skill or plugin versions, context files Who runs it Tessl, in an eval sandbox Your code Enterprise offers single-tenant or self-hosted deployment and your own LLM How you start CLI; sign up free Pricing Free $0 (1,000 credits a month); Team $100/month; Enterprise custom Sourcesevals docs pricing
Sigmabench	Your feature commits	Changed-file overlap	Vendor-run	Book a call	Not published
Details: Sigmabench Tasks Your codebase, with the same benchmark as the public leaderboard, which builds tasks from feature commits with LLM-written prompts How it grades Overlap between the files the agent changed and the files the original commit changed (Jaccard); accuracy, consistency and speed What it can vary Coding agent and model pairs (CLI agents); public runs use each harness's default reasoning level Who runs it Sigmabench Your code The site says benchmarks are read-only and SOC 2-compliant How you start Book a benchmark call Pricing Not published Sourcessite methodology
Open source and self-run
CodeProbe	Merged PRs, commits	PR tests or oracle	Your machine	Install the CLI	Free, open source
Details: CodeProbe Tasks Mined from merged PRs and commits How it grades Tests the original PR touched, an oracle answer, or both; file and symbol overlap; cost, tokens and latency What it can vary Agent (Claude Code; Copilot in preview; Codex unsupported), model, preamble, MCP and tool surface, timeouts, turn limits; reasoning level not stated Who runs it You; open-source CLI from Sourcegraph (Apache-2.0) Your code Yes; runs locally, with agents in containers How you start uv tool install codeprobe Pricing Free, open source Sourcesrepo
Microsoft Scope	Tasks you define	LLM judge	Your infrastructure	Deploy it yourself	Free, open source
Details: Microsoft Scope Tasks Tasks you define, with an optional starting codebase How it grades An LLM judge against criteria you define What it can vary Agent (GitHub Copilot, Claude Code, VS Code with Copilot), model, agent version, skills, MCP servers, extensions, personas; reasoning level not stated Who runs it You; software you deploy, not a hosted service (MIT) Your code Yes; self-managed How you start Deploy on Kubernetes, or a local stack with Docker Compose Pricing Free, open source Sourcesannouncement repo
self-bench	Your merged PRs	Generated hidden tests	Hosted or self-host	Sign in with GitHub	Not published
Details: self-bench Tasks Merged PRs; an agent writes hidden tests and a reference solution, kept only if they fail without it, pass with the original, pass on rerun and pass review How it grades Hidden tests; accuracy against cost What it can vary Models, harnesses, reasoning settings, sandbox Who runs it Hosted app, or self-host (MIT) Your code Self-host option; models and sandboxes run on your org's own keys How you start Sign in with GitHub at app.selfbench.dev, or self-host Pricing Not published Sourcesrepo site
RepoAgentBench	Your merged PRs	The PR's own tests	Your machine	Install the CLI	Free, open source
Details: RepoAgentBench Tasks Merged PRs How it grades The PR's own tests, which must fail before and pass after; pass/fail What it can vary Agent (Claude Code, aider) and model through aider; Codex and Gemini CLI on the roadmap; reasoning level not stated Who runs it You; open-source CLI (MIT), early alpha Your code Yes; local-first, code never leaves your machine How you start pip install repoagentbench Pricing Free, open source Sourcesrepo
RepoGauge	Your bugfix commits	The fix's own tests	Your machine	Clone the repo	Not published
Details: RepoGauge Tasks Bugfix commits from your default branch, with the original fix and the tests it made pass How it grades The tests the human fix made pass, through the SWE-bench harness; pass rate, cost per solve and latency What it can vary Agent (Claude CLI, Codex CLI, opencode, SWE-agent, Aider, Cursor) and model Who runs it You, locally; hosted access by request Your code Yes; everything runs on your machine How you start Clone the repo, then uv run repogauge mine Pricing Not published Sourcessite repo
Supabase Evals	Supabase scenarios only	Scorer code per eval	Your machine	Clone the repo	Free, open source
Details: Supabase Evals Tasks Supabase scenarios (prompt, scorer, starting state), not your repo How it grades Scorer code per eval: builds, tests and SQL checks What it can vary Agent (Claude Code, Codex, OpenCode), runtime, model, MCP servers, skills Who runs it You; open source (Apache-2.0) Your code Yes; runs locally in Docker sandboxes How you start Clone the repo and pnpm install Pricing Free, open source Sourcesrepo results
Harbor	Tasks you write	Your test scripts	Your machine or cloud	Install the CLI	Not published
Details: Harbor Tasks Tasks you write (an instruction plus tests), or published datasets such as Terminal-Bench How it grades Each task's test script writes a reward; RewardKit adds LLM or agent judges What it can vary Agent (Claude Code, Codex, Cursor CLI, OpenCode and many more), model, reasoning effort, skills, MCP servers, sandbox Who runs it You (Apache-2.0); Hosted Harbor on Harbor Hub by access request Your code Yes; a Docker sandbox on your machine by default, cloud sandboxes optional How you start uv tool install harbor Pricing Not published Sourcessite docs repo
Margin Evals	Suites you write	Your test scripts	Your machine	Install the CLI	Free, open source
Details: Margin Evals Tasks Suites you write (a prompt and a test script per case), or published suites such as SWE-Bench-Pro How it grades Each case's test script (exit 0 passes); accuracy, tokens, runtime and full traces What it can vary Agent (Claude Code, Codex, OpenCode, Gemini CLI, Warp Code, Cursor, Pi), model, reasoning level, MCP servers, skills, instructions and prompts Who runs it You; a local eval runtime (AGPL-3.0) Your code Yes; containerized evals on your machine How you start Install the margin CLI Pricing Free, open source Sourcessite docs repo
BenchFlow	Tasks you write	Your verifiers	Your machine or cloud	Install the CLI	Free, open source
Details: BenchFlow Tasks Tasks you write (a task file plus a verifier), or existing benchmarks How it grades Each task's verifier writes a reward What it can vary Any ACP agent (Claude Code, Codex, OpenCode, Gemini), model, sandbox (Docker, Daytona, Modal); skill evals Who runs it You; open source (Apache-2.0) Your code Yes; Docker on your machine, or cloud sandboxes How you start uv tool install benchflow Pricing Free, open source Sourcesrepo
Coder Eval	YAML tasks you write	Weighted criteria	Your machine or CI	Install the CLI	Free, open source
Details: Coder Eval Tasks YAML tasks you write How it grades Weighted criteria scored 0–1, from file checks and code similarity to LLM-graded rubrics; cost What it can vary Agent (Claude Code, Codex, Antigravity, OpenCode, Pi), model, tools, prompts and skills, as A/B variants Who runs it You; open source from UiPath (Apache-2.0) Your code Yes; sandboxed, locally or in CI How you start pip install coder-eval Pricing Free, open source Sourcesrepo
Agentic Coding Harness and Benchmarks	Datasets you write	LLM judge, 0–100	Your machine	Clone the repo	Free, open source
Details: Agentic Coding Harness and Benchmarks Tasks A dataset you write: repo, pinned commit and problem statement; private repos supported How it grades An independent LLM judge scores each result 0–100; a cost/quality frontier across models What it can vary Harness (Claude Code, Codex, pi, oh-my-pi, kiro-cli, Strands), model, hosting path (Bedrock, LiteLLM, vLLM) Who runs it You; sample code (MIT-0), not meant for production use Your code Yes; the harness clones the repo into a temporary directory where you run it How you start Clone the repo and run its /benchmark skill or script Pricing Free, open source Sourcesrepo your own repo

Where others are stronger

Compliance: Vals lists SOC 2 and GDPR in its trust center, and Sigmabench says its benchmarks are SOC 2-compliant. Stet has no SOC 2 report or DPA yet.
Runs on your machine: Every tool in the open-source and self-run group runs on your own machine or infrastructure, and Tessl Enterprise can be self-hosted. Stet, including Local mode, is invite-only today.
Agents: Harbor integrates many more agents, including aider, OpenCode and Copilot CLI, and Superconductor also runs Amp, Factory Droid, Grok Build, OpenCode and Pi. Stet runs Claude Code, Codex and Cursor.
Your own tasks: Scope, Harbor, Margin Evals, BenchFlow and Coder Eval run tasks and graders you define. Stet builds its tasks from your merged PRs.

These don't compare coding agents on your code. They test a skill or plugin, check repo rules, turn a repo into tasks, run custom evaluations, or route each task to a model or harness.

Tool	Does	Works with	Runs on
Skills and plugins
Caliper	Skill with vs. without	Claude Code, Codex, +2	Your machine
Details: Caliper What it does Runs a skill's tasks with and without the skill and reports the difference, and whether the agent picked the skill; grades with an LLM judge, Python assertions and activation checks Works with Claude Code, Codex, Pi, Hermes Where it runs You; open source (MIT) Sourcesrepo
Claude Code plugin evals	Plugin with vs. without	Claude Code	Claude Code
Details: Claude Code plugin evals What it does Scores a plugin on test cases, repeats them without the plugin and reports the difference; graders check the reply with a regex, a tool call or an LLM-judged rubric Works with Claude Code Where it runs Claude Code, on your Claude plan or API bill Sourcesdocs
SkillOpt	Trains a skill file	Claude Code, Codex, +2	Your machine
Details: SkillOpt What it does Trains a skill file by running tasks, proposing edits and keeping them only when a validation set improves Works with Integration shells for Claude Code, Codex, Copilot and Devin Where it runs You; open source from Microsoft (MIT) Sourcesrepo
Rule checks
Tessl Verifiers	Checks repo rules	Not published	Local, pre-commit, CI
Details: Tessl Verifiers What it does Checks each agent change against your repo's rules, written in plain text; an LLM judge gives pass or fail per rule Works with Not published Where it runs Locally, in a pre-commit hook or in CI Sourcesannouncement
Task generators
Repo2RLEnv	Builds tasks from a repo	Any Harbor agent	Your machine or cloud
Details: Repo2RLEnv What it does Turns merged PRs, commit history, security advisories and more into Harbor tasks with verifiers; Tasksmith uses an agent to build a task from one merged PR Works with Any Harbor agent Where it runs You (Apache-2.0 and MIT); Tasksmith runs on Daytona or Modal Sourcesrepo docs
Custom evaluation services
Surge AI Enterprise	Custom evals, any domain	Your models and tools	Surge engagement
Details: Surge AI Enterprise What it does Builds evaluations around your workflows, quality standards and operating constraints with your teams; compares models on quality, cost, latency, reliability and failure modes; monitors production for regressions; its experts range from engineers to doctors and lawyers Works with Your models, tools, data and production workflows Where it runs An engagement with Surge's team; share pilot tasks for results within a week Sourcesenterprise
Model and harness routers
Not Diamond	Model router	Your harness and gateway	API, in your gateway
Details: Not Diamond What it does Predicts which model to use for each input, to cut cost while keeping accuracy; built for coding agents Works with Your existing harnesses and gateway Where it runs Recommendations come through an API and run in your gateway and harness; SOC 2 and ISO 27001 compliant Sourcessite
LiteLLM Auto Router	Model router	Not published	LiteLLM gateway
Details: LiteLLM Auto Router What it does Picks a model inside the LiteLLM gateway; an experimental mode estimates whether a cheaper model can complete the task Works with Not published Where it runs LiteLLM gateway; design partners can try it on production traffic Sourcesbenchmark post
Databricks Smart Routing	Model router (Beta)	Claude Code, Codex	Databricks gateway
Details: Databricks Smart Routing What it does Picks a model for each coding task by its complexity, at the start of the session; with Omnigent it also picks the harness Works with Claude Code and Codex Where it runs Databricks Unity Gateway (Beta) Sourcesannouncement
Omnigent	Agent meta-harness	Claude Code, Codex, Pi	Local or cloud sandbox
Details: Omnigent What it does Combines models, harnesses and techniques; switches between agents with one-line changes; policies for spend caps, model routing and risk-based escalation Works with Claude Code, Codex, Pi and your own agents Where it runs A local OS sandbox or a remote cloud sandbox; open source (Apache 2.0) Sourcessite

Public benchmarks

These rank models and agents on fixed public tasks, not on your repo.

Benchmark	Tasks	Compares	Grading
Artificial Analysis Coding Agent Index	3 benchmarks, 303 tasks	Agents, models, settings	Pass@1, 3 attempts
Details: Artificial Analysis Coding Agent Index Tasks DeepSWE v1.1 (113 tasks), Terminal-Bench 4.0 (66) and SWE-Atlas-QnA (124) What it compares Agents, models and execution settings; cost, tokens and time per task How it grades Binary pass/fail, pass@1 averaged over three attempts; equal weight per benchmark Sourcesindex
SWE-bench	Python GitHub issues	Models and agents	% resolved
Details: SWE-bench Tasks 2,294 real GitHub issues from 12 Python repos; Verified is a 500-task human-filtered subset What it compares Models and agents; the default Verified view runs every model in mini-SWE-agent How it grades % of task instances resolved Sourcesleaderboard
Terminal-Bench	Terminal-Bench 4.0 tasks	Model and agent pairs	Resolution rate
Details: Terminal-Bench Tasks Terminal-Bench 4.0 tasks, listed on Harbor Hub What it compares Model and agent pairs, with tokens and cost How it grades Resolution rate, with a ± range Sourcesleaderboard
FrontierCode	Maintainer-written tasks	Models and effort levels	Tests, rubrics, verifiers
Details: FrontierCode Tasks From Cognition. FrontierCode 1.1: 100 tasks in Main, 150 in Extended, crafted by the open-source maintainers of the source repos What it compares 46 models across reasoning efforts, with cost and output tokens How it grades Measures mergeability with unit tests, rubrics and verifiers; a weighted rubric score, zero if a blocking criterion fails; runs that used solution-bearing internet sources score zero Sourcesleaderboard announcement
DeepSWE	113 from-scratch tasks	Models and effort levels	Hand-written verifiers
Details: DeepSWE Tasks DeepSWE v1.1: 113 tasks written from scratch, not adapted from commits or PRs, across 91 repos and 5 languages What it compares Models and effort levels, all in mini-swe-agent; average cost per task How it grades Hand-written verifiers test behavior; pass@1 with a ± range Sourcesleaderboard
CursorBench	Real Cursor sessions	Models and effort levels	Agentic graders
Details: CursorBench Tasks CursorBench 4.0: tasks based on real sessions from Cursor's engineering team, with short task descriptions What it compares Models and effort levels, with cost, tokens and steps per task How it grades Agentic graders score each task Sourcesleaderboard announcement
Real-SWE	25 private-codebase tasks	17 models, native harness	Tests, pass@1 of 5 runs
Details: Real-SWE Tasks 25 tasks from licensed private codebases in TypeScript, Go, Python, Ruby and Java What it compares 17 models, each in its native harness How it grades Verifiers based on or taken from the codebases' own tests; resolution rate, pass@1 over 5 runs with a 95% confidence interval Sourcesbenchmark
Surge sudo L7	60 engineer-written tasks	Models in coding agents	Graders, expert criteria
Details: Surge sudo L7 Tasks 60 tasks written by engineers, mostly in private production repositories, plus open-source projects What it compares Models in coding agents, with cost per task How it grades Grader models score each task against expert-written criteria across eight dimensions, using tests and static analysis as evidence Sourcesleaderboard methodology
Voratiq leaderboard	Production codebase work	26 agents, head to head	Whose code gets merged
Details: Voratiq leaderboard Tasks Work in production codebases What it compares 26 agents, head to head How it grades Ratings from whose code gets merged Sourcesleaderboard
Margin trackers	50 SWE-Bench-Pro tasks	Claude Code and Codex	Daily pass rate, 95% CI
Details: Margin trackers Tasks A curated, contamination-resistant subset of SWE-Bench-Pro, 50 tasks a day What it compares Claude Code and Codex with their current top model, day by day How it grades Daily, weekly and monthly pass rates with 95% confidence intervals; flags statistically significant drops SourcesClaude Code tracker Codex tracker

Get a free trial or book a demo

stetsh

Tools for evaluating coding agents on your own repo

Comparison

Where others are stronger

Related tools

Public benchmarks