# 1. Prior art

Assessed 2026-07-26. Everything below was checked against live sources, not memory; where I
could not verify something I say so.

## Summary table

| Project | Category | Blocks you? | Steal |
|---|---|---|---|
| promptfoo | OSS eval CLI | No | Assertion vocabulary, YAML shape, provider abstraction, viewer |
| inspect-ai (UK AISI) | OSS eval framework | No | `--epochs`, bootstrap scoring, solver/scorer split, log format |
| Braintrust | Hosted eval platform | No (signup wall) | "Experiment" mental model, confidence indicators in UI |
| LangSmith | Tracing/observability | No | Nothing |
| SWE-bench harness | Benchmark runner | No | Task-instance schema, layered Docker caching, `--cache_level` |
| Anthropic `skill-creator` | First-party, built in | **Partially — watch it** | Executor/Grader/Comparator agent split, blind A/B, browser viewer |
| OpenAI Codex `exec` | Agent runtime | No (it's your substrate) | JSONL event schema |
| AgentLinter | Static CLAUDE.md linter | No — complementary | `npx`, zero-signup, 0–100 score framing |
| `probe-and-refine-tuning` | Research code | **No, but it's your method** | McNemar, mixed-effects models, the whole statistical design |
| ETH `AGENTbench` | Research benchmark | No | Task set, and your best marketing citation |

---

## promptfoo

**What it does.** Open-source LLM eval CLI. Declarative YAML: prompts × providers × test cases
with assertions. Ships a `Claude Agent SDK` provider, an OpenAI Codex SDK provider, an OpenCode
provider, and a documented ["Evaluate Coding Agents"](https://www.promptfoo.dev/docs/guides/evaluate-coding-agents/)
guide. Assertions include `contains-json`, `is-json`, arbitrary JavaScript, cost/latency
thresholds, `llm-rubric` (LLM-as-judge), and trace assertions over intermediate tool calls.
Sandboxed working directories, tool allow/deny lists, `--repeat N`.

**Why it doesn't solve this.** Verified from their own docs:

- **No statistics.** `--repeat 3` is documented as a way to observe variance. There is no
  aggregation across repeats, no confidence interval, no significance test. This is the exact
  gap [00-verdict.md](00-verdict.md) says is the whole product.
- **No config ablation.** No parameter sweep, no systematic config variation, nothing that
  parses a markdown file into sections.
- **No per-trial repo isolation.** Working dirs are relative to the config file; the guide
  emphasizes read-only sandboxes. A coding-agent eval where the agent *must* write to a repo
  and each trial needs a clean checkout is outside the model.
- The Claude Agent SDK provider **defaults to not loading `CLAUDE.md`** (`setting_sources`
  is opt-in). Your entire subject of study is a flag they had to add.

**Steal:** the assertion type names (users know them), the YAML file shape, the provider
abstraction boundary, and the local web viewer. Do not compete on breadth of providers.

**Blunt take:** promptfoo is a superset of your *plumbing* and has none of your *thesis*.
A reasonable person could ask why this isn't a promptfoo plugin. Answer honestly in your README:
because the unit of iteration is a repo checkout and a config file, not a prompt, and because
the statistics have to be first-class, not a `--repeat` flag.

## inspect-ai (UK AI Security Institute)

**What it does.** Serious Python eval framework: `Task` = dataset + solver(s) + scorer.
`--epochs` runs each sample multiple times and aggregates, explicitly to handle model
stochasticity and flaky pass/fail. Built-in statistical bootstrap, pass@k, model-graded QA,
custom metrics. 200+ pre-built evals in `inspect_evals`.

**Why it doesn't solve this.** It's a framework for evaluating *models*, not repo config. No
concept of a config file under test, no ablation, no git worktree management, no first-class
notion of "the same agent binary, two different configs, same repo." You would be building your
tool *inside* inspect-ai rather than using it.

**Steal:** `--epochs` is the right name and the right default-facing concept — use it or
mirror it (`--trials`). Steal the solver/scorer separation (it maps to runner/assertion) and
their eval-log format (structured, replayable, re-analyzable without re-running — exactly the
requirement in [ADR-0006](adr/0006-result-storage.md)).

**Should you build on it instead of from scratch?** Considered and rejected — see
[ADR-0001](adr/0001-language-and-distribution.md). Short version: it drags in a large dependency
and an abstraction (`Sample`/`Solver`) that fits poorly over "clean git checkout per trial," and
it costs you the sub-5-minute zero-config install.

## Braintrust

Eval-first commercial platform: datasets, scoring functions, experiment tracking, prompt
iteration, with statistical confidence indicators in the UI. Genuinely the closest thing to
"statistics done properly" in the commercial tier.

**Why it doesn't solve this.** Hosted; requires signup and sending your repo's eval results to
a third party. Your stated constraint is "usable in under five minutes with no signup." It is
also generic-LLM-shaped, not coding-agent-shaped: no repo checkouts, no config ablation.

**Steal:** the word "experiment" and the discipline of showing a confidence indicator next to
every headline number.

## LangSmith

Trace-first observability, strongest with LangGraph. Not an A/B harness, not repo-aware.
Nothing to take. Mention it in prior art only so readers know you looked.

## SWE-bench harness

Fully containerized evaluation harness (`swebench` on PyPI, `swebench.harness.run_evaluation`),
layered Docker images with `--cache_level none|base|env|instance`, plus `sb-cli` for hosted runs.

**Why it doesn't solve this.** It evaluates *patches against a fixed benchmark*. It has no notion
of agent config, no A/B, no ablation. It is also heavy: Docker images per instance.

**Steal:** the task-instance JSON schema (repo, base commit, patch, test patch, FAIL_TO_PASS /
PASS_TO_PASS) — that schema is nearly exactly your mined-task schema
([04-task-suite.md](04-task-suite.md)), and adopting its field names buys interoperability for
free. Also steal the *idea* of `--cache_level`: environment setup is the dominant non-LLM cost
and must be cached across trials.

## Anthropic `skill-creator` (first-party)

**What it does.** Built into Claude Code. Four modes — Create, **Eval**, Improve, **Benchmark** —
and four composable agents: Executor (runs the skill against eval prompts), Grader (scores
against expectations), **Comparator (blind A/B between skill versions)**, Analyzer (suggests
improvements). Runs prompts twice — with and without the skill — scores both against assertions,
and displays results in a browser viewer. Works on claude.ai as well as the CLI.

**Why it doesn't solve this — today.** It is scoped to `SKILL.md`, not `CLAUDE.md`/`AGENTS.md`.
It uses LLM-as-judge for the blind comparison, has no confidence intervals, no repeat-and-
aggregate statistics, no section-level ablation, and no cross-vendor support.

**This is your displacement risk, not your competitor.** The Comparator agent is 80% of an A/B
harness. Extending it to `CLAUDE.md` is a small product decision. Plan for it
([11-risks.md](11-risks.md)) and lean into the two things a first-party tool is structurally
unlikely to ship: (a) running the *other* vendor's agent on the same suite, (b) statistical
conservatism that tells users their change did nothing.

**Steal:** the four-agent decomposition is a genuinely good architecture for the *assertion*
layer, and the "run twice, once with and once without" framing is the clearest possible way to
explain the tool to a new user.

## OpenAI Codex `codex exec`

Your substrate rather than a competitor. Verified flags in
[03-architecture.md](03-architecture.md). No eval, A/B, or config-testing features.

**Steal:** its JSONL event schema (`thread.started`, `turn.started`, `item.started`,
`item.completed`, `turn.completed`) is the more expressive of the two agent event streams —
use it as the shape of your normalized internal trace format.

## AgentLinter

Free, MIT, `npx agentlinter`, 100% local. Static analysis of `CLAUDE.md`/`SOUL.md`/`TOOLS.md`
against 25+ rules (instruction counter, context bloat, progressive disclosure, anti-patterns),
scored 0–100 across eight dimensions, with auto-fix suggestions.

**Why it doesn't solve this.** It never runs an agent. It scores your config against
*documented best practices* — which is exactly the "intuition with no feedback loop" you are
trying to replace. It is the static counterpart to your empirical tool.

**Steal — and this is the highest-value steal in this document:** its *distribution*. `npx`,
no signup, no API key, runs in seconds. That is the bar for your first-run experience.
Also consider explicit interop: AgentLinter tells you which sections *look* like bloat;
`ablate` tells you which ones *measurably are*. A one-line "we complement AgentLinter" in the
README is worth more than pretending it doesn't exist.

## `asashepard/probe-and-refine-tuning` + arXiv 2606.20512

**What it does.** "Probe-and-Refine Tuning of Repository Guidance for Coding Agents." Iteratively
probes individual guidance lines to determine impact on agent task success, then removes or
rewrites the ones that don't contribute. Reports ~34% resolution vs. 23% baseline on SWE-bench.
Statistics: **McNemar's test** for paired outcomes plus **mixed-effects models**. MIT licensed,
code released.

**Why it doesn't block you.** Verified from the repo: Python 3.11+, SWE-bench 2.0.0+,
tree-sitter, Docker, and a self-hosted OpenAI-compatible inference endpoint; evaluated on
SWE-bench Verified only; tested with Qwen3.5-35B-A3B; 4 stars, 0 forks; documentation is oriented
at reproducing paper tables. It does not run on an arbitrary user's repo, does not support Claude
Code or Codex, and does not have a CLI a working dev would install.

**Steal: the entire statistical design.** Mixed-effects modelling of line presence → outcome is
the correct answer to "which lines earn their keep," and it is *more* efficient than the
leave-one-out ablation you sketched. This is the single most useful thing I found.
[07-ablation.md](07-ablation.md) is built on it.

**Cite it.** Prominently. "Productizing a published method" beats "reinventing it."

## ETH Zürich AGENTbench / Gloaguen et al. (arXiv 2602.11988), arXiv 2601.20404, arXiv 2605.10039

Covered in [00-verdict.md](00-verdict.md). Not competitors — they are your evidence base, your
effect-size prior, and your launch-post citations. 2605.10039 in particular is the paper you must
read before promising anyone a detectable pass-rate delta.

---

## Should you build this at all?

Yes — but the market position is narrower than "config eval." Every generic eval tool is one
`--repeat` flag away from the plumbing and infinitely far from the statistics; the one project
with the right statistics is unrunnable research code; the first-party tool is scoped to skills
and uses an LLM judge. The uncontested position is:

> **a zero-signup CLI that tells you, with a defensible confidence interval, which parts of your
> agent config you can safely delete — and how many tokens per turn you get back.**

Nobody occupies that. Go there.
