# ablate — plan index

A tool that treats repo-level coding-agent config (`CLAUDE.md`, `AGENTS.md`, `.cursorrules`,
`skills/`) as code under test: run an agent headlessly N times per task under config A and
config B, check outcomes with deterministic assertions, report the delta with a confidence
interval, and attribute the delta to individual config sections by ablation.

Working name: **ablate** (the repo name; see [ADR-0009](adr/0009-naming.md)).

## Read in this order

| # | Doc | What it settles |
|---|---|---|
| 0 | [00-verdict.md](00-verdict.md) | **Read first.** Two research findings that change the product's framing. |
| 1 | [01-prior-art.md](01-prior-art.md) | What exists, why it doesn't solve this, what to steal, where you'd be duplicating. |
| 2 | [02-scope.md](02-scope.md) | v1 feature list, non-goals, what I argue you should cut. |
| 3 | [03-architecture.md](03-architecture.md) | Language, distribution, runner, isolation, agent abstraction, result store. |
| 4 | [04-task-suite.md](04-task-suite.md) | Task schema, assertion vocabulary, LLM-judge policy, git-history miner. |
| 5 | [05-statistics.md](05-statistics.md) | Sampling, CIs, MDE, multiple comparisons, how to phrase "it's noise". |
| 6 | [06-cost.md](06-cost.md) | Dollar model of a default run, the controls, the first-run number. |
| 7 | [07-ablation.md](07-ablation.md) | Section parsing, the search algorithm, cost, output shape. |
| 8 | [08-surfaces.md](08-surfaces.md) | CLI, terminal report, markdown report, GitHub Action — with mockups. |
| 9 | [09-milestones.md](09-milestones.md) | Four demoable increments with hour estimates. |
| 10 | [10-distribution.md](10-distribution.md) | README structure, landing page, the launch artifact. |
| 11 | [11-risks.md](11-risks.md) | Risks, early signals, responses, kill criteria. |
| 12 | [12-open-questions.md](12-open-questions.md) | What I could not resolve. Ordered by blast radius. |
| 13 | [13-llm-auth-and-bare-mode.md](13-llm-auth-and-bare-mode.md) | Practical explanation of LLM use, auth, `--bare`, ambient config, and leakage. |

## Decision records

Every decision where a competent engineer could reasonably pick differently.

| ADR | Decision |
|---|---|
| [0001](adr/0001-language-and-distribution.md) | Python + `uvx`, not Go or TypeScript |
| [0002](adr/0002-trial-isolation.md) | `git worktree` per trial, not containers |
| [0003](adr/0003-agent-invocation.md) | CLI subprocess adapters, not SDK bindings |
| [0004](adr/0004-no-llm-judge.md) | No LLM-as-judge in v1, deterministic assertions only |
| [0005](adr/0005-statistical-method.md) | Paired cluster bootstrap, not normal-approximation or McNemar |
| [0006](adr/0006-result-storage.md) | Append-only JSONL run log + SQLite index |
| [0007](adr/0007-ablation-search.md) | Randomized subset screening + ridge, not exhaustive leave-one-out |
| [0008](adr/0008-task-miner-scope.md) | Miner proposes, human curates; not an automated pipeline |
| [0009](adr/0009-naming.md) | Keep `ablate` |
| [0010](adr/0010-primary-metric.md) | Token/turn cost is the **primary** metric; pass rate is secondary |
