# ablate

[![ci](https://github.com/ablate-dev/ablate/actions/workflows/ci.yml/badge.svg)](https://github.com/ablate-dev/ablate/actions/workflows/ci.yml)
[![license](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
[![python](https://img.shields.io/badge/python-3.11%2B-blue)](pyproject.toml)

**`ablate` measures whether your `CLAUDE.md` actually does anything — and tells
you which parts of it you can delete.**

Run a coding agent N times under config A and config B, check outcomes with
deterministic assertions, and report the delta with a confidence interval. Then
attribute that delta to individual sections. Not "is config A better than
config B", but **which of these 400 lines can I delete.**

No LLM judge, anywhere ([ADR-0004](docs/adr/0004-no-llm-judge.md)). No hosted
service, no accounts, no telemetry.

## What it looks like

```
  $ ablate compare CLAUDE.md /dev/null --trials 5
  ablate 0.4.0 · claude-code 2.1.220 · claude-sonnet-5 · 10 tasks × 5 trials × 2 arms

  A  CLAUDE.md                  76 lines ·     748 tokens/turn
  B  (no config)                 0 lines ·       0 tokens/turn

──────────────────────────────────────────────────────────────────────────────────

  PASS RATE                                                         ● INCONCLUSIVE

      A       0.56   █████████████████████░░░░░░░░░░░░░░░░░
      B       0.58   ██████████████████████░░░░░░░░░░░░░░░░

      Δ = -2.0pp   95% CI [-32.0pp, +20.0pp]   BCa, 10k resamples

      Underpowered. The data are consistent with anything from -32.0pp to +20.0pp.
      To resolve to ±5.0pp you would need roughly 5 trials × 316 tasks.

  TOKENS PER TURN                                                     ● REGRESSION

      A     14,122   ██████████████████████████████████████
      B     10,090   ███████████████████████████░░░░░░░░░░░

      Δ = +4,032 (+40.0%)   95% CI [+35.6%, +44.3%]

      A measurably regresses tokens per turn: up 40.0% [+35.6%, +44.3%].
```

> **These numbers are synthetic.** That output is the bundled demo fixture, not
> a measurement — reproduce it exactly with `ablate demo`, which costs nothing.
> A real headline number goes here once one has been run. Publishing an invented
> one would be the exact failure this tool exists to catch.

## See it for $0

```bash
ablate demo
```

No API key, no network, no repo. Prints the full report so you can decide
whether to trust the thing before spending anything.

## Five-minute path

```bash
ablate cost CLAUDE.md                        # $0 — what your config costs per turn
ablate init                                  # $0 — scaffold .ablate/tasks/
# fill in the TODOs in .ablate/tasks/*.yaml
ablate try                                   # ≤ $2 — smallest real run, 18 trials
ablate compare CLAUDE.md /dev/null --trials 5   # ~$28 — the real thing
```

Then attribute the result to individual sections:

```bash
ablate sections CLAUDE.md                    # which sections earn their keep
ablate apply --drop s11,s06 CLAUDE.md --write
```

`ablate sections` runs in two stages: it screens every section on a cheap model
using a randomized subset design, then confirms a shortlist with dedicated
paired A/Bs on the target model. **Screening ranks; only the confirmatory stage
produces verdicts, and only an EQUIVALENT verdict licenses a deletion.**
`ablate apply` archives whatever it removes under `.ablate/removed/`, so no
deletion is irreversible.

`compare`, `try` and `sections` need the `claude` CLI on `PATH` and an
authenticated session or `ANTHROPIC_API_KEY`. They call it once per trial inside
an isolated `git worktree`, spending real money against your own account. Each
prints the run count, the estimated cost, and the effect size the design can
actually resolve **before** executing anything, and stops at `--max-spend`
(default `$25`).

## Why not just eyeball it?

Because three 2026 studies measured exactly this, and disagreed with intuition.

- **Gloaguen et al. (ETH Zürich), [arXiv 2602.11988](https://arxiv.org/abs/2602.11988)** — four
  coding agents over SWE-bench Lite and a 138-repo benchmark. LLM-generated
  context files *reduced* task success by ~3% and raised inference cost by >20%.
  Human-written ones gained ~4%.
- **[arXiv 2605.10039](https://arxiv.org/abs/2605.10039)** — a factorial study of four config
  structure variables across 1,650 Claude Code sessions. **No detectable effect**
  on instruction adherence for any variable, after multiple-testing correction.
- **[arXiv 2601.20404](https://arxiv.org/abs/2601.20404)** — AGENTS.md across 10 repos and 124
  PRs: 28.6% median runtime reduction and 16.6% fewer output tokens, with
  comparable completion.

The effects are small, sometimes negative, and mostly show up in **cost** rather
than in success. That is not something you can see by reading your config.

## How it decides

Every comparison resolves to exactly one of four verdicts, from the confidence
interval and an equivalence margin `m` (default ±5pp; `--margin` to change).

| Verdict | When | Means |
|---|---|---|
| 🔴 **REGRESSION** | interval sits below −m | A measurably worse than B |
| 🟢 **IMPROVEMENT** | interval sits above +m | A measurably better than B |
| 🔵 **EQUIVALENT** | interval fits inside ±m | **Bounded. Safe to prefer the cheaper config.** |
| 🟡 **INCONCLUSIVE** | anything else | Not enough data to say — here's what it would cost |

**EQUIVALENT and INCONCLUSIVE are opposite findings**, and conflating them is
the specific thing that makes people distrust eval tools. "We looked hard and
there is nothing there" is what licenses a deletion. "We didn't look hard
enough" licenses nothing. They never share a colour, a sentence, or a code path
— and every INCONCLUSIVE verdict ends with the tasks, trials and runs that would
resolve it.

## What it costs

| Command | Cost | Time | What |
|---|---:|---:|---|
| `ablate demo` | **$0** | 3 s | Bundled run. No API key, no network. |
| `ablate cost` | **$0** | instant | Per-section token cost. Exact arithmetic. |
| `ablate try` | **≤ $2** | ~6 min | 3 tasks × 3 trials on Haiku. Smoke test, not a measurement. |
| `ablate compare` | **~$28** | ~25 min | 8 tasks × 5 trials × 2 arms on Sonnet. |
| `ablate sections` | **~$110** | ~90 min | Screen on Haiku + confirm on Sonnet, K=12. |

Every spending command prints its estimate and its resolvable effect size
before it starts, and honours `--max-spend`. In CI the ceiling is mandatory and
breaching it fails the job.

## What it can't tell you

The honest limits, up front rather than after you have spent $28.

- **Pass rate is a guardrail, not a headline.** Published pass-rate effects of
  config changes are 3–4 percentage points. A default 8-task × 5-trial run
  resolves about 32 points. It will usually return INCONCLUSIVE on pass rate,
  and says so instead of pretending otherwise
  ([ADR-0010](docs/adr/0010-primary-metric.md)).
- **Token cost is where the signal is.** The same run resolves a ~16% change in
  tokens per turn, and measured cost effects are 16–28%. That asymmetry is why
  tokens are the primary metric.
- **Below 8 tasks there is no confidence interval at all.** `ablate init`
  scaffolds three and `ablate try` runs three — enough to prove the plumbing,
  not to measure anything.
- **Δ is an average over *your* suite**, for *your* agent at *your* model
  version. It is not a general claim about the config.
- **Screening is a ranking, not a measurement.** Ridge shrinks coefficients
  toward zero (~75% of true magnitude in simulation) and screening intervals
  cover ~88% rather than the nominal 95%. Read the *order* of the screening
  table, not its numbers — which is why the confirmatory stage is not optional.
- **Attribution is section-level with line ranges**, not per-line.
  `--granularity bullet` makes it literally per-item and roughly triples the cost.

## Method

Every interval comes from one engine: a paired cluster bootstrap with BCa
correction, pairing at task level, resampling tasks and then trials within arm
([ADR-0005](docs/adr/0005-statistical-method.md)). Coverage is
[validated empirically](tests/test_calibration.py), not assumed — an A/A
calibration confirms ~95% of intervals contain zero when the truth is zero.

The shortcut this avoids is pooling every trial into a binomial CI, which treats
correlated trials as independent. For a single arm's pass rate that interval
comes out ~2× too narrow. For the paired difference the error is smaller —
1.05× at moderate between-task heterogeneity, 1.3× when tasks respond very
differently — because pairing already cancels task difficulty. Both figures are
measured, not asserted.

## What a task looks like

A task is a prompt plus deterministic checks, pinned to a commit. `ablate init`
scaffolds three; this repo's own suite is in [`.ablate/tasks/`](.ablate/tasks).

```yaml
id: convention-cost-json
description: Add a --format json option to `ablate cost`.

setup: uv venv -q && uv pip install -q -e ".[dev]"
prompt: |
  `ablate cost` prints a human-readable table. Add a `--format text|json`
  option. Follow the conventions already used by the other commands.

assert:
  - id: tests-pass
    type: exit_code
    run: .venv/bin/python -m pytest tests/test_cli.py -q

  - id: test-added
    type: files_touched
    must_include: ["tests/**"]

  - id: no-bare-except
    type: absent
    pattern: 'except\s*:'
    in: diff

tags: [convention, smoke]
```

Six assertion types: `exit_code`, `files_touched`, `present`/`absent`, `lint`,
`script`. All deterministic, all runnable offline, all checkable by hand.

## In CI

```yaml
- uses: ablate-dev/ablate@v1
  with:
    config: CLAUDE.md
    baseline: origin/${{ github.base_ref }}
    max-spend: 15          # required — no default, on purpose
    fail-on: never         # regression | inconclusive | never
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
```

Posts a sticky PR comment whose footer states the MDE for the run — how large a
regression would have had to be before that green check could have caught it.
Start with `fail-on: never`; the first thing most teams learn is that their
suite is underpowered for pass rate, which is worth knowing before it gates a
merge. Full example in [`docs/examples/ablate.yml`](docs/examples/ablate.yml).

## Prior art

Generously and specifically: this stands on other people's work.

- **[promptfoo](https://github.com/promptfoo/promptfoo)** — the assertion vocabulary and YAML
  shape are modelled on it.
- **[inspect-ai](https://github.com/UKGovernmentBEIS/inspect_ai)** (UK AISI) — `--epochs`,
  bootstrap scoring, and the solver/scorer split.
- **`probe-and-refine-tuning`** ([arXiv 2606.20512](https://arxiv.org/abs/2606.20512)) — the
  screening-then-confirm method this project's ablation stage is built on.
- **SWE-bench harness** — the task-instance schema.
- **AgentLinter** — static `CLAUDE.md` linting; complementary, and the zero-signup
  `npx` framing is worth copying.

Full comparison in [`docs/01-prior-art.md`](docs/01-prior-art.md).

## Install

```bash
uvx --from ablate-cli ablate demo
```

The distribution is `ablate-cli` because `ablate` on PyPI belongs to an
unrelated deep-learning reporting tool. The command you type is still `ablate`.

From source:

```bash
uv venv && uv pip install -e ".[dev]" && source .venv/bin/activate
ablate --help
```

## Status

**Milestone 4 of 4** — the planned v1 surface is complete. Three things are
done in this repo but not yet true in the world:

- **Not on PyPI yet**, so `uvx` doesn't work. The publish job is written and
  gated on a repository variable.
- **`ablate demo` replays a synthetic fixture**, labelled on every run.
- **Nothing has been run against a live agent yet.** Until
  `tools/q2_injection_equivalence.py` runs, treat every result as measuring a
  *system-prompt appendix* rather than a `CLAUDE.md` — see
  [Q2](docs/12-open-questions.md).

Full design rationale in [`docs/`](docs/README.md); start with
[`docs/00-verdict.md`](docs/00-verdict.md). Progress in
[`CHANGELOG.md`](CHANGELOG.md).

## Repository layout

| Path | What |
|---|---|
| [`ablate/`](ablate) | The CLI — tasks, assertions, runner, adapters, statistics, reports. |
| [`.ablate/tasks/`](.ablate/tasks) | This repo's own task suite. |
| [`docs/`](docs) | Design docs and ADRs. Read before proposing anything structural. |
| [`tests/`](tests) | pytest suite — runs against real git repos, not mocks. |

## Contributing

See [`CONTRIBUTING.md`](CONTRIBUTING.md). Tests and lint must pass:

```bash
uv run pytest -q
uv run ruff check .
```

## License

[Apache-2.0](LICENSE)
