# 8. Surfaces

These mockups are the marketing assets. The terminal report in particular is the screenshot that
gets posted, so it is specified before the code.

## CLI command set

Nine commands. Everything else is a flag.

| Command | Purpose |
|---|---|
| `ablate demo` | Replay a bundled recorded run. $0, no API key, no network. |
| `ablate init` | Detect test command + config file, scaffold `.ablate/` with 3 tasks. |
| `ablate cost [FILE]` | Static token cost per section. $0, no agent runs. |
| `ablate try` | Smallest real run. 3×3×2 on the cheap model, ≤$2. |
| `ablate compare A B` | The A/B. `B` may be `/dev/null` or `--drop <section-ids>`. |
| `ablate ablate [FILE]` | Full section attribution. |
| `ablate report [RUN]` | Re-analyze a stored run. Zero API calls. |
| `ablate apply` | Write a recommended edit, archiving removed sections. |
| `ablate mine` | *Experimental.* Propose tasks from git history. |

### Example invocations

```bash
# Zero-cost first contact
uvx ablate demo

# Five-minute path
uvx ablate init
uvx ablate try

# Is my CLAUDE.md doing anything at all?
ablate compare CLAUDE.md /dev/null --trials 5

# Did the 60 lines I added last week help?
ablate compare CLAUDE.md --config-b <(git show HEAD~20:CLAUDE.md) --trials 8

# Which sections earn their keep?
ablate ablate CLAUDE.md

# What would deleting these two cost me?
ablate compare CLAUDE.md --drop s11,s06 --trials 10

# Re-analyze last night's run with a tighter equivalence margin. Free.
ablate report --last --margin 0.03

# Compare across vendors on the same suite
ablate compare CLAUDE.md AGENTS.md --agent codex --trials 5

# CI mode
ablate compare CLAUDE.md --config-b origin/main:CLAUDE.md \
  --trials 5 --yes --max-spend 15 --format junit --fail-on regression
```

### Universal flags

`--trials N` · `--tasks DIR` · `--tasks-only IDS` · `--tasks-tag TAGS` ·
`--agent claude-code|codex` · `--model ID` · `--cheap` · `--concurrency N` · `--max-turns N` ·
`--max-spend USD` · `--margin PCT` · `--alpha A` · `--seed N` (shuffle and bootstrap only — not
agent sampling; see [05-statistics.md](05-statistics.md)) · `--fresh` · `--keep-worktrees` ·
`--format text|md|json|junit|pr-comment` · `--fail-on regression|inconclusive|never` · `--yes`.

Suite subsetting ended up as two flags rather than one. `--tasks` stayed the directory, since
that is what shipped in M1 and changing its meaning would have broken existing invocations;
`--tasks-only` takes ids or globs (`fix-*,add-cli-flag`) and `--tasks-tag` matches the `tags:`
field on a task. Both change *which estimand is being estimated* — Δ is an average over the tasks
in the suite — so they are cost controls, not analysis options.

## Terminal report

```
$ ablate compare CLAUDE.md /dev/null --trials 5

  ablate 0.4.1 · claude-code 2.1.220 · claude-sonnet-5 · 8 tasks × 5 trials × 2 arms

  A  CLAUDE.md              412 lines · 4,180 tokens/turn
  B  (no config)              0 lines ·     0 tokens/turn

  Estimated: 80 runs · ~$28.40 · ~25 min          [enter to run, q to quit]

  ████████████████████████████████████████  80/80   24m 51s   $26.18

──────────────────────────────────────────────────────────────────────────────────

  PASS RATE                                                        ● EQUIVALENT

      A  0.58   ██████████████████████░░░░░░░░░░░░░░░░
      B  0.55   █████████████████████░░░░░░░░░░░░░░░░░

      Δ = +2.5pp   95% CI [−4.1, +9.2]   paired cluster bootstrap, 10k resamples

      No detectable difference. The effect is at most ±5pp in either direction.

  TOKENS PER TURN                                                 ● IMPROVEMENT (B)

      A  14,220   ████████████████████████████████████
      B  10,040   █████████████████████████░░░░░░░░░░░

      Δ = −4,180 (−29.4%)   95% CI [−31.1%, −27.8%]

  TURNS TO COMPLETION                                              ● EQUIVALENT

      A  7.4   B  7.9      Δ = +0.5   95% CI [−0.9, +1.8]

──────────────────────────────────────────────────────────────────────────────────

  PER TASK                                     (descriptive only — no per-task tests)

    task                       A       B      Δ
    fix-null-deref           5/5     4/5    +20pp
    add-cli-flag             4/5     4/5      0
    refactor-parser          2/5     3/5    −20pp
    convention-new-handler   5/5     3/5    +40pp
    restraint-small-fix      3/5     3/5      0
    add-readme-badge         5/5     5/5      0   ⚠ no signal — all pass both arms
    migrate-config           0/5     0/5      0   ⚠ no signal — none pass either arm
    fix-race-condition       2/5     4/5    −40pp

    ⚠ 2 of 8 tasks contributed no signal. Replacing them would tighten the interval
      by roughly 15% at the same cost.

──────────────────────────────────────────────────────────────────────────────────

  VERDICT

    Your 412-line CLAUDE.md costs 4,180 tokens on every turn and does not
    measurably change whether tasks pass (±5pp, 95% confidence).

    This does NOT mean it has no effect — it means any effect is smaller than
    this suite can resolve. To detect a 3pp change you would need roughly
    10 trials × 40 tasks (800 runs, ~$280).

    Next:  ablate ablate CLAUDE.md      # which sections are carrying the weight?

  Run 2026-07-26T14-02-11Z · 80 trials · ablate report --last to re-analyze
```

Design rules encoded above:

- Cost, run count and ETA are shown **before** the confirm prompt, never after.
- Every headline number carries an interval. There is no bare "not significant" anywhere.
- The four verdicts get four distinct glyphs and colours: ● EQUIVALENT (blue), ● IMPROVEMENT
  (green), ● REGRESSION (red), ● INCONCLUSIVE (yellow). EQUIVALENT is deliberately *not* grey —
  it is a finding, not an absence.
- Per-task rows are explicitly labelled descriptive. No stars, no p-values.
- Degenerate tasks are flagged with the cost of fixing them.
- Every INCONCLUSIVE / EQUIVALENT verdict ends with the price of a sharper answer.
- The footer always tells you how to re-analyze for free.

## Markdown report

`--format md` writes `.ablate/runs/<id>/report.md` — for PR comments, blog posts, and the
launch artifact. Same content, plus a reproducibility block:

```markdown
# ablate report — CLAUDE.md vs. no config

**Verdict: EQUIVALENT on pass rate, −29.4% tokens per turn.**

| Metric | A (CLAUDE.md) | B (none) | Δ | 95% CI | Verdict |
|---|---:|---:|---:|---|---|
| Pass rate | 0.58 | 0.55 | +2.5pp | [−4.1, +9.2] | ● EQUIVALENT |
| Tokens/turn | 14,220 | 10,040 | −29.4% | [−31.1, −27.8] | ● IMPROVEMENT (B) |
| Turns | 7.4 | 7.9 | +0.5 | [−0.9, +1.8] | ● EQUIVALENT |

<details><summary>Per-task breakdown (descriptive)</summary>…</details>

## Method
Paired cluster bootstrap (10,000 resamples, BCa), pairing at task level, resampling
tasks then trials within arm. Equivalence margin ±5pp. α = 0.05.
Minimum detectable effect for this design: 32pp on pass rate, 16% on tokens.

## Reproduce
```
ablate compare CLAUDE.md /dev/null --trials 5 --seed 7
```
agent `claude-code 2.1.220` · model `claude-sonnet-5` · repo `acme/widgets@3f9a1c2`
80 trials · raw log: [`trials.jsonl`](trials.jsonl) (1.2 MB)
```

The **Method** and **Reproduce** blocks are not optional. They are the difference between a
screenshot people believe and a screenshot people argue with, and they are what makes the launch
artifact ([10-distribution.md](10-distribution.md)) defensible.

## GitHub Action

```yaml
# .github/workflows/ablate.yml
name: ablate
on:
  pull_request:
    paths: ['CLAUDE.md', 'AGENTS.md', '.ablate/**']

jobs:
  config-regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: astral-sh/setup-uv@v5
      - uses: ablate-dev/ablate-action@v1
        with:
          config: CLAUDE.md
          baseline: origin/${{ github.base_ref }}
          trials: 5
          max-spend: 15
          fail-on: regression        # regression | inconclusive | never
          comment: true
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
```

`fail-on` semantics, mapping directly onto the four verdicts:

| Value | Fails the PR when |
|---|---|
| `regression` *(default)* | Verdict is REGRESSION on any guardrail metric. |
| `inconclusive` | REGRESSION **or** INCONCLUSIVE — "prove it didn't break anything." Strict; expensive. |
| `never` | Never fails. Comment-only mode. Recommend this for the first two weeks of adoption. |

PR comment (sticky, updated in place):

```markdown
### ablate · CLAUDE.md changed · ✅ no regression detected

|  | main | this PR | Δ | 95% CI |
|---|---:|---:|---:|---|
| Pass rate | 0.58 | 0.61 | +2.5pp | [−4.9, +10.1] ● EQUIVALENT |
| Tokens/turn | 14,220 | 11,830 | **−16.8%** | [−18.2, −15.1] ● IMPROVEMENT |

This PR removes 2 sections (1,581 tokens/turn) with no detectable effect on pass rate.

<sub>8 tasks × 5 trials × 2 arms · 80 runs · $26.18 · claude-sonnet-5 ·
MDE for this design: 32pp. A change smaller than that would not be visible here.
[full report](…) · [raw log](…)</sub>
```

The MDE disclosure in the footer is the detail that makes the Action trustworthy rather than
theatrical: it tells reviewers exactly how much reassurance a green check is worth. Every other
CI eval integration omits this, and every one of them is quietly lying.

**Cost guardrails are mandatory in CI**, where a runaway loop bills a credit card unattended:
`max-spend` is required (no default), `--yes` never bypasses it, breaching it fails the job, and
the Action refuses to run on `pull_request_target` or on forks unless explicitly opted in
(otherwise any fork PR can spend your API budget).
