# 7. Ablation

The headline feature: given a config file, report which parts of it earn their keep.

## Parsing config into ablation units

`config/parse.py` splits a markdown config into an ordered list of `Section`s.

**Default granularity: markdown heading.** Split on `##` and `###`; content before the first
heading becomes an implicit "preamble" section. Each section records:

```python
Section(
    id="s04",
    heading="Testing conventions",
    level=2,
    line_range=(87, 112),
    text="...",
    tokens=418,              # exact, via the provider's count_tokens endpoint
    required=False,          # user can pin a section as never-ablated
)
```

`--granularity bullet` splits top-level list items within a section into separate units. This is
what makes "line-level attribution" literally true rather than approximately true — but it
typically triples `K` and therefore the cost, so it is opt-in and the CLI says so.

**Be precise in the marketing.** The default output is *section-level attribution with line
ranges*, which is what a user actually wants (nobody deletes one line of a bulleted list). Call
it that. Claiming per-line attribution and delivering per-heading attribution is the kind of
small dishonesty that a tool selling statistical rigour cannot afford.

Rendering a variant is `"\n\n".join(s.text for s, keep in zip(sections, mask) if keep)`, plus a
stable header. Variant identity is `sha256` of the rendered bytes, which is what the trial cache
keys on.

### Guardrails on parsing

- Sections a config author marks with `<!-- ablate:keep -->` are pinned in every variant. This is
  the escape hatch for legally- or safety-required content (licence rules, "never touch prod").
- Sections under ~20 tokens are merged with their neighbour by default: they cannot plausibly
  carry a measurable effect and each one you keep costs a column in the design matrix.
- If `K > 24`, refuse and tell the user to run at `--granularity section` or to split the file.
  The design cost grows linearly in K and the interpretability collapses well before 24.

## The algorithm

[ADR-0007](adr/0007-ablation-search.md). Three stages.

### Stage 0 — free, exact: static token cost

Before any agent runs, print each section's exact token count and what it costs per turn. This
costs one `count_tokens` call and zero variance. For a lot of users this alone justifies the
install, and it is available in `ablate cost` as a standalone command that never touches an agent.

### Stage 1 — randomized subset screening (the core idea)

Exhaustive leave-one-out needs `K+1` variants and, crucially, tells you nothing about interactions
between sections. Instead, sample random keep-masks and regress outcome on section presence.

```
R = 2K + 8 variants
variant 0: all sections kept          (the reference)
variant 1: all sections dropped       (the floor)
variants 2..R: each section independently kept with p = 0.5,
               resampled until the design matrix is well-conditioned
                (no section constant across variants, max pairwise |corr| < 0.5)

for each variant v, each task t: run 1 trial
fit:  y_{v,t} = α_t + Σ_k β_k · mask_{v,k} + ε
      (task fixed effects α_t absorb between-task difficulty;
       ridge penalty λ chosen by leave-one-task-out CV)
```

`β_k` is the estimated marginal effect of *keeping* section `k`, averaged over the contexts in
which it appeared. Confidence intervals on `β_k` come from the same cluster bootstrap as
everywhere else: resample tasks, refit, take BCa percentiles.

Why this beats leave-one-out at equal budget:

- Every trial informs *every* `β_k`, because every trial varies many sections at once. Leave-one-
  out spends a whole task-set on one section's estimate.
- It is a standard fractional-factorial screening design, which is a hundred-year-old solved
  problem, and it is what `probe-and-refine-tuning` does with mixed-effects models
  ([01-prior-art.md](01-prior-art.md)). Do not invent something new here.
- The all-dropped variant gives you the empty-config floor for free, which is the number most
  users actually want ("what happens if I delete the whole thing?").

Run on the cheap model by default ([06-cost.md](06-cost.md)), `--no-screen` to run on target.

The task fixed effects `α_t` are the reason N=1 per (variant, task) is acceptable here: pooling
across R≈32 variants gives ~32 observations per task, and the regression borrows strength across
tasks. This is the whole reason the design is affordable.

### Stage 2 — confirmatory A/B

Take the top-3 sections by |β| (plus any section whose β is indistinguishable from zero but whose
token cost is large — the *deletion candidates*, which are the commercially interesting ones) and
run a dedicated, pre-specified, full-N A/B on the target model: full config vs. full config minus
that section. One comparison each, no multiplicity adjustment needed, four-verdict output from
[05-statistics.md](05-statistics.md).

The stage-1 output is labelled **screening — not confirmatory** everywhere it appears.

## Cost of full attribution

For `K=12`, 8 tasks (from [06-cost.md](06-cost.md)):

| Approach | Runs | Cost | Gives you |
|---|---:|---:|---|
| Exhaustive LOO, N=5, target model | 520 | $182 | Per-section marginal effect, no interactions |
| **Screen (cheap, N=1) + confirm (target, N=5)** | **496** | **~$110** | Ranked β for all K, CIs, empty-config floor, plus confirmed verdicts on 3 |
| Screen only | 256 | $26 | Ranked candidates, screening-grade only |
| Stage 0 only | 0 | **$0** | Exact per-section token cost |

Cost scales as `O(K · T)` for screening plus `O(c · T · N)` for `c` confirmations — linear in K,
not exponential, and it is the linearity that makes this shippable.

## Output

```
$ ablate ablate CLAUDE.md --tasks .ablate/tasks --trials 5

CLAUDE.md — 412 lines, 11 sections, 4,180 tokens per turn

  SCREENING (256 runs, haiku-4.5, N=1 per cell) — ranking only, not confirmatory
  ┌────┬──────────────────────────────┬────────┬─────────┬──────────────────┬──────┐
  │ #  │ Section                      │ Tokens │ β pass  │ 95% CI           │ FDR  │
  ├────┼──────────────────────────────┼────────┼─────────┼──────────────────┼──────┤
  │ s07│ Error handling patterns      │    311 │  +0.14  │ [+0.04, +0.24]   │ .03 *│
  │ s02│ Repo layout                  │    286 │  +0.09  │ [−0.01, +0.19]   │ .11  │
  │ s09│ Commit message format        │    142 │  +0.02  │ [−0.06, +0.10]   │ .62  │
  │ s04│ Testing conventions          │    418 │  +0.01  │ [−0.08, +0.09]   │ .81  │
  │ s01│ Preamble / tone              │    203 │  −0.01  │ [−0.09, +0.08]   │ .84  │
  │ s06│ "IMPORTANT" reminders        │    377 │  −0.03  │ [−0.12, +0.06]   │ .62  │
  │ s11│ Historical decisions log     │  1,204 │  −0.05  │ [−0.15, +0.04]   │ .48  │
  │ …  │ (4 more)                     │  1,239 │         │                  │      │
  └────┴──────────────────────────────┴────────┴─────────┴──────────────────┴──────┘
  Empty-config floor: 0.44 pass rate  ·  Full config: 0.58  ·  Δ = +0.14 [+0.02, +0.26]

  CONFIRMED (240 runs, sonnet-5, N=5, paired, pre-specified)
  ┌──────────────────────────────┬──────────────────┬──────────────┬──────────────┐
  │ Dropping…                    │ Δ pass rate      │ Δ tokens/turn│ Verdict      │
  ├──────────────────────────────┼──────────────────┼──────────────┼──────────────┤
  │ s11 Historical decisions log │ −1pp [−7, +6]    │ −1,204 (−29%)│ EQUIVALENT   │
  │ s06 "IMPORTANT" reminders    │ +2pp [−5, +9]    │   −377 ( −9%)│ EQUIVALENT   │
  │ s07 Error handling patterns  │ −13pp [−21, −5]  │   −311 ( −7%)│ REGRESSION   │
  └──────────────────────────────┴──────────────────┴──────────────┴──────────────┘

  RECOMMENDATION
    Delete s11 and s06 — 1,581 tokens/turn (−38%), no detectable effect on
    pass rate (each within ±5pp at 95% confidence).
    Keep s07 — dropping it measurably regresses pass rate by 13pp.
    Remaining 6 sections: screening was inconclusive. To resolve them you would
    need ~5 trials × 20 tasks each (~$70 per section).

    ablate apply --drop s11,s06 --write CLAUDE.md
```

Design notes on that output:

- **Token cost is a column on every row**, because it is exact and it is what turns an
  EQUIVALENT verdict into an action.
- **The recommendation is deletions, not additions.** `ablate` never proposes new config text
  ([02-scope.md](02-scope.md), non-goals).
- **The unresolved sections are named and priced**, not silently dropped. "Six sections we
  couldn't resolve, here's what resolving them costs" is a service; omitting them is a lie by
  selection.
- `ablate apply` performs the edit and leaves the removed sections in
  `.ablate/removed/<timestamp>.md` with a one-line git-style provenance header, so nothing is
  ever unrecoverable.
