# 5. Statistics

This is the part competitors get wrong, so it is specified before the UI. Everything here is
implementable in ~150 lines of numpy.

## The estimand

Not "is config A better." Precisely:

> **Δ = the average, over the tasks in this suite, of the per-task difference in metric between
> config A and config B, for this agent at this model version.**

Three scope limits go in every report footer, because pretending otherwise is how these tools
lose trust: it is an average over *your* suite (not "in general"), for *your* agent version, and
it is an estimate of a population parameter that has real between-task heterogeneity.

## Sampling design

- **Paired by task.** Both arms run every task. Between-task difficulty variance — the largest
  variance component by far — cancels in the paired difference. This alone is worth roughly a
  3–5× reduction in required N versus an unpaired design.
- **Not paired at the trial level.** There is no way to make trial *r* of arm A and trial *r* of
  arm B share randomness: current frontier models expose no temperature or seed control, and
  agents are inherently stochastic. Common-random-numbers variance reduction is unavailable.
  Anyone who claims otherwise is fooling themselves.
- **Balanced.** Equal N per (task, arm). Unbalanced designs are supported by the estimator but
  the default scheduler always produces balanced work.
- **Shuffled execution order.** The work queue is shuffled so that a run aborted by the budget
  ceiling or a crash yields a roughly balanced partial design rather than "all of arm A."
- **Interleaved in time.** Do not run all of arm A on Monday and arm B on Tuesday. Model-side
  changes, rate-limit-induced retries, and provider load are real confounders. Shuffling handles
  this automatically; it is worth saying out loud in the docs.

Estimator, with per-task rates `p̂_Ai`, `p̂_Bi` over N trials each:

```
d_i = p̂_Ai − p̂_Bi          (or the per-task mean difference, for continuous metrics)
Δ̂   = (1/T) Σ_i w_i d_i    (w_i = task weight, default 1)
```

## Confidence interval: paired cluster bootstrap (BCa)

[ADR-0005](adr/0005-statistical-method.md).

```
repeat 10,000 times:
    resample T tasks with replacement            # cluster level
    for each resampled task, resample its N trials
        with replacement, within each arm        # trial level
    recompute Δ̂*
CI = BCa percentile interval over the Δ̂* distribution
```

Why this and not the alternatives:

- **Not a normal-approximation binomial CI on pooled trials.** Trials within a task are
  correlated through task difficulty. Pooling 200 trials and computing a binomial CI treats them
  as 200 independent observations when the effective sample size is closer to T=20. It produces
  an interval roughly 3× too narrow. This is the single most common error in this space and the
  reason the tool exists.
- **Not McNemar's test.** McNemar is correct for *paired binary* outcomes — the same subject
  measured twice. Here the pairing is at task level and each task yields a *rate*, not a single
  binary outcome. `probe-and-refine-tuning` uses McNemar legitimately because it runs N=1 per
  (task, config). With N>1 the bootstrap dominates: it uses the within-task replicate information
  McNemar throws away.
- **Not a t-test on the d_i.** T is small (8–30) and d_i is a difference of bounded proportions,
  visibly non-normal near the boundaries. The bootstrap is assumption-light and costs 40ms.
- **BCa, not plain percentile.** Percentile intervals under-cover when the statistic is skewed,
  which it is when per-task rates sit near 0 or 1. BCa's bias and acceleration corrections are
  ~25 extra lines. **Below T=8 tasks, refuse to emit a CI at all** and say so — bootstrap coverage
  at 5 clusters is not honest, and printing a wide interval implies more rigor than exists.

Same machinery, unchanged, for continuous metrics (tokens, turns, wall time, custom `script`
metrics). That generality is why the bootstrap is the right call.

## Minimum detectable effect at realistic N

For a paired design, `SE(Δ̂) ≈ sqrt( (σ²_δ + E[p_A(1−p_A) + p_B(1−p_B)]/N) / T )`, and the MDE at
80% power / α=0.05 two-sided is `2.80 × SE`.

**Pass rate**, worst case (`p ≈ 0.5`, per-task true-effect SD 0.05):

| Tasks T | Trials N | Agent runs | SE(Δ̂) | **MDE (pp)** |
|---:|---:|---:|---:|---:|
| 8 | 3 | 48 | 0.145 | **41** |
| 8 | 5 | 80 | 0.113 | **32** |
| 20 | 5 | 200 | 0.072 | **20** |
| 20 | 10 | 400 | 0.051 | **14** |
| 40 | 10 | 800 | 0.036 | **10** |
| 100 | 10 | 2,000 | 0.023 | **6.4** |

At `p ≈ 0.8` these improve by ~20% (16pp at T=20/N=5). They do not improve by an order of
magnitude.

Every row is reproduced from the formula above by
[`tests/test_stats.py`](../tests/test_stats.py), so the table cannot drift away from the
implementation. The `T=8, N=3` row originally read `0.132 / 37pp`, which the formula does not
produce; it was corrected to `0.145 / 41pp` on 2026-07-27. The remaining rows were already exact.

**Set this against the literature** ([00-verdict.md](00-verdict.md)): measured pass-rate effects
of config changes are **3–4 percentage points**. Detecting that needs ~2,000+ agent runs.
At Sonnet-5 rates and a modest task, that is a four-figure bill for a single comparison.

**This is the central honest fact about the product and it must be surfaced in the tool itself,
not buried in docs.** `ablate` should refuse to imply it can see effects it cannot see.

**Token cost per task**, same design, assuming per-trial token CV ≈ 25% around a 200k-token mean:

| Tasks T | Trials N | Agent runs | MDE (% of mean tokens) |
|---:|---:|---:|---:|
| 8 | 5 | 80 | ~16% |
| 20 | 5 | 200 | ~10% |
| 20 | 10 | 400 | ~7% |

Literature effect sizes on cost are **16–28%**. So at **T=20, N=5 — 200 runs, an affordable
run — the token metric is comfortably detectable and the pass-rate metric is not.** That
asymmetry is the entire argument for [ADR-0010](adr/0010-primary-metric.md).

And one metric is free and exact: **the token cost of the config text itself**, on every turn,
measured by counting tokens with zero variance and zero API calls. A 400-line `CLAUDE.md` is
~4,000 tokens re-sent on every turn of every session forever. `ablate` should print that number
before it runs anything.

## Choosing tasks for statistical power

- **Mid-difficulty tasks are worth several times their cost.** A task that all trials pass in
  both arms, or that no trial passes in either, contributes `d_i = 0` with near-zero variance:
  it costs full price, adds no information about the effect, and *attenuates* Δ̂ toward zero.
  `ablate` reports per-task pass rates so degenerate tasks are visible, and warns:
  `task 'add-readme-badge': 5/5 pass in both arms — contributes no signal, consider replacing.`
- **Do not silently drop degenerate tasks from the headline.** Excluding tasks based on their
  observed outcome is a forking path that biases Δ̂. `--exclude-degenerate` exists, is off by
  default, and any report produced with it is labelled as a secondary analysis.
- Prefer **more tasks over more trials** when the budget is fixed. `T` divides the whole variance;
  `N` only divides the within-task component. Going from N=5 to N=10 at fixed T buys less than
  going from T=10 to T=20 at fixed N — and costs the same.

## Multiple comparisons

Two places they arise.

**Across tasks.** Per-task deltas are *descriptive*. `ablate` never emits a per-task p-value and
never says "task 7 significantly regressed." Per-task rows in the report show point estimates and
raw counts with no inferential decoration. Selecting the largest of 20 noisy per-task deltas and
calling it a finding is the classic failure and the tool must make it structurally impossible.

**Across ablated sections.** This is the real problem: K sections means K simultaneous
comparisons, and at K=12 with α=0.05 the chance of at least one spurious "this section matters"
is ~46%.

Response, in three layers:

1. **Screening, not testing.** The ablation stage is explicitly framed as *ranking candidates*,
   not testing hypotheses. Its output is an ordered list with effect sizes and CIs, labelled
   "screening — not confirmatory."
2. **Benjamini–Hochberg FDR at q=0.10** applied across sections, reported as an
   adjusted-significance column. FDR, not Bonferroni: you are screening for follow-up, so
   controlling the false-*discovery* rate is the right error to control, and Bonferroni at K=15
   would leave you unable to detect anything at any affordable N.
3. **Confirmatory re-run.** The top and bottom candidates get a dedicated confirmatory A/B at
   full N, with a single pre-specified comparison and no adjustment needed. `ablate` prints the
   command:
   `ablate compare --drop "Testing conventions" --trials 10   # confirmatory, ~$8`

## The four-verdict taxonomy

Every comparison resolves to exactly one of four verdicts. This is the core UX decision and it is
what makes "no effect" a usable answer instead of a disappointment.

Let `[lo, hi]` be the 95% CI on Δ̂ and `m` the equivalence margin (default: 5pp for pass rate,
5% for continuous metrics; `--margin` to change).

| Verdict | Condition | Report language |
|---|---|---|
| **REGRESSION** | `hi < −m` | "Config B measurably regresses pass rate by 12pp [−19, −6]." |
| **IMPROVEMENT** | `lo > +m` | "Config B measurably improves pass rate by 9pp [4, 15]." |
| **EQUIVALENT** | `−m ≤ lo` and `hi ≤ +m` | "**No detectable difference.** The effect is at most ±5pp in either direction. Safe to prefer the cheaper config." |
| **INCONCLUSIVE** | otherwise | "**Underpowered.** The data are consistent with anything from −18pp to +11pp. To resolve to ±5pp you need ~10 trials × 40 tasks (~800 runs, ~$96)." |

EQUIVALENT is a **positive finding**, not a null result, and the report must present it that way.
It is the finding that licenses deletion. This is a two-one-sided-tests (TOST) equivalence claim
in bootstrap clothing; describe it as such in the methods section for anyone who checks.

## How to say "indistinguishable from noise" so users trust it

Users distrust a null result when it feels like the tool gave up. Four rules:

1. **Never print a bare "no significant difference."** Always print the interval. "Between −18pp
   and +11pp" tells the user something; "not significant" tells them nothing and reads as failure.
2. **Distinguish EQUIVALENT from INCONCLUSIVE, loudly.** These are opposite findings that
   frequentist convention collapses into the same non-rejection. Conflating them is the specific
   thing that makes people distrust eval tools. "We looked hard and there's nothing there"
   and "we didn't look hard enough" must never share a label or a colour.
3. **Always attach the price of an answer.** Every INCONCLUSIVE verdict ends with the concrete
   N and dollar cost that would resolve it, and the exact command to run. This converts
   "the tool failed" into "the tool told me what an answer costs" — which is a service.
4. **Show the guardrail alongside.** "Pass rate: no detectable difference (±5pp). Tokens: −31%
   [−38%, −24%]." A null on one metric is far more credible when it sits next to a decisive
   result on another metric from the same data — it demonstrates the instrument works.

## Things that will be got wrong and must be guarded against in code

- **Peeking / optional stopping.** `ablate run` computes statistics once, at the end. No live
  p-values. If `--watch` ever ships, it shows counts only. Users who re-run a run repeatedly and
  stop when they like the answer are doing sequential testing without correction; document it,
  and consider an "analyses run against this dataset: 4" counter in the report footer.
- **Rerunning until significant.** The result store makes it trivial to `--trials 5` then
  `--trials 10` then `--trials 20` and report whichever crosses. The run log makes it *visible*;
  the report footer should note when a run extends an earlier one.
- **The variance floor.** A model version bump changes the baseline. The report records
  `agent_version` and `model` per trial and warns loudly if a comparison mixes them.
- **`--alpha` fiddling.** Supported, but the report always prints the alpha used, prominently.
