# ADR-0005 — Paired cluster bootstrap (BCa) as the single inferential engine

**Status:** accepted · **Date:** 2026-07-26

## Context

Design: T tasks × N trials × 2 arms. Trials within a task are correlated through task difficulty.
The same machinery must serve binary outcomes (pass rate), continuous outcomes (tokens, turns,
custom metrics), and the ablation regression's coefficients.

## Decision

One method everywhere: **cluster bootstrap over tasks, then trials within arm, 10,000 resamples,
BCa interval.** Refuse to emit an interval below 8 tasks.

## Rejected: normal-approximation binomial CI on pooled trials

What most tools in this space do, explicitly or by accident. It treats 200 correlated trials as
200 independent observations. Effective sample size is nearer T=20 than 200, so the interval
comes out too narrow and the tool confidently reports effects that aren't there. **This is the
specific error `ablate` exists to not make**, and it should be named in the README's method
section.

> **Amended 2026-07-27, after measuring it** ([`tests/test_calibration.py`](../../tests/test_calibration.py)).
> The original "√N ≈ 2–3× too narrow" figure is right for a **single arm's pass rate** — measured
> 2.1× at T=20, N=5 with near-deterministic tasks, exactly as the design effect predicts. It is
> **overstated for the paired difference**, which is what `ablate compare` actually reports:
> measured 1.05× at per-task effect SD 0.15 and 1.33× at 0.30. Pairing already cancels between-task
> difficulty, so all that remains for the cluster stage to recover is the between-task *effect*
> heterogeneity, which is bounded for proportions.
>
> This does not change the decision, for two reasons. The residual gap is real, grows with
> heterogeneity, and no number of trials reveals it — only resampling tasks does. And in the
> opposite regime (near-deterministic tasks, small true effect) the pooled interval is ~2× too
> **wide**; a tool that hardcoded a "pooling is too narrow" correction would be wrong there. The
> bootstrap needs no such factor because it adapts to whichever regime the suite is in, and that
> is the stronger argument for it than the headline multiplier was.
>
> The README quotes both measured figures. Quoting the single-arm number for the paired delta
> would be the same species of overclaiming this project is positioned against.

## Rejected: McNemar's test

Correct for paired binary outcomes — one subject, two conditions, discordant-pair counting. It is
what `probe-and-refine-tuning` uses, legitimately, because that work runs N=1 per (task, config).

Rejected here because with N>1 the observation per task is a *rate*, not a binary. McNemar would
require collapsing 5 trials to a single pass/fail per task, throwing away the within-task
replicate information that is most of what you paid for. It also gives a p-value, not an interval,
and the four-verdict taxonomy needs an interval.

## Rejected: paired t-test on the per-task differences

Cheap and nearly right. Rejected because T is small (8–30) and `d_i` is a bounded difference of
proportions that is visibly non-normal when per-task rates sit near 0 or 1 — which is exactly
where a lot of tasks sit. The bootstrap costs 40ms and drops the assumption.

## Rejected: mixed-effects / hierarchical model (or a Bayesian version)

Statistically the most principled option: task random effects, trial-level binomial likelihood,
partial pooling. It is what the probe-and-refine paper uses alongside McNemar, and a Bayesian
version would give a posterior over Δ that maps beautifully onto the four verdicts.

Rejected for v1 on dependency and explicability grounds: it needs `statsmodels` or a PPL
(violating [ADR-0001](0001-language-and-distribution.md)'s dependency budget and the `uvx` cold
start), the fitting is fragile at small T with separation in the binomial, and — decisively — a
bootstrap is explicable to a skeptical reader in three sentences while a hierarchical model is
not. For a tool whose product is credibility, explicability is a functional requirement.

The ablation stage does use a regression with task fixed effects
([07-ablation.md](../07-ablation.md)), but its uncertainty still comes from the same cluster
bootstrap — refit on each resample — rather than from model-based standard errors. One
inferential story throughout.

## Rejected: plain percentile bootstrap

Simpler by ~25 lines. Under-covers when the statistic is skewed, which it is near the rate
boundaries. BCa's bias-correction and acceleration terms are worth the lines.

## Consequences

- Everything is a resampling: no closed forms, no distributional assumptions, one code path for
  binary/continuous/regression uncertainty.
- **Coverage must be validated empirically**, not assumed. M2 includes an A/A calibration:
  run the same config against itself 20 times and confirm ~95% of intervals contain zero. Publish
  it. If coverage is off, there's an unmodelled variance component and the fix is stratification,
  not a wider interval ([R6](../11-risks.md)).
  **Done** — [`tests/test_calibration.py`](../../tests/test_calibration.py) runs it as a
  simulation (no API calls) at (T=8, N=5), (20, 5) and (20, 10), and additionally under
  between-task effect heterogeneity. Coverage lands in [0.90, 0.99] in every cell. The tests fail
  if it drifts, so this is a standing check rather than a one-time result.
- The T≥8 floor means `ablate try` (3 tasks) *cannot* produce a CI. That is correct and the
  report says so explicitly rather than printing a number it doesn't believe.
