# ADR-0010 — Token/turn cost is the primary metric; pass rate is the guardrail

**Status:** accepted · **Date:** 2026-07-26 · **Supersedes the original brief's framing**

## Context

The brief describes the tool's output as "the pass-rate delta with a confidence interval." Three
2026 studies and the power arithmetic in [05-statistics.md](../05-statistics.md) say that is not
achievable at any budget a solo dev's users will spend.

- Measured pass-rate effects of config changes: **3–4 percentage points**
  (Gloaguen et al., arXiv 2602.11988). One well-powered factorial study at 1,650 sessions found
  **no detectable effect at all** (arXiv 2605.10039).
- Measured cost effects: **16–28%** reduction in runtime and output tokens
  (arXiv 2601.20404, at 124 PRs).
- MDE at 20 tasks × 5 trials (200 runs, ~$70): **~20pp** on pass rate, **~10%** on tokens.

So at an affordable N the token metric is comfortably detectable and the pass-rate metric is
not — by roughly an order of magnitude in the ratio of effect size to MDE.

## Decision

- **Primary metric: tokens (and turns) per task.** Continuous, low variance, large effect size,
  and partially measurable with *zero* variance (the config text's own token cost is arithmetic,
  not an experiment).
- **Guardrail metric: pass rate.** Always reported with equal visual weight, and the equivalence
  verdict on it is what licenses a deletion.
- **The product claim becomes a non-inferiority claim:**
  > "These 340 lines cost 4,100 tokens on every turn. Removing them changes pass rate by
  > −1pp ± 9pp — no detectable difference. Delete them."

## Rejected: pass rate as primary, per the original brief

It is the metric users *ask* for and the one that sounds like it matters. Rejected because a tool
whose headline number is almost always "indistinguishable from noise" reads as broken, and
because promising a resolution the design cannot deliver is exactly the overclaiming this project
positions itself against. Making the undeliverable metric the headline would be a self-inflicted
credibility wound.

## Rejected: both as co-primary with no designated headline

The strongest counter-argument, and it wins for the **report UI** — the terminal and markdown
reports do show both with equal prominence, because a user deciding whether to delete something
needs both numbers side by side. It loses for the **marketing**, where a single headline claim is
required and it has to be one that survives scrutiny. See [Q3](../12-open-questions.md); this is
the one decision here I'd most want the maintainer to confirm.

## Rejected: wall-clock time as primary

The AGENTS.md paper's headline (−28.6% median runtime) and intuitively meaningful. Rejected
because wall time is contaminated by provider load, rate-limit backoff, network, and local
machine state — none of which are properties of the config. It is reported as a tertiary
diagnostic, never a verdict metric.

## Consequences

- **The product's floor is deterministic.** `ablate cost` measures per-section token cost exactly,
  with no agent runs, no statistics, and no possibility of a null result. Even in the worst world
  where nothing about agent *behaviour* is measurable at affordable N, the tool still tells you a
  true and useful number. This is what makes [R4](../11-risks.md) survivable.
- The report's visual hierarchy, the README's first sentence, and the launch artifact's title all
  follow from this decision, which is why it is an ADR and not an implementation detail.
- It reframes EQUIVALENT from a disappointing null into the *desired* outcome: the finding that
  grants permission to delete. That reframe is arguably the whole product.
- Users arriving expecting "does my config make the agent better" must be met honestly and early —
  in the README's "What it can't tell you" section, not after they've spent $28.
