# 6. Cost

Prices verified 2026-07-26 (Anthropic first-party API): Opus 5 `$5 / $25` per MTok in/out;
Sonnet 5 `$3 / $15` (introductory `$2 / $10` through 2026-08-31); Haiku 4.5 `$1 / $5`.
Cache reads bill at ~0.1× input; cache writes at 1.25× (5-min TTL) or 2× (1-hour TTL).
OpenAI pricing is not modelled here — see [Q1](12-open-questions.md).

## Unit cost of one trial

A trial is one agent run on one task. Modelling a mid-sized agentic coding task at ~8 turns
with prompt caching active — the dominant cost is **cumulative billed input across turns**,
because every turn re-sends the conversation:

| Component | Tokens | Sonnet 5 (list) | Sonnet 5 (intro) | Haiku 4.5 |
|---|---:|---:|---:|---:|
| Cache reads | 340,000 | $0.102 | $0.068 | $0.034 |
| Cache writes | 40,000 | $0.150 | $0.100 | $0.050 |
| Fresh input | 20,000 | $0.060 | $0.040 | $0.020 |
| Output | 12,000 | $0.180 | $0.120 | $0.060 |
| **Total** | | **≈ $0.49** | **≈ $0.33** | **≈ $0.16** |

Planning figures used throughout: **$0.35 per trial on Sonnet 5**, **$0.10 on Haiku 4.5**,
**$0.90 on Opus 5**. Observed range on real tasks is roughly 0.4×–2.5× these; the tool must
measure rather than assume, which is why `total_cost_usd` from the agent's own JSON output is
the source of truth and the estimator recalibrates from the first few completed trials.

## End-to-end cost of a default run

`ablate compare` with the default `--trials 5` over a suite of 8 tasks, two configs:

```
8 tasks × 5 trials × 2 arms = 80 agent runs
80 × $0.35  ≈  $28      (Sonnet 5)
80 × $0.10  ≈  $8       (--cheap, Haiku 4.5)
Wall clock: ~25 min at --concurrency 4, 75s median per trial
```

Cross-referencing [05-statistics.md](05-statistics.md): 8 tasks × 5 trials gives an MDE of ~32pp
on pass rate and ~16% on tokens. So the default run **can** resolve a token-cost question and
**cannot** resolve a pass-rate question — and the report says so explicitly rather than
pretending. A 20-task suite at N=5 costs ~$70 and gets pass-rate MDE to ~20pp.

## The first run a curious user does

Three entry points, in increasing order of commitment. The design goal is that **the first thing
anyone runs costs $0**, so the decision to spend is made *after* seeing what the output looks
like.

| Command | Cost | Wall clock | What it is |
|---|---:|---:|---|
| `uvx ablate demo` | **$0.00** | 3 s | Replays a bundled recorded run against a real OSS repo. No API key, no network. Prints the full report, ablation table and all. |
| `uvx ablate try` | **≤ $2** | ~6 min | 3 tasks × 3 trials × 2 arms = 18 runs on Haiku 4.5. Real agent, real repo, real report. Labelled **SMOKE TEST — not a measurement**; the verdict will read INCONCLUSIVE and the report explains why and what a real run costs. |
| `ablate compare` | ~$28 | ~25 min | The real thing. Prints the estimate and asks for confirmation first. |

`ablate try` is deliberately honest about being underpowered rather than quietly producing a
number the user might believe. A tool whose selling point is statistical conservatism cannot
open with a misleading demo.

## Ablation cost

The expensive command. With `K` config sections, exhaustive leave-one-out is `(K+1)` variants;
the screening design in [07-ablation.md](07-ablation.md) uses `R ≈ 2K + 8` variants.

For K=12 sections, 8 tasks:

| Stage | Design | Runs | Model | Cost |
|---|---|---:|---|---:|
| Screen | 32 variants × 8 tasks × 1 trial | 256 | Haiku 4.5 | **$26** |
| Confirm | 3 candidates × 8 tasks × 5 trials × 2 arms | 240 | Sonnet 5 | **$84** |
| **Total** | | 496 | | **~$110** |

Exhaustive leave-one-out at the same power would be ~13 variants × 8 tasks × 5 trials on
Sonnet = 520 runs ≈ $182, and would give no attribution for interactions. The screening design
is both cheaper and more informative.

**The cheap-model screening stage carries a real risk**: config sensitivity may not transfer
across model tiers — a section that matters for Haiku may not matter for Sonnet. The confirmatory
stage on the real model is what contains that risk, and it is not optional. `--screen-model` is
configurable; `--no-screen` runs the whole design on the target model at ~4× the price.

## Cost controls

**1. Task suite size.** `--tasks a,b,c` and `--tasks-tag smoke` subset the suite. `ablate init`
generates three tasks, not thirty, deliberately.

**2. Cheap-model mode.** `--cheap` maps to Haiku 4.5 (~3.5× cheaper). `--model` sets it
explicitly. Any report generated on a non-target model carries a banner.

**3. Response caching — three independent layers.**
   - *Provider prompt caching.* Free, automatic, and the largest lever: the system prompt, tool
     definitions and repo context are identical across trials of the same task, so cache reads at
     0.1× dominate. The runner should deliberately keep the invariant prefix byte-identical
     across trials (stable tool ordering, no timestamps, no per-trial IDs in the prompt) —
     see the silent-invalidator list in the Anthropic prompt-caching docs. Getting this wrong
     silently multiplies the bill by ~3×; getting it right is free.
   - *Trial-level memoization.* Keyed on `(task_id, variant_hash, agent, model, replicate)`.
     Editing one section of your config only re-runs the variants whose hash changed. This is
     what makes iterative use affordable.
   - *Environment cache.* Not dollars but wall clock: `setup` output is cached per
     `(base_commit, lockfile hash)` and hardlinked into worktrees.

**4. Incremental re-runs.** `ablate compare` after editing your config re-uses every unchanged
trial. Typical second run on an edited config: 30–60% of the first run's cost.
`--fresh` forces a full re-run.

**5. Hard spend ceiling.** `--max-spend` (default `$25`, configurable in `.ablate/config.yaml`).
Behaviour:

```
Estimated cost: $28.40  (80 runs × ~$0.355)
This exceeds --max-spend $25.00.
  [c] continue anyway    [r] raise ceiling to $30    [t] reduce to --trials 3 (~$17)    [q] quit
>
```

The ceiling is also enforced *during* the run against the running total of actual reported
costs, not just the pre-run estimate. On breach the run stops cleanly and writes a partial-results
marker; because the queue was shuffled, the partial data is still analyzable and the report says
`PARTIAL — 61/80 trials completed, budget ceiling reached`.

In non-interactive contexts (`--yes`, CI) exceeding the ceiling is a hard failure with a non-zero
exit, never a silent continue.

**6. Pre-run estimate always.** Even under the ceiling, `ablate` prints the estimate, the run
count, and the projected MDE *before* executing anything, so the user can judge whether the
spend will actually answer their question. Spending $28 to learn "inconclusive" is the worst
outcome, and it is avoidable at estimate time.
