# Changelog

All notable changes to this project are documented here. Format loosely
follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versions
follow [SemVer](https://semver.org/).

## [Unreleased]

## [0.5.0] - 2026-07-28

First release published to PyPI: `pip install ablate-cli`.

Everything in this release came out of one session running the whole CLI
against a repo it had never been tried on — a static HTML/CSS site with no
Python, no test framework, and an ordinary Claude subscription login. Four of
the nine commands were broken in ways the test suite could not see, because
every existing test ran against this repository.

### Fixed

- **A run that measured nothing rendered as a run that found nothing wrong.**
  With every trial failing, the PR comment read "✅ no regression detected"
  over eighteen errors. `bootstrap_delta` was right to return NaN for an empty
  sample — zero would be a measurement — but four renderers then formatted
  that NaN as a number. Unmeasured arms are now `None` rather than `0.0`, a
  green check requires having measured something, and the comment carries the
  error warnings it had been dropping. `MetricAnalysis.measured` is the single
  question renderers ask.

- **`--format json` did not emit valid JSON.** Bare `NaN` and `Infinity`
  tokens, which RFC 8259 does not define: `JSON.parse` and Go's
  `encoding/json` both reject the document, so the CI-facing artifact was
  unparseable exactly when something had gone wrong. Now `null`, serialized
  with `allow_nan=False`.

- **Trials that could not authenticate ran to completion.** `claude --bare` —
  load-bearing for isolation — reads only `ANTHROPIC_API_KEY` and the
  provider variables; OAuth and the keychain are never consulted. Anyone
  logged in with a subscription and no API key had preflight pass (it checked
  `claude --version`, which reports the binary, not credentials) and then lost
  100% of trials to "Not logged in". Preflight now checks the credential
  `--bare` will actually use, and offers the two real routes: set a key and
  keep full isolation, or `--injection native` and accept that ambient config
  enters every trial. Under `--yes` or without a TTY it refuses rather than
  choosing on the user's behalf.

- **The runner no longer spends a whole budget on the same error.** Six
  consecutive failures with nothing having succeeded aborts and quotes the
  agent's own message. Only before the first success — a hard suite where most
  trials fail is a measurement, not a misconfiguration.

- **`ablate --version` raised `RuntimeError`.** `version_option()` infers the
  distribution from the command name, which is `ablate`; the distribution is
  `ablate-cli`.

- **`ablate try` failed on every invocation** with `TypeError`. `--tasks-only`
  and `--tasks-tag` were added as options without being added to the callback
  signature, so the headline "smallest real run" command had never run.

- **`ablate init` silently overwrote hand-written task files** with TODO
  stubs. It now skips them and reports what it kept. The suite is the most
  expensive artifact a user of this tool produces, and `ablate apply` archives
  every line it removes.

### Added

- **`--injection {append,native}`** on `compare`, `sections` and `try`. The
  adapter had supported both delivery modes since M1 and no command surface
  exposed either.
- `ablate init` reports authentication status, so the problem surfaces at
  setup rather than during the first paid run.

### Changed

- **Renamed.** The GitHub organisation is `ablate-dev` and the PyPI
  distribution is `ablate-cli`. Neither of the previous names was actually
  available: `github.com/ablate-ai` belongs to an unrelated project, and
  `ablate` on PyPI is a deep-learning experiment reporter. The import package
  and the command stay `ablate`; only the install name changes, so the
  zero-install path is now `uvx --from ablate-cli ablate`.

### Added

- **`tools/q2_injection_equivalence.py`** — the experiment
  `docs/09-milestones.md` says must run in M1 and never did. It runs the *same
  config* through both delivery mechanisms (`--bare` + appended system prompt
  vs. the config on disk) and asks the tool's own paired comparison whether
  they differ. Until it runs, every `ablate` result should be read as measuring
  a system-prompt appendix. `ClaudeCodeAdapter(injection=...)` is the knob.
- **`--tasks-only` and `--tasks-tag`** — suite subsetting, the first and
  cheapest cost control in `docs/06-cost.md` and the only one that had never
  been implemented. Tasks gain a `tags:` field.
- **Steady-state cost** (Q7). The first trial of each (task, arm) pays a
  prompt-cache write; the report now prints the warm per-trial mean beside the
  raw total, because the blended figure overstates what the next run costs.

### Changed

- **Screening intervals are now honest, and wider.** λ is re-tuned inside every
  bootstrap resample. Holding it fixed treated a value chosen *from the data*
  as if it were known in advance: measured coverage 89% → 97% in a null world
  (interval ~4x wider) and 85% → 88% where real effects are present (~14%
  wider). BCa was not implicated — a plain percentile interval with fixed λ
  measured 89% too.
- The regression's numeric core was restructured to make that affordable:
  per-task sufficient statistics are stacked arrays, so a leave-one-out design
  is `total - grams[i]` rather than a Python-level re-sum. Identical maths,
  985s → 4.5s for a screening analysis.

### Fixed

- Expected errors printed a traceback and exited **0** outside a git repo. Every
  `AblateError` carries a sentence written for the user; a stack trace buried it
  and the zero exit code would have made a failed CI step look green.
- `ablate cost` no longer requires a git checkout. It is the zero-cost
  first-contact command and pure arithmetic on a file.
- Config cost is quoted per 1,000 turns. A modest config printed as `~$0.000`
  per session, which reads as free — the opposite of the point.

## [0.4.0] - 2026-07-27

Milestone 4 (`docs/09-milestones.md`): it's a product. This completes the
planned v1 surface.

### Added

- **`--format md|json|junit|pr-comment`.** The markdown report carries Method
  and Reproduce blocks, which `docs/08-surfaces.md` is explicit are not
  optional — they are what makes a screenshot defensible. Every format states
  the minimum detectable effect for the design that produced it.
- **`--fail-on regression|inconclusive|never`**, mapping onto the four
  verdicts, checked across every metric rather than only pass rate. A config
  that holds pass rate steady while inflating token cost is a regression.
- **GitHub Action** (`action.yml`) with a sticky PR comment, step summary and
  run artifacts. `max-spend` is required with no default. The action refuses
  `pull_request_target` outright — it runs base-repo secrets against PR code,
  which for an agent-executing action is a credential-exfiltration primitive —
  and skips fork PRs unless `allow-forks` is set.
- **Codex adapter** (`--agent codex`). Written against the documented
  interface and **not verified against the binary**; see Q1.
- **PyPI publish job**, gated on a repository variable so enabling it is a
  settings change. Holds only `id-token: write`.

### Fixed

- An all-zero token count is now treated as "the agent did not report usage"
  rather than as a measurement of zero. Previously an agent that reports no
  token data would have made its arm look spectacularly cheap. `cost_usd=None`
  continues never to be coerced to 0, so `--max-spend` reports the trials it
  could not account for instead of treating them as free.
- A metric that no task reported in both arms returns NaN with an explanation
  instead of dividing by an empty weight sum.

### Known limitations

- `uvx --from ablate-cli ablate` needs the first PyPI release; the job is
  written but the repo needs a trusted publisher configured.
- `ablate demo` still replays a **synthetic** fixture, labelled as such on
  every run. Recording one against a real repository is the remaining M4 item.
- The Codex adapter is untested against a real `codex` binary, and Q1 —
  whether `codex exec --json` reports token usage at all — is still open. If
  the answer is no, the primary metric is unavailable for Codex arms and
  cross-vendor comparison degrades to pass rate.

## [0.3.0] - 2026-07-27

Milestone 3 (`docs/09-milestones.md`): the headline feature. Not "is config A
better than config B", but which of these 400 lines can I delete.

### Added

- **`ablate ablate`** — two-stage section attribution. Stage 1 screens every
  section from a randomized subset design (`R = 2K + 8` variants, each section
  kept at p=0.5) on a cheap model, fitting ridge with task fixed effects.
  Stage 2 promotes a shortlist to dedicated, pre-specified, full-N paired A/Bs
  on the target model. Screening ranks; only confirmation produces verdicts.
- **`ablate apply`** — performs a deletion and archives what it removes to
  `.ablate/removed/<timestamp>.md` with a provenance header. Previews by
  default; `--write` performs the edit. Pinned sections cannot be dropped.
- **Design conditioning checks** — a design where a section never varies, or
  where two sections are collinear, is resampled rather than fitted. If none
  conditions, the tool raises: there is no honest caveat that repairs a
  collinear design.
- **Ridge with task fixed effects**, absorbed by the within transformation so
  the penalty never touches task difficulty. λ by leave-one-task-out CV.
  Intervals on β come from the same cluster bootstrap as everything else.
- **Benjamini-Hochberg FDR** at q=0.10 across sections. FDR not Bonferroni:
  screening produces a shortlist, so the false-*discovery* rate is the right
  error to control.
- **Config parsing for ablation** — `<!-- ablate:keep -->` pins, small-section
  merging, a 24-section ceiling, `--granularity bullet`, and mask rendering
  with a content hash that the trial cache keys on.

### Changed

- INCONCLUSIVE now distinguishes two cases that had shared one sentence. An
  interval that spans zero is underpowered and carries a price for resolving
  it; an interval that *excludes* zero but straddles the equivalence margin is
  a threshold problem, and no number of extra trials fixes it. The old wording
  produced reports reading "could not tell whether it matters" directly under
  an interval of [−57pp, −28pp].
- A required-design quote is only shown when it exceeds the design already
  run. For a wide margin the arithmetic will otherwise recommend collecting
  *fewer* tasks than were just collected.
- ADR-0007's claim that ridge shrinkage is "conservative in the right
  direction" for the deletion question was amended after measurement. It is
  backwards — shrinking β toward zero makes a section that matters look more
  deletable. The confirmatory stage is what contains that risk, which is a
  sharper reason for it than the original.
- BCa lives in one place; the paired-comparison bootstrap and the ablation
  regression share it rather than carrying separate copies.

### Known limitations

Screening is a ranking, not a measurement, and the report says so on every
run: simulated against known effects, coefficients land at ~75% of true
magnitude and screening intervals cover ~88% against a nominal 95%. Both
figures are pinned by tests. The confirmatory stage is calibrated to nominal
and is not optional.

## [0.2.0] - 2026-07-27

Milestone 2 (`docs/09-milestones.md`): the numbers now mean something. Every
delta carries a confidence interval or an explicit statement of why it has
none, and no run starts without showing what it will cost and what it can
actually resolve.

### Added

- **Paired cluster bootstrap with BCa intervals** (`ablate/stats/`), per
  ADR-0005: resample tasks, then trials within arm, 10,000 times. One engine
  for binary and continuous metrics alike. No interval is emitted below 8
  tasks — bootstrap coverage at that few clusters is not trustworthy, and the
  report says so rather than printing a number it does not believe.
- **A/A calibration** (`tests/test_calibration.py`): coverage is measured, not
  assumed. ~95% of intervals contain zero when the truth is zero, checked at
  three designs and under between-task effect heterogeneity.
- **The four-verdict taxonomy** — REGRESSION / IMPROVEMENT / EQUIVALENT /
  INCONCLUSIVE, four distinct colours, and EQUIVALENT deliberately not grey.
  Every INCONCLUSIVE carries the tasks, trials and runs a conclusive answer
  would cost.
- **Minimum detectable effect and power calculator** (`ablate/stats/power.py`),
  shown before the confirm prompt so a run that cannot answer the question can
  be declined before it is paid for.
- **`ablate cost`** — per-section token cost, $0 and no network. The product's
  deterministic floor (ADR-0010): arithmetic, not an experiment, so it cannot
  come back inconclusive.
- **`ablate demo`** — replays a bundled run with no API key and no network, so
  the first thing anyone runs costs nothing. The bundled fixture is
  **synthetic** and labelled as such on every run; M4 replaces it with a real
  recording.
- **`ablate try`** — smallest real run, 3×3×2 on Haiku 4.5 under a $2 ceiling,
  labelled SMOKE TEST and honest that three tasks measure nothing.
- **`ablate report`** — re-analyze a stored run with a different `--margin` or
  `--alpha` at zero cost. Every re-analysis is counted and the count is printed:
  re-deciding a verdict after seeing the data is not protected by the stated α.
- **Cost controls** — pre-run estimate with recalibration from observed costs,
  `--max-spend` (default $25) enforced mid-run against real spend, `--cheap`,
  and a clean partial-results stop that stays analyzable because the queue is
  shuffled. Under `--yes`/CI, breaching the ceiling is a hard failure, never a
  silent continue.
- **SQLite trial index and memoization** (`--fresh` to bypass): editing one
  config section re-runs only the arm whose hash changed. The index is a
  derived cache, rebuildable from the JSONL logs, which remain the source of
  truth (ADR-0006).
- **Run manifests** beside each `trials.jsonl`, so a report can be regenerated
  after the config on disk has moved on.
- Markdown section parsing (`ablate/config/parse.py`) and deterministic token
  estimation (`ablate/config/tokens.py`).

### Fixed

- `--model` and `--cheap` were never passed to the agent. They changed the cost
  *estimate* while every trial still ran on the CLI's default model — so
  `--cheap` quietly billed at full rates. The model is now sent to the adapter
  and recorded on every trial, and a comparison that spans two models warns
  loudly instead of silently averaging two experiments.

### Changed

- The terminal report was rewritten to `docs/08-surfaces.md`. The
  `[statistics: not yet implemented]` marker is gone.
- ADR-0005's "pooled binomial CIs are 2–3× too narrow" claim was amended after
  being measured. It holds for a single arm's pass rate (2.1×) and is
  overstated for the paired difference (1.05–1.33×, depending on between-task
  heterogeneity). Both measured figures are now quoted; see the ADR for why
  the decision is unchanged.
- The landing page moved out of `main`. `web/` shipped inside this repo
  through v0.1.0; it is not on `main` anymore. Its source and history are on
  the `landing-page` branch.

  *(Corrected 2026-07-27: this entry originally said the page had been split
  into `ablate-ai/web` via `git subtree split`. No such repo exists, and
  `ablate-ai` turned out to belong to someone else entirely — see the
  Unreleased section.)*

### Not yet implemented

Section-level ablation (`ablate ablate`, `ablate apply`) is M3. The markdown
and JUnit report formats, the GitHub Action and the Codex adapter are M4 —
see `docs/09-milestones.md`.

## [0.1.0] - 2026-07-26

Milestone 1 (`docs/09-milestones.md`): proves the hard, boring part works —
isolation is real, config swap changes agent behavior, telemetry parses,
trials are reproducible enough to compare.

### Added

- Task schema and YAML loader, with all five assertion types (`exit_code`,
  `files_touched`, `present`/`absent`, `lint`, `script`).
- `git worktree`-per-trial isolation with locked checkout/teardown under
  concurrency.
- Claude Code adapter: `--bare` + `--append-system-prompt-file`, so the
  config under test is the only config in play.
- Trial runner and scheduler: `(task × variant × replicate)` expanded into
  a shuffled queue, run under a concurrency semaphore.
- Append-only JSONL trial log.
- `ablate init` — detects the test command and config file, scaffolds three
  template tasks (bug-fix, convention-adherence, restraint).
- `ablate compare A B --trials N` with a point-estimate terminal report,
  explicitly labelled `[statistics: not yet implemented]`.
- Landing page source (`web/`).

### Not yet implemented

Statistics (paired cluster bootstrap, confidence intervals, the four-verdict
taxonomy), section-level ablation, the SQLite index, cost controls, and the
Codex adapter are M2–M4 — see `docs/09-milestones.md`. `ablate compare`'s
report is a point estimate on purpose, not a placeholder for a bug.

[Unreleased]: https://github.com/ablate-dev/ablate/compare/v0.4.0...HEAD
[0.4.0]: https://github.com/ablate-dev/ablate/compare/v0.3.0...v0.4.0
[0.3.0]: https://github.com/ablate-dev/ablate/compare/v0.2.0...v0.3.0
[0.2.0]: https://github.com/ablate-dev/ablate/compare/v0.1.0...v0.2.0
[0.1.0]: https://github.com/ablate-dev/ablate/releases/tag/v0.1.0
