# 11. Risks and kill criteria

Each risk: what it is, the **early signal** you'd actually observe, and what you do.

## R1 — First-party displacement (Anthropic / OpenAI) — *high likelihood, high impact*

`skill-creator` already ships in Claude Code with Eval/Benchmark modes, an Executor/Grader/
Comparator/Analyzer agent split, blind A/B between versions, and a browser result viewer. It is
built in, free, and scoped to `SKILL.md`. Extending it to `CLAUDE.md` is a small product decision.

**Early signal.** Any of: a `CLAUDE.md` mode in skill-creator release notes; `claude` gaining an
eval or benchmark subcommand; a `/memory eval` command; Codex shipping an `AGENTS.md` A/B flow;
a first-party blog post about measuring config effectiveness.

**Response.** Do not try to out-feature them on the A/B loop — you will lose, because they can
put it behind a slash command in the tool people already have open. Retreat to the three things
a first-party tool is structurally unlikely to ship:
1. **Cross-vendor.** Anthropic will never run Codex on your suite. Make "same suite, both agents"
   a headline feature, which is why the Codex adapter stays in v1 despite the cost.
2. **Statistical conservatism.** A first-party tool has a commercial incentive to report that its
   config format helps. You have an incentive to be right. Lean on it hard.
3. **CI enforcement** on a repo you control, with a spend ceiling and an audit log.

If they ship the full thing including statistics and ablation: that is a kill signal (see below),
and the right move is to pivot `ablate` into the cross-vendor + CI layer on top of their runner,
or to shut it down cleanly and write the retrospective.

## R2 — Cost as an adoption ceiling — *high likelihood, medium impact*

A default run is ~$28 and a full ablation is ~$110. Most individual devs will not casually spend
$110 on a config question. Worse, the modal outcome of a cheap run is INCONCLUSIVE, so the user
pays and learns nothing actionable.

**Early signal.** High `ablate demo` / `ablate try` counts relative to `ablate compare`; issues
titled "can I run this on a cheaper model"; users running with `--trials 1`; abandoned runs at
the spend prompt.

**Response.**
- The `$0` entry points (`demo`, `cost`) are not a nicety, they are the funnel. `ablate cost` in
  particular gives real value — exact per-section token cost — for free, forever.
- Make the **estimate + projected MDE** appear before the confirm prompt so nobody spends money
  on a run that mathematically cannot answer their question. This is the most important
  anti-churn feature in the product.
- Push the cheap-screening / expensive-confirmation split everywhere.
- If the signal is strong: build a **shared task-suite registry** so users don't each pay to
  discover the same thing, and lean the product toward the free static analysis + one paid
  confirmatory run.

## R3 — Task authoring friction — *high likelihood, high impact*

The tool is worthless without a task suite, and writing good tasks is real work that happens
*before* any value is delivered. This is the classic eval-tool adoption killer.

**Early signal.** `ablate init` runs with no subsequent `ablate compare`; issues asking "what
should my tasks be"; suites in the wild containing only the three untouched templates.

**Response.**
- `ablate init` templates must be *runnable as-is* against the detected test command, not
  `TODO`-filled skeletons. A user should be able to `init` then `try` with zero editing, even if
  the resulting tasks are mediocre.
- The three archetypes (bug-fix / convention / restraint) are documented as recipes, with a
  worked example each.
- This is the trigger that promotes `ablate mine` from M5 to urgent. Defer the miner, but keep
  the design ready ([ADR-0008](adr/0008-task-miner-scope.md)) precisely so you can build it fast
  if this signal fires.
- Consider a `.ablate/tasks` sharing convention so a project can ship its suite and every
  contributor benefits from one authoring effort.

## R4 — Config edits genuinely have no measurable effect — *medium likelihood, existential*

arXiv 2605.10039 found no detectable effect of four config-structure variables across 1,650
Claude Code sessions. If the honest answer for most users' configs is "we can't tell," a tool
that says so repeatedly is correct and unloved.

**Early signal.** Across your own dogfooding plus the first ~10 external users: the modal verdict
on the *token* metric is also EQUIVALENT, not just on pass rate. That is the fatal version —
if the config doesn't even change token cost measurably, there is nothing to sell.

**Response.** This risk is why [ADR-0010](adr/0010-primary-metric.md) exists. Note the asymmetry:
- **Pass rate showing no effect is fine.** It is the expected result, it is the product (permission
  to delete), and the token metric carries the value.
- **Token cost showing no effect is impossible by construction** — config text is tokens, and
  `ablate cost` measures it exactly with no statistics at all. The floor of the product's value
  is a deterministic measurement that cannot come back null.

That floor is the reason to build this. Even in the worst world where nothing about agent
behaviour is measurable at affordable N, "your config costs 4,180 tokens per turn, here's the
per-section breakdown, and here is a rigorous demonstration that removing 1,581 of them changes
nothing we can detect" remains a true, useful, shippable product.

## R5 — Agent CLI churn breaks the adapters — *high likelihood, low-medium impact*

Both `claude` and `codex` ship frequently; flags, JSON schemas and defaults move. The docs I
verified on 2026-07-26 note that `--bare` "will become the default for `-p` in a future release."

**Response.** Pin and assert: `preflight()` records the agent version and warns on an untested
major; adapters parse defensively (unknown event types ignored, missing fields → `None`, never a
crash); every trial record stores `agent_version`; the report refuses to pool trials across
differing versions without `--allow-mixed-versions`. Add a scheduled CI job that runs `ablate try`
against latest `claude` and `codex` weekly, so you learn from a red build rather than a user.

## R6 — Non-determinism swamps everything / results don't replicate

Two runs of the same comparison give materially different verdicts, and someone posts about it.

**Early signal.** Your own back-to-back runs disagreeing on verdict. Test whether: run the same
comparison three times in M2 and check verdict stability.

**Response.** This is a *coverage* question and is answerable empirically: run the same comparison
20 times against a known-null config change and confirm ~95% of intervals contain zero. Publish
that calibration check. If coverage is bad, the bootstrap is misspecified (most likely: an
unmodelled variance component like time-of-day model load) and the fix is a stratification, not a
wider interval.

## R7 — Safety: running untrusted agents in a worktree

The runner executes an agent with `acceptEdits` and a Bash allowlist in a git worktree on the
user's machine. Worktrees isolate the filesystem *within the repo*, not the machine.

**Response.** Documented prominently, not solved in v1: prefer `--allowedTools` over
`--dangerously-skip-permissions`; the CI path runs in a disposable runner; a `--sandbox` flag
that shells out to a container is a v2 feature. In CI, refuse to run on fork PRs by default —
otherwise any fork can spend your API budget and execute code as your workflow.

## R8 — Legal/reputational blowback from the launch artifact

Publicly ablating someone else's repo config and concluding "60% of it is useless" can read as an
attack.

**Response.** [10-distribution.md](10-distribution.md): your own repo is subject #1; the
third-party subject is contacted before publication and offered co-signing or anonymization.

---

## Kill criteria

Abandon this and build something else if **any** of these become true.

1. **First party ships the whole thing.** Anthropic (or OpenAI) ships `CLAUDE.md`/`AGENTS.md`
   A/B *with repeated trials, aggregated statistics, and section-level attribution*, built into
   the CLI. Not "they ship an eval mode" — that's R1 and survivable. Ships-with-statistics is
   terminal for the differentiator.
2. **Six weeks post-launch: fewer than 5 people have run `ablate compare` against a real repo
   more than once.** Repeat use by a handful of strangers is the minimum viable signal that the
   output changes behaviour. Stars, demo runs and HN points do not count.
3. **The token metric comes back EQUIVALENT for the majority of real configs you test.** This
   would mean the deterministic floor from R4 isn't there either, and there is nothing to sell.
4. **You cannot get verdict stability.** If re-running the same comparison flips the verdict more
   than ~1 time in 20 after M2's calibration work, the instrument doesn't measure anything and
   shipping it is worse than not shipping it. Fix it or stop; do not ship a plausible-looking
   number you don't trust.
5. **Milestone 1 exceeds 35 hours.** If the isolation + adapter layer takes double its estimate,
   the four-weekend budget is gone and the honest options are to descope to "static token-cost
   analyzer only" (which is `ablate cost`, a genuinely useful weekend project) or to stop.

If you stop, ship `ablate cost` as a standalone 200-line tool and write the retrospective. A
precise, free, deterministic per-section token-cost analyzer for agent config files is a real
contribution and costs one weekend, and it is the correct minimum viable version of this idea.
