# 12. Open questions

Ordered by how much the answer changes the plan. Q1–Q4 should be resolved before or during
milestone 1; Q5 onward are yours to answer.

---

### Q1 — Does `codex exec --json` report token usage or cost? — **RESOLVED**

**Status: resolved 2026-07-27 against codex-cli 0.145.0. Usage yes, cost no.** I originally checked the OpenAI non-interactive docs and the DeepWiki reference for
`codex exec`; neither documents the fields of `turn.completed`, and neither confirms token or
cost reporting. I also could not verify a working-directory flag (`--cd`/`-C`) — mitigated by
using the subprocess `cwd`, which is more robust anyway.

**Why it matters.** If Codex doesn't report tokens, the primary metric ([ADR-0010](adr/0010-primary-metric.md))
is unavailable for the Codex arm, and cross-vendor comparison — your main anti-displacement
moat — degrades to pass rate only, which [05-statistics.md](05-statistics.md) says you can't
resolve at affordable N.

**How to resolve:** run `codex exec --json "say ok"` and read the events. Five minutes with the
binary installed. Do this before committing to M4.

**If the answer is no:** either derive token counts client-side by tokenizing the reconstructed
transcript from `item.completed` events (approximate, but the *ratio* between arms is what
matters and systematic error cancels in a paired design), or fall back to turn count and wall
time as the Codex primary metrics. Do not print an estimated dollar figure for Codex.

> **Resolved 2026-07-27, by running it.** `codex exec --json "say ok"`:
>
> ```
> {"type":"thread.started","thread_id":"019fa2b5-..."}
> {"type":"turn.started"}
> {"type":"item.completed","item":{"id":"item_0","type":"error","message":"Skill descriptions were shortened..."}}
> {"type":"item.completed","item":{"id":"item_1","type":"agent_message","text":"ok"}}
> {"type":"turn.completed","usage":{"input_tokens":16138,"cached_input_tokens":5504,
>  "cache_write_input_tokens":0,"output_tokens":5,"reasoning_output_tokens":0}}
> ```
>
> **Usage is reported; cost is not.** So the primary metric (ADR-0010) *is*
> available for Codex arms and cross-vendor comparison does not degrade to pass
> rate. `cost_usd` stays `None`, and `--max-spend` reports the trials it could
> not account for rather than treating them as free.
>
> Five minutes with the binary found four bugs that were not guessable from the
> docs, which is the whole argument for resolving open questions empirically:
>
> 1. `cache_write_input_tokens` was not among the spellings the parser accepted.
> 2. `reasoning_output_tokens` is reported separately and is billed as output.
>    Dropping it understated the primary metric on every trial.
> 3. `agent_message` text is nested under `item`, so the result was never captured,
>    and every `item.completed` was counted as a tool call.
> 4. **The dangerous one.** A routine "skill descriptions were shortened" notice
>    arrives as `item.type == "error"` on every single run. The parser treated
>    error items as fatal, which would have marked *every* Codex trial failed —
>    a bug that would have presented as a config effect.
>
> Two operational findings, also not in any doc: `codex exec` reads stdin and
> blocks if it is left attached, and it refuses untrusted directories. Git
> worktrees are accepted, which is what this tool needs.
>
> Still open: no `model` field appears in the stream, so the mixed-model warning
> has nothing to check on a Codex arm.

---

### Q2 — Is `--bare` + `--append-system-prompt-file` behaviourally equivalent to a real `CLAUDE.md`? *(blocks the whole measurement's validity)*

`--bare` is what guarantees no ambient config leaks into a trial, but it also skips `CLAUDE.md`
loading, so the config must be injected another way. An appended system prompt is not
byte-identical to how Claude Code renders `CLAUDE.md` (position in the prompt, surrounding
framing, whether it's marked as project memory).

**Why it matters.** If the two differ materially, `ablate` measures the effect of *a system-prompt
appendix* and reports it as the effect of *your CLAUDE.md*. That would be a correctness bug
disguised as a design choice, and the launch artifact would be built on it.

**How to resolve:** in M1, run one task ×15 trials under both mechanisms and compare pass rate,
tokens and turns. Equivalent → keep `--bare`. Not equivalent → invert the default: don't use
`--bare`, write `CLAUDE.md` to the worktree, and neutralize ambient config by pointing `HOME`
and `CLAUDE_CONFIG_DIR` at a scratch directory per trial.

**I have a mild preference for the second approach regardless**, because it measures the real
mechanism. `--bare` is cleaner in theory; on-disk is more faithful. This is worth your judgement.

---

> **Update 2026-07-27 — the experiment is now runnable, and has not been run.**
> `tools/q2_injection_equivalence.py` settles this for about the price of a
> `compare`: it runs the **same config** through both delivery mechanisms and
> asks the tool's own paired comparison whether they differ.
>
> ```
> python tools/q2_injection_equivalence.py --trials 5
> ```
>
> `ClaudeCodeAdapter(injection=...)` is the mechanism: `"append"` is the
> production path (`--bare` + `--append-system-prompt-file`), `"native"` drops
> both and lets the CLI discover the config the worktree already contains.
> Exit code is 0 for equivalent, 1 for measurably different, 2 for unsettled.
>
> Note the confound the script also prints: arm B has no `--bare`, so ambient
> user config can leak into it. Run it from a clean checkout with no
> `~/.claude` customizations, or the difference measured may be your dotfiles.
>
> **Until this runs, treat every `ablate` result as measuring a system-prompt
> appendix.** That may well be the same thing as a `CLAUDE.md` — but it is
> currently an assumption, and this project does not otherwise ship those.

---

### Q3 — Should the default primary metric be tokens, or should it be user-selectable per task?

[ADR-0010](adr/0010-primary-metric.md) makes tokens primary because it's the only metric with an
effect size the budget can see. But "my config saves tokens" is a weaker pitch than "my config
makes the agent better," and a user who came for correctness may feel bait-and-switched.

**Options:** (a) tokens primary always, pass rate as guardrail — my recommendation; (b) both
co-primary, shown side by side with equal weight and no designated headline; (c) user picks per
suite.

**I need your call**, because it determines the report's visual hierarchy and the README's first
sentence — both of which are hard to change after launch. My recommendation is (b) in the report
UI and (a) in the marketing: show both with equal prominence, but let the *headline claim* in
the README and the launch post be the token one, because it's the one that survives scrutiny.

---

### Q4 — What is the equivalence margin, and who picks it?

Default ±5pp is a guess. It has no principled basis, and it single-handedly determines how often
the tool says EQUIVALENT (deletable) versus INCONCLUSIVE (unresolved) — which is the difference
between the tool being useful and the tool being a shrug.

**The honest problem:** a margin the user picks after seeing the interval is a forking path. A
margin the tool picks is arbitrary.

**Options:** fixed ±5pp with `--margin` documented as "set before you look"; or derive it from
the observed run-to-run variance of the *same* config against itself (an A/A run) — principled,
defensible, and costs an extra arm. I lean toward offering `ablate compare --self-check` as an
optional A/A calibration that sets the margin empirically, defaulting to ±5pp otherwise. Your
call on whether that's worth the complexity in v1.

---

### Q5 — Who is the actual first user, and is it you?

The plan assumes "individual dev or small team already using a coding agent daily." That's a
description, not a person. The M1–M4 sequencing, the cost defaults, and whether the GitHub Action
matters at all depend on which of these is true:

- **Solo dev, own repo, curiosity-driven.** Then the Action is nearly worthless and `ablate cost`
  + `ablate compare` is the whole product. Cut M4's Action work.
- **Small team with a shared `CLAUDE.md` that keeps growing.** Then the Action is the product —
  it's the only thing that stops accretion — and it should be M3, not M4.
- **You, dogfooding, with the launch post as the actual goal.** Then optimize entirely for the
  artifact and the Action is a v2 concern.

These lead to materially different builds. **Which is it?**

---

### Q6 — Does agent-config sensitivity transfer across model tiers?

The cheap-screening stage ([07-ablation.md](07-ablation.md)) assumes a section that matters for
Haiku 4.5 also matters for Sonnet 5. That assumption is unverified and plausibly false — smaller
models may be *more* instruction-sensitive, which would make screening systematically biased
rather than just noisy.

The confirmatory stage contains the damage, but if transfer is poor the screening stage wastes
its budget on the wrong candidates. Resolvable empirically in M3 (run the same screen on both
models on one repo and correlate the βs) but only worth doing if the ablation costs bite.

---

### Q7 — Non-independence between trials via provider-side caching

Trials of the same task share a prompt prefix, so trial 2 hits a warm cache. That's a cost win
and possibly a *behavioural* confound if cache state affects sampling in any way (it shouldn't,
but I can't verify it doesn't). Also: the first trial of a task pays a cache write and is
systematically more expensive, which biases per-trial cost estimates if you don't account for it.

Low impact, but the cost accounting should report **mean cost excluding the first trial per
(task, arm)** as well as the raw mean, or the cost model in [06-cost.md](06-cost.md) will
overstate steady-state cost.

> **Update 2026-07-27 — the cost half is done; the behavioural half is not.**
> `Analysis.steady_state_cost_usd` is the mean with the first trial of each
> (task, arm) excluded, and the report prints it beside the raw total as
> `$X.XX/trial warm`. "First" is chronological rather than replicate 1, since
> the queue is shuffled and the trial that warms the cache is whichever ran
> first. `estimate_run` documents that it should be recalibrated on warm costs
> for the same reason.
>
> The behavioural half remains open and is probably not resolvable from
> outside: whether cache *state* affects sampling. The reason to think it does
> not is that a cache hit returns the same prefix computation, and the sampler
> still draws fresh from the same distribution — but that is a statement about
> how it ought to work, not an observation. If it were false, trials within a
> task would be more alike than the bootstrap's within-task resampling assumes
> and the intervals would be too narrow. Nothing in the current design would
> notice.

---

### Q8 — Apache-2.0 or MIT?

`probe-and-refine-tuning` (whose method you're building on) is MIT; AgentLinter is MIT. I'd
default to Apache-2.0 for the patent grant given this is adjacent to commercial vendors' feature
roadmaps, but MIT is more idiomatic in this corner of the ecosystem and reduces friction for the
interop conversations in [10-distribution.md](10-distribution.md). No strong view; pick one now
so it's in the first commit.

---

### Q9 — Do you want `.ablate/` in the user's repo or in a cache dir?

In-repo means task suites get committed and shared (good for teams, good for the sharing
convention in R3) but pollutes the repo and puts worktrees near the working tree. Out-of-repo
(`~/.cache/ablate/<repo-hash>/`) is cleaner but makes suites private and unshareable.

My lean: **task definitions in-repo (`.ablate/tasks/`, committed), everything else out-of-repo**
(worktrees, run logs, env cache in `~/.cache/ablate/`). That's a two-location design with a bit
more code and, I think, clearly the right ergonomics — but it's the kind of thing that's annoying
to change later, so decide before M1.
