# 3. Architecture

## Language and distribution

**Python 3.11+, distributed as a PyPI package designed to be run via `uvx ablate`.**
Rationale and rejected alternatives: [ADR-0001](adr/0001-language-and-distribution.md).

The five-minute promise, concretely:

```bash
uvx ablate init          # detects test command, writes .ablate/ with 3 template tasks
uvx ablate try           # runs the smallest possible real comparison, ~$2, ~6 min
```

No `pip install`, no virtualenv, no signup. `uvx` fetches and runs in one step. Users without
`uv` get `pipx run ablate` and `pip install ablate` as documented fallbacks.

Runtime dependencies, deliberately few: `click` (CLI), `rich` (terminal report), `pydantic`
(schemas), `numpy` (bootstrap), `PyYAML`. **No `scipy`** — the bootstrap and the ridge fit are
30 lines of numpy each and scipy is a heavy, slow-to-resolve dependency that would visibly hurt
the `uvx` cold start. **No `pandas`.**

## Component map

```
ablate/
  cli.py            # click entrypoints
  config/
    parse.py        # markdown -> Section[] (headings, line ranges, token counts)
    variant.py      # Section[] + keep-mask -> rendered config text
  runner/
    worktree.py     # git worktree lifecycle, env cache
    schedule.py     # (task x config x trial) -> queue, concurrency, budget guard
    trial.py        # one trial: checkout -> swap config -> invoke -> capture
  agents/
    base.py         # AgentAdapter protocol
    claude_code.py
    codex.py
  assert/
    types.py        # exit_code, files_touched, regex, lint, script
    run.py
  stats/
    bootstrap.py    # paired cluster bootstrap, MDE, power
    ablation.py     # screening design + ridge attribution
  store/
    log.py          # append-only JSONL
    index.py        # SQLite index over the log
  report/
    terminal.py
    markdown.py
    junit.py        # for the GitHub Action
  mine/             # experimental, milestone 5
```

## Runner design

### Isolation: `git worktree` per trial

Decision and alternatives: [ADR-0002](adr/0002-trial-isolation.md).

```
repo/                          # user's repo, never touched
.ablate/
  worktrees/
    t01-cfgA-r3/               # one worktree per trial, detached at task base_commit
  env-cache/
    <base_commit>-<lockhash>/  # node_modules, .venv, etc. hardlinked into worktrees
  runs/
    2026-07-26T14-02-11Z/
      trials.jsonl
      traces/<trial_id>.jsonl
```

Per-trial lifecycle:

1. `git worktree add --detach .ablate/worktrees/<trial_id> <task.base_commit>`
2. Materialize the environment: if `.ablate/env-cache/<key>` exists, hardlink/copy it in;
   otherwise run `task.setup` once and populate the cache. This is the SWE-bench `--cache_level`
   idea, borrowed wholesale — environment setup dominates non-LLM wall time.
3. Write the config variant to `task.config_path` inside the worktree.
4. Invoke the agent with `cwd` = the worktree.
5. Capture: `git diff`, files touched, agent trace, token/turn/cost counters, wall time.
6. Run assertions inside the worktree.
7. `git worktree remove --force` (or keep, under `--keep-worktrees`, for debugging).

Worktrees are cheap (one shared object database, no re-clone) and give perfect filesystem
isolation for the thing agents actually mutate. They do **not** isolate the network, installed
system packages, or the ability of a hostile agent to `rm -rf ~`. That is a documented limitation
plus a loud warning in `ablate init`, not something v1 solves.

### Config swap

The config is swapped by **writing the file on disk inside the worktree**, not by passing flags.

Reasons: it is vendor-neutral (every agent reads its config from the repo); it is exactly what
the user's real environment does, so you're measuring the real mechanism; and it composes with
ablation (a variant is just different file bytes).

The "config B is nothing" case is `--config-b /dev/null`, which deletes the file in the worktree.

**Critically: pair the on-disk swap with the agent's own config-discovery suppression.** For
Claude Code that is `--bare`, which skips auto-discovery of hooks, skills, plugins, MCP servers,
auto-memory, and `CLAUDE.md`; the config under test is then injected explicitly. Without this,
the user's `~/.claude/CLAUDE.md` and any project `.mcp.json` leak into every trial and you are
measuring a mixture. For Codex the equivalents are `--ignore-user-config` and `--ignore-rules`.

Because `--bare` skips `CLAUDE.md`, the config under test is delivered via
`--append-system-prompt-file <path-to-variant>` rather than by writing `CLAUDE.md` — with the
file also written into the worktree so that anything reading it from disk sees it too.

> **Open question ([12](12-open-questions.md), Q2):** injecting the config as an appended system
> prompt is not byte-identical to how Claude Code loads `CLAUDE.md` in a normal session (position
> in the prompt, surrounding framing). If those differ materially, measured effects may not
> transfer to real usage. This needs an empirical check in milestone 1: run the same task with
> (a) `--bare` + `--append-system-prompt-file`, (b) no `--bare` and `CLAUDE.md` on disk, and
> confirm the outcomes are comparable. If they are not, invert the default: don't use `--bare`,
> and instead neutralize ambient config by pointing `HOME` at a scratch dir.

### Agent invocation abstraction

[ADR-0003](adr/0003-agent-invocation.md): subprocess CLI adapters, not SDK bindings.

```python
class AgentAdapter(Protocol):
    name: str
    def preflight(self) -> PreflightResult: ...      # binary present? version? auth?
    def build_command(self, t: TrialSpec) -> list[str]: ...
    def parse_events(self, stdout: Iterable[str]) -> TrialTelemetry: ...
```

`TrialTelemetry` is the normalized contract every adapter must fill:

| Field | Claude Code source | Codex source |
|---|---|---|
| `input_tokens`, `output_tokens`, `cache_read`, `cache_write` | `usage` in the `result` message | **unverified** — see below |
| `cost_usd` | `total_cost_usd` in `--output-format json` | **unverified** |
| `turns` | `num_turns` | count of `turn.completed` events |
| `tool_calls` | `tool_use` blocks in `stream-json` | `item.completed` events of tool type |
| `session_id` | `session_id` | `thread.started` |
| `exit_status` | process exit code | process exit code |

**Claude Code — verified flags** (from `code.claude.com/docs/en/headless`, fetched 2026-07-26):
`-p/--print`, `--output-format text|json|stream-json`, `--bare`, `--allowedTools`,
`--permission-mode` (`dontAsk`, `acceptEdits`), `--max-turns`, `--append-system-prompt`,
`--append-system-prompt-file`, `--system-prompt`, `--settings <file-or-json>`,
`--mcp-config`, `--agents`, `--json-schema`, `--verbose`, `--include-partial-messages`,
`--continue`, `--resume <session_id>`, `--dangerously-skip-permissions`. JSON result payload
includes `result`, `session_id`, `total_cost_usd`, a per-model cost breakdown, and `num_turns`.
Under `--bare`, auth must come from `ANTHROPIC_API_KEY` or an `apiKeyHelper` in `--settings`;
OAuth/keychain is skipped. SIGTERM aborts the turn and exits 143.

Baseline invocation:

```bash
claude --bare -p "$TASK_PROMPT" \
  --append-system-prompt-file "$VARIANT" \
  --output-format json \
  --permission-mode acceptEdits \
  --allowedTools "Bash,Read,Edit,Write,Glob,Grep" \
  --max-turns 40
```

`--permission-mode acceptEdits` plus an explicit `--allowedTools` list is preferred over
`--dangerously-skip-permissions`: unattended runs never block on a prompt, but a trial that tries
something outside the allowlist aborts loudly instead of silently doing it.

**Codex — verified flags** (from `learn.chatgpt.com/docs/non-interactive-mode`, fetched
2026-07-26): `codex exec "<prompt>"`, `codex exec -` (prompt on stdin), `--json` (JSONL:
`thread.started`, `turn.started`, `turn.completed`, `item.started`, `item.completed`),
`-o/--output-last-message <path>`, `--output-schema <path>`, `--sandbox
read-only|workspace-write|danger-full-access`, `--ephemeral`, `--ignore-user-config`,
`--ignore-rules`, `--skip-git-repo-check`, `-c <config>`, `--full-auto`,
`codex exec resume --last|<SESSION_ID>`. Progress goes to stderr; only the final agent message
goes to stdout.

Baseline invocation:

```bash
codex exec --json --ephemeral --ignore-user-config --ignore-rules \
  --sandbox workspace-write "$TASK_PROMPT"
```

**Two verified-negative facts about Codex, both of which are risks:**

1. I could **not** verify a working-directory flag (`--cd`/`-C`). Mitigation: set the subprocess
   `cwd` to the worktree. This is more robust than a flag anyway and works for both adapters, so
   make `cwd` the universal mechanism and never depend on a directory flag.
2. I could **not** verify that the `--json` stream carries token usage or cost. If it does not,
   the Codex adapter must estimate cost from token counts it derives itself, or report cost as
   unavailable. **Do not fabricate a cost number for Codex.** Print `cost: n/a (not reported by
   codex exec)` and let the token/turn metrics carry the comparison. This is tracked as
   [Q1](12-open-questions.md) and must be resolved by running the binary before milestone 4.

### Scheduling and the budget guard

The scheduler expands `(task × config-variant × trial)` into a work queue, shuffles it (so a mid-
run abort doesn't systematically starve one arm — important for the statistics), and executes
under a semaphore.

Before the first trial: estimate total cost, print it, and if it exceeds `--max-spend`
(default `$5`) prompt for confirmation. **During** the run, maintain a running spend total from
each trial's reported cost and hard-stop when the ceiling is hit, writing a partial-results
marker. A partial run is still analyzable — the shuffle guarantees the completed trials are a
roughly balanced sample of the design.

## Result storage and re-analysis

[ADR-0006](adr/0006-result-storage.md): append-only JSONL as the source of truth, SQLite as a
derived index.

One line per trial in `runs/<run_id>/trials.jsonl`:

```json
{
  "trial_id": "t03-cfgA-r2",
  "run_id": "2026-07-26T14-02-11Z",
  "task_id": "fix-null-deref",
  "variant_id": "A",
  "variant_hash": "sha256:9f2c...",
  "section_mask": [1,1,1,1,1,1,1,1],
  "replicate": 2,
  "agent": "claude-code",
  "agent_version": "2.1.220",
  "model": "claude-sonnet-5",
  "started_at": "...", "duration_s": 74.2,
  "telemetry": {"input_tokens": 184203, "cache_read": 171000, "output_tokens": 9120,
                "turns": 7, "tool_calls": 19, "cost_usd": 0.121},
  "assertions": [{"id": "tests-pass", "type": "exit_code", "passed": true},
                 {"id": "no-new-deps", "type": "files_touched", "passed": true}],
  "passed": true,
  "diff_sha": "sha256:11ab...",
  "error": null
}
```

Properties this buys:

- **Re-analysis without re-running.** `ablate report <run_id>`, `ablate report --alpha 0.1`,
  `ablate ablate --from <run_id>` all read the log. Every statistical knob is a post-hoc
  decision. This is the single most cost-relevant architectural choice in the document.
- **Incremental re-runs.** Trials are keyed by `(task_id, variant_hash, agent, model, replicate)`.
  `ablate run` looks up that key in the SQLite index; a hit is reused unless `--fresh`. Edit one
  section of your config and only the variants whose hash changed get re-executed.
- **Crash resumption** falls out of append-only-ness for free.
- **Auditability.** The launch artifact ([10-distribution.md](10-distribution.md)) is only
  credible if you can publish the raw log and let people re-run the analysis.

The SQLite index (`store/index.db`) holds `(key) -> (run_id, offset)` plus denormalized columns
for fast filtering. It is fully rebuildable from the JSONL; deleting it is never data loss.

Raw agent traces go to `traces/<trial_id>.jsonl` (gzipped), referenced by path, not inlined —
they are large and rarely needed.
