# 13. LLM use, auth, and bare mode

This note summarizes the practical distinction between what `ablate` does itself and what the
external coding agent does.

## Short answer

`ablate` is not an LLM product by itself. It is an experiment runner.

The LLM is used inside the external coding agent, such as Claude Code or Codex. `ablate` prepares
the experiment, runs the agent, records telemetry, checks deterministic assertions, and computes
statistics.

`ablate` does **not** use an LLM to judge, score, summarize, or decide whether a task passed.

## What `ablate` does

For a real comparison, `ablate`:

1. Creates isolated trial worktrees.
2. Writes a config variant into each worktree.
3. Invokes the configured agent, such as `claude` or `codex`.
4. Captures telemetry such as tokens, turns, cost, wall time, files touched, and exit status.
5. Runs deterministic assertions, such as tests, scripts, regex checks, or file checks.
6. Computes statistical intervals and verdicts.

The agent is the part that calls an LLM. `ablate` is the harness around it.

## When users need auth

Users need an authenticated agent CLI for commands that run new trials:

- `ablate try`
- `ablate compare`
- `ablate ablate`

Users do not need an API key for commands that do not call an agent:

- `ablate demo`
- `ablate cost CLAUDE.md`
- offline parsing, reporting, or re-analysis of existing run logs

The product wording should be careful here:

> Real trials require an authenticated Claude Code or Codex CLI. Depending on the agent and
> isolation mode, that may mean an API key, an API key helper, or an existing local login session.

## Why an API key may still be needed

Many users already have Claude Code or Codex configured locally. Some may use a normal logged-in
subscription flow rather than an API key. In principle, if the agent CLI can run headlessly using
that existing login, `ablate` should not require a separate API key.

The complication is isolation.

For Claude Code, the planned invocation uses `--bare` so the experiment does not accidentally load
uncontrolled local configuration. Under `--bare`, Claude Code may skip the normal config and auth
discovery paths. In that case, the subprocess still needs explicit auth, such as:

- `ANTHROPIC_API_KEY`
- an `apiKeyHelper` supplied through `--settings`

This is not because `ablate` itself needs an LLM key. It is because the isolated Claude Code
process still has to authenticate with Anthropic.

The Codex adapter is documented as unverified against the real binary. Before making a hard
product claim for Codex, test whether `codex exec --ephemeral --ignore-user-config --ignore-rules`
can use the user's existing authenticated session.

## What `--bare` mode means

`--bare` is a Claude Code mode intended for headless or scripted use where fewer hidden local
inputs are loaded automatically.

In the `ablate` design, `--bare` is used to make trials cleaner. The goal is for the only meaningful
difference between two trial arms to be the config variant that `ablate` deliberately injects.

Without isolation, Claude Code may automatically discover and load local context that is outside
the experiment.

## Ambient config

"Ambient config" means configuration that exists around the user or repository, but is not the
specific config file currently being tested.

Examples include:

- global user `CLAUDE.md`
- project `CLAUDE.md`
- Claude Code settings
- hooks
- skills
- plugins
- MCP server config such as `.mcp.json`
- local defaults discovered by the agent CLI
- auth or config stored in the user's agent environment

This ambient config is useful in normal interactive coding. It makes the agent behave according to
the user's local setup.

For `ablate`, it can corrupt the measurement.

## What "leaking" means

Leaking means hidden local configuration influences both sides of the experiment.

Suppose `ablate` is trying to measure:

```text
full CLAUDE.md
vs
no CLAUDE.md
```

If Claude Code still automatically loads a global `CLAUDE.md`, project MCP config, hooks, skills,
or other settings, then the real experiment becomes:

```text
full CLAUDE.md + hidden local config
vs
hidden local config
```

Or:

```text
variant A + hidden instructions + hidden tools
vs
variant B + hidden instructions + hidden tools
```

That hidden influence is the leak. The result may still be useful in some cases, but it is no
longer a clean measurement of the config file under test.

## How `ablate` avoids leakage

The design uses `--bare` and then explicitly injects the config variant:

```bash
claude --bare -p "$TASK_PROMPT" \
  --append-system-prompt-file "$VARIANT" \
  --output-format json \
  --permission-mode acceptEdits \
  --allowedTools "Bash,Read,Edit,Write,Glob,Grep" \
  --max-turns 40
```

The config is also written into the worktree so code or scripts that read it from disk can still see
the expected file.

## Tradeoffs

There are two main tradeoffs.

First, auth may become less convenient. Existing interactive auth might not be visible in `--bare`
mode, so an API key or `apiKeyHelper` may be required.

Second, realism may be affected. Injecting a `CLAUDE.md` file as an appended system prompt may not
be byte-identical to how Claude Code normally loads `CLAUDE.md`. That is already tracked as an open
question in [12-open-questions.md](12-open-questions.md).

The decision is a product tradeoff:

- use isolation to get cleaner experiments;
- or use normal discovery to better match day-to-day user behavior.

The current design starts with cleaner experiments, then calls out the realism check as something
that must be validated empirically.
