# ADR-0004 — No LLM-as-judge in v1

**Status:** accepted (hard line, not "not yet") · **Date:** 2026-07-26

## Context

Every comparable tool offers one. promptfoo has `llm-rubric`; Anthropic's `skill-creator` uses a
blind LLM Comparator as its primary A/B mechanism. Judges are attractive because they assert on
things deterministic checks can't reach: "did it follow the house style," "is this a good
implementation."

## Decision

`ablate` ships five deterministic assertion types and no LLM judge. The `script` type is the
escape hatch for anything else, and a user who wants a judge can shell out to one from `script` —
but it will never be a first-class type, and never contribute to a headline number.

## Rationale

**The statistical argument, which is decisive.** [05-statistics.md](../05-statistics.md) puts the
minimum detectable effect at ~20pp on pass rate at 200 runs. A judge with even 10% grading noise
inflates the variance of the outcome variable, widening intervals and pushing more comparisons
into INCONCLUSIVE — in a product whose entire value proposition is producing a *usable* verdict.
Spending money to make your instrument less sensitive is a strange thing to do.

**The bias argument, which is worse.** Judge error is not independent of the treatment. A config
that makes the agent write more explanatory prose may make the judge rate it higher regardless of
correctness. That is a *correlated* error term, and unlike noise it doesn't average out with N —
it biases Δ̂ in an unknown direction that no amount of sampling fixes. There is no clean way to
detect this from inside the tool.

**The positioning argument.** The tool's differentiator is that its numbers can be checked. Every
`ablate` assertion is a command you could run yourself and get the same answer. That property is
worth more than assertion breadth, and it is destroyed the moment one number in the report came
from a model's opinion.

## Rejected: judge as an opt-in, clearly-labelled secondary metric

The most reasonable counter-proposal, and I could be argued into it for v2. Rejected for v1
because "clearly labelled" does not survive contact with a screenshot: the number would end up in
someone's blog post without the caveat, and the tool's credibility travels with it. Also, once
it exists it becomes the path of least resistance for users who don't want to write assertions —
which is most users.

## Consequences

- Some genuinely interesting questions are out of scope ("does my style guide section make the
  code more idiomatic?"). Accept this; the answer is a better `script` assertion or a lint rule.
- Task authoring is harder, which feeds [R3](../11-risks.md). Mitigated by making `init`
  templates runnable as-is and by documenting the three archetypes.
- This is a **README-worthy statement**, not an implementation detail. It goes in the "How it
  decides" section as a positive claim.
