# ADR-0007 — Randomized subset screening + ridge, not exhaustive leave-one-out

**Status:** accepted · **Date:** 2026-07-26

## Context

Given K config sections, attribute the config's effect to individual sections within a budget a
solo dev will actually spend. Cost is measured in agent runs at ~$0.10–0.50 each.

## Decision

Three stages ([07-ablation.md](../07-ablation.md)):

0. **Static token cost** per section — exact, zero variance, zero API calls.
1. **Randomized subset screening**: `R ≈ 2K + 8` variants with each section independently kept at
   p=0.5, one trial per (variant, task), fit `y ~ task fixed effects + Σ β_k · mask_k` with a
   ridge penalty. Run on the cheap model.
2. **Confirmatory paired A/B** at full N on the target model, for the top-|β| sections and the
   large-token/near-zero-β deletion candidates.

## Rejected: exhaustive leave-one-out

The obvious design, and what the original brief described: K+1 variants, drop one section each.

Rejected because each trial informs exactly one β. With `R = K+1` variants you spend your whole
budget getting K independent, low-precision estimates, and you learn nothing about interactions
(a section that only matters when another is absent is invisible). The randomized design uses
every trial to inform every β, is a standard fractional-factorial screening design with a
century of theory behind it, and at K=12 costs less for more information
([06-cost.md](../06-cost.md): ~$110 vs ~$182).

## Rejected: greedy / recursive group ablation (binary search)

Drop half the sections, recurse into whichever half moved the metric. Attractive `O(K log K)`
scaling.

Rejected because the greedy path is driven by *noisy* per-step estimates, so an early wrong turn
is unrecoverable and invisible — and it assumes effects are sparse and non-interacting, which is
the assumption most likely to be false in prose instructions. It also produces no uncertainty
estimate for the sections it pruned early, so the report couldn't distinguish "we showed this
doesn't matter" from "we stopped looking."

## Rejected: exact Shapley values

The theoretically correct attribution for interacting components. Requires `2^K` coalitions;
sampling approximations need far more evaluations than the screening design for a quantity most
users won't interpret. Ridge β on a p=0.5 random design is already a reasonable estimate of
average marginal contribution and is explicable in one sentence.

## Rejected: LASSO instead of ridge

LASSO's sparsity is superficially appealing ("it picks the sections that matter"). Rejected
because the interesting output here is the sections whose coefficient is *reliably near zero* —
the deletion candidates — and LASSO zeroes coefficients as a modelling choice, which destroys the
distinction between "measured to be zero" and "shrunk to zero." That distinction is the product.
Ridge keeps every coefficient with an honest interval.

## Consequences

- The stage-1 output is **screening-grade** and must be labelled as such everywhere it appears.
  Ranked candidates for follow-up, not verdicts.
- Design-matrix conditioning must be checked, not assumed: reject and resample any design where a
  section is constant across variants or where max pairwise |corr| > 0.5. A collinear design
  produces confidently wrong βs.
- The ridge penalty λ is chosen by leave-one-task-out CV. It biases coefficients toward zero,
  which is *conservative in the right direction* for the "is it safe to delete" question but
  should be disclosed in the method section.

  > **Amended 2026-07-27, after implementing and measuring it.** The parenthetical above is
  > backwards. Shrinking β toward zero makes a section that *does* matter look **more** deletable,
  > not less — the dangerous direction, not the safe one. Simulation puts coefficients at ~75% of
  > true magnitude and coverage on true-null sections at ~85% rather than the nominal 95%
  > ([`tests/test_regression.py`](../../tests/test_regression.py)).
  >
  > What actually contains the risk is stage 2, and this sharpens why it is mandatory: a genuinely
  > important section that shrinkage flattens toward zero becomes a large-token/near-zero-β
  > *deletion candidate*, which is precisely the set promoted to a confirmatory A/B. The pipeline
  > is safe; the stated reason was not. Ridge is still right, for the LASSO reason below.
  >
  > Consequence for the report: rank by |β|, and never present the magnitudes as effect sizes.
- Cheap-model screening assumes cross-tier transfer of config sensitivity, which is unverified
  ([Q6](../12-open-questions.md)). The confirmatory stage contains the risk and is not optional.
- Cost is `O(K·T)` for screening plus `O(c·T·N)` for c confirmations — linear in K. That
  linearity is what makes the feature shippable at all.
