# Verdict: build it, but change the headline claim

You asked me to push back where I think you're wrong. Here is the pushback, up front, because
it propagates into every other document.

## Finding 1 — the effect you want to measure is smaller than your budget can see

Three 2026 papers measured exactly what your tool measures.

- **Gloaguen et al., ETH Zürich** ("Evaluating AGENTS.md: Are Repository-Level Context Files
  Helpful for Coding Agents?", arXiv 2602.11988) ran four coding agents across SWE-bench Lite
  (300 tasks) and a purpose-built 138-repo benchmark. LLM-generated context files **reduced**
  task success by roughly 3% (−0.5pp on SWE-bench Lite, −2pp on their own benchmark) and raised
  inference cost by >20%. Human-written files gained roughly 4%.
- **arXiv 2605.10039** ran a factorial study of four config-file structure variables across
  **1,650 Claude Code CLI sessions / 16,050 function-level observations** and found *no
  detectable effect on instruction adherence for any variable after multiple-testing correction.*
- **arXiv 2601.20404** measured AGENTS.md presence across 10 repos / 124 PRs and found a
  **28.6% median runtime reduction and 16.6% output-token reduction**, with comparable task
  completion.

Read those together. The pass-rate effect of config changes is in the **single-digit percentage
points**, and one well-powered study at N=1,650 sessions found nothing at all. The **cost**
effect is in the **tens of percent** and shows up at N=124.

Now the arithmetic (derived in [05-statistics.md](05-statistics.md)). A paired design with
20 tasks × 5 trials per arm — 200 agent runs, well past what a solo dev will pay for casually —
has a minimum detectable effect of roughly **20 percentage points** on pass rate at 80% power.
You cannot see a 3pp effect. Neither can anyone else at that budget.

If you ship a tool whose headline number is a pass-rate delta, at realistic N it will print
"indistinguishable from noise" almost every time, and the users who don't understand why will
conclude the tool is broken.

**What I recommend instead:** make **token and turn cost the primary metric** and pass rate the
guardrail. Cost is continuous (not Bernoulli), has 5–10× the effect size, and needs an order of
magnitude fewer trials for the same power. The product claim becomes:

> "These 340 lines of your CLAUDE.md cost 4,100 tokens on every turn. Deleting them changes your
> pass rate by −1pp ± 9pp — indistinguishable from noise. Here is the evidence. Delete them."

That is a *non-inferiority* claim, it is statistically honest at N you can afford, it matches the
literature, and it is a better product: nobody's actual problem is "my CLAUDE.md might be
slightly wrong," it's "my CLAUDE.md is 400 lines of superstition I'm afraid to delete." You are
selling **permission to delete**, and permission to delete is licensed by a tight CI around zero
plus a big token number — not by a positive pass-rate delta.

See [ADR-0010](adr/0010-primary-metric.md).

## Finding 2 — the headline feature already exists as research code

`asashepard/probe-and-refine-tuning` (MIT, arXiv 2606.20512) implements iterative
per-line repository-guidance ablation with McNemar tests and mixed-effects models. It is
research replication code: SWE-bench Verified only, requires Docker plus a self-hosted
OpenAI-compatible inference endpoint, tested with Qwen3.5-35B, 4 stars, 0 forks, documented for
reproducing paper tables rather than for use on your own repo.

This does not kill your project. It validates the idea, it gives you a citation, and its
statistical methodology is better than anything shipping commercially. **Steal the stats,
ship the ergonomics.** But you should read the paper before you write code, and you should cite
it in your README — being the person who productized a published method is a stronger position
than being the person who appeared to independently invent it.

## Finding 3 — Anthropic is one product decision away from your feature

`skill-creator` already ships in Claude Code with four modes (Create / **Eval** / Improve /
**Benchmark**) and four agents (Executor, Grader, **Comparator**, Analyzer), including blind A/B
comparison of skill versions with a browser-based result viewer. It is built in. It is free.
It does not currently cover `CLAUDE.md`, and it does not do statistics or ablation.

The gap between "A/B tests a SKILL.md" and "A/B tests a CLAUDE.md" is small. Treat this as the
primary displacement risk ([11-risks.md](11-risks.md)) and build the things a first-party tool
is structurally unlikely to build: cross-vendor (Claude Code *and* Codex on the same suite),
statistical honesty as the selling point, and CI enforcement.

## So: build it?

Yes, with the reframe. It is a four-weekend project with a real audience, a defensible niche,
and a launch artifact that writes itself. But go in knowing that the honest version of this tool
will spend most of its output telling users their config change did nothing measurable — and
that this is the feature, not the bug.
