# 4. Task suite

## Design principle

A task is a **prompt plus a set of deterministic checks, pinned to a commit.** It is not a
benchmark instance and does not need to be hard. What it needs to be is *discriminating* — a task
every trial passes and a task no trial passes both contribute zero information and cost full
price. See [05-statistics.md](05-statistics.md) on why mid-difficulty tasks are worth 3× their
cost.

## Schema

`.ablate/tasks/<id>.yaml`. Field names deliberately echo SWE-bench where they overlap.

```yaml
id: fix-null-deref-in-parser
description: Agent must fix the crash and add a regression test.

base_commit: 3f9a1c2          # optional; defaults to HEAD
setup: uv sync --frozen       # optional; cached per (base_commit, lockfile hash)
timeout_s: 600

prompt: |
  The CSV parser crashes on files with a trailing empty line.
  Fix it and add a regression test.

# Optional: paths the agent must not be able to see or touch.
hide:
  - tests/test_parser_trailing.py    # the answer, if this task was mined from a PR

assert:
  - id: tests-pass
    type: exit_code
    run: uv run pytest -q
    expect: 0

  - id: regression-test-added
    type: files_touched
    must_include: ["tests/**"]

  - id: no-vendored-deps
    type: files_touched
    must_not_include: ["vendor/**", "**/node_modules/**"]

  - id: no-broad-except
    type: absent
    pattern: 'except\s*:'
    paths: ["src/**/*.py"]

  - id: lint
    type: lint
    run: uv run ruff check .
    expect: 0

weight: 1.0                    # optional; relative weight in the aggregate metric
```

`passed` for a trial is the conjunction of all assertions unless an assertion sets
`required: false`, in which case it is reported but excluded from the pass/fail roll-up. Use
`required: false` for guardrail checks you want visible without letting them dominate.

## Assertion vocabulary

Five types. All deterministic, all cheap, all runnable offline.

| Type | Config | Semantics |
|---|---|---|
| `exit_code` | `run`, `expect` (default `0`), `timeout_s` | Run a shell command in the worktree; compare exit status. The workhorse — this is your test suite. |
| `files_touched` | `must_include`, `must_not_include` (globs) | Computed from `git diff --name-only` against `base_commit`. Catches scope creep, unrequested refactors, vendored deps, and edits to files the task said not to touch. |
| `present` / `absent` | `pattern` (regex), `paths` (globs), `in`: `diff`\|`files` (default `diff`) | Regex over the produced diff or the resulting file contents. Cheap proxy for "did it use the idiom we asked for." |
| `lint` | `run`, `expect` | Structurally identical to `exit_code`; separate type so reports can group "correctness" vs "style" and because it's the natural home for a future rule-level breakdown. |
| `script` | `run`, `expect`, `capture_json` | Escape hatch. Any executable, run in the worktree with `ABLATE_TRIAL_DIR`, `ABLATE_BASE_COMMIT`, `ABLATE_DIFF_FILE` in the environment. With `capture_json: true`, stdout is parsed as JSON and merged into the trial record — this is how users add **continuous** custom metrics (bundle size, benchmark ms, coverage %), which are far more statistically efficient than binary ones. |

### AST assertions: deliberately omitted

You listed "AST or regex presence/absence." Regex ships; AST does not, because a language-generic
AST matcher means either embedding tree-sitter (a heavy native dependency that fights `uvx`
cold-start) or shipping per-language matchers you can't maintain solo.

The `script` type covers it: users who need AST matching write six lines of Python with `ast` or
call `semgrep`/`ast-grep` and return an exit code. Document that pattern with a worked example
and revisit if it turns out to be the most-requested feature.

## LLM-as-judge: not allowed in v1

Full reasoning in [ADR-0004](adr/0004-no-llm-judge.md). The short version, which belongs in the
README because it is a positioning statement, not just a technical one:

> An LLM judge adds variance to the thing you are trying to measure the variance of. You are
> already fighting a ~20 percentage-point minimum detectable effect; a judge with 10% grading
> noise, correlated with the very config you're testing, makes the measurement uninterpretable.
> Every assertion in `ablate` is a deterministic check you could run yourself.

This is a hard architectural line, not a "not yet." If it is ever added it must be as an
explicitly-flagged, separately-reported, never-in-the-headline-number metric.

## `ablate init` — the five-minute path

1. Detect the test command: `pytest` / `npm test` / `cargo test` / `go test ./...` /
   `Makefile:test`, in that order, confirming with the user.
2. Detect the config file: `CLAUDE.md`, `AGENTS.md`, `.cursorrules`, first match wins.
3. Write three template tasks, prefilled with the detected test command and marked `TODO` where
   a prompt is needed. The templates are the three archetypes that actually discriminate:
   - **bug-fix**: "fix X and add a regression test" — tests as the assertion.
   - **convention-adherence**: "add a new Y" + `present`/`absent` assertions on the repo's
     idioms. This is the archetype most sensitive to config content and should be the default
     example everywhere in the docs.
   - **restraint**: "make this small change" + `files_touched.must_not_include` on everything
     unrelated. Catches the over-engineering failure mode that verbose configs cause.
4. Print the estimated cost of the first run and the exact next command.

## `ablate mine` — the git-history miner

Ships **experimental**, proposes only, never auto-trusted. [ADR-0008](adr/0008-task-miner-scope.md).

### Pipeline

```
gh pr list --state merged --limit 300 --json ...
  -> hard filters
  -> scoring
  -> top N -> draft task YAML + a review checklist -> human keeps or discards
```

### Hard filters (a PR is discarded outright if any fails)

| Filter | Rationale |
|---|---|
| Touches ≥1 test file **and** ≥1 non-test source file | Without a test change there is no assertion to write. |
| ≤ 8 files changed, ≤ 300 changed lines | Multi-concern PRs make tasks whose failure is uninformative. |
| Exactly one non-test source *directory* touched | Proxy for single-concern. |
| Linked issue body or PR body ≥ 200 chars | The body *is* the prompt. A one-line PR gives an underspecified task. |
| No changes to lockfiles, CI config, or generated files | Environment drift and noise. |
| `git revert <merge>` applies cleanly | If it doesn't, the task can't be constructed. |
| **The test suite passes at the parent commit, and fails after revert** | The critical filter. It proves the environment is reproducible at that commit *and* that the new test actually detects the bug. Costs one CPU-bound suite run per candidate; nothing else validates the task. |
| Merged within the last 12 months | Older commits usually can't reproduce their environment. |

### Scoring (rank the survivors)

Rank by `0.4·specificity + 0.3·testability + 0.3·recency`, where specificity is a crude readability/
length signal on the issue body, testability is (added test assertions ÷ changed source lines),
and recency is a decay from merge date. This ranking is a convenience for the human reviewer, not
a correctness claim; don't over-engineer it.

### Constructing the task

- `base_commit` = the merge's first parent.
- `prompt` = the linked issue body, or the PR description with any explicit implementation
  instructions and file paths stripped (you are testing whether the agent finds the right file,
  so don't tell it).
- `hide` = the test files the PR added. They are applied to the worktree *after* the agent runs,
  which is how you avoid the agent reading the answer while still using the real test as the
  assertion.
- `assert` = one `exit_code` on the project test command, run **after** re-applying the hidden
  test files. Plus a `files_touched.must_include` on the source directory the real PR touched.

### Diff similarity — how to score without being brittle

**The recommendation is: don't score diff similarity as a pass/fail assertion.** The real PR's
diff is one of many correct implementations; exact or near-exact matching penalizes correct
solutions and is exactly the brittleness you flagged.

Instead:

1. **Behavior is the assertion.** The hidden test from the real PR is the ground truth. It passes
   or it doesn't. This is the SWE-bench `FAIL_TO_PASS` model and it is correct.
2. **Similarity is a reported diagnostic, never a gate.** Compute two cheap, interpretable
   numbers and print them for the human:
   - **File-set Jaccard**: `|files_agent ∩ files_real| / |files_agent ∪ files_real|`. Robust,
     zero dependencies, and genuinely informative — an agent that fixed the bug in a totally
     different file solved a different problem.
   - **Hunk-locality**: fraction of the agent's changed lines that fall within ±20 lines of a
     line the real PR changed. Insensitive to naming, formatting, and implementation choice;
     sensitive to "did it edit the right region."
3. **Never use token-level or embedding similarity.** Token similarity is brittle; embedding
   similarity is an LLM judge with extra steps and is excluded by [ADR-0004](adr/0004-no-llm-judge.md).

If a mined task's hidden test passes but file-set Jaccard is 0, the reviewer should look at it —
usually the test is weaker than it appeared. Surface that pairing in the review checklist.
