Writing a fair suite.

A receipt is only as trustworthy as the suite behind it. This is how to write a suite that measures whether a skill actually helps - and keeps measuring honestly as the model underneath it changes. Same rules and phrasing as the Report #001 methodology.

Scaffold a starting point with driftproof init <dir>; its three example cases already follow every rule here. The full guide also lives in the repo as AUTHORING.md.

The one rule everything serves

Grounding policy (codified)

Baseline = the same prompt, no SKILL.md

Driftproof runs each case twice: with_skill (the SKILL.md is the system prompt) and baseline (the identical prompt, no skill). A good case is one the base model gets wrong or mediocre on its own and right with the skill - that gap is the skill's value. A case both modes ace, or both flunk, measures the model, not the skill. (Caveat: skills that assume tools or context a bare baseline never had can show inflated lift - keep cases answerable from the prompt + skill text alone.)

Anchor every rubric at 0.80

Rubrics are graded by an LLM judge on a 0–1 scale. Anchor them so a fully correct, idiomatic answer scores 0.80, and reserve 0.81–1.00 for exemplary work only. Subtract for concrete, named errors:

SCORING ANCHOR (apply strictly): a fully correct, idiomatic response scores 0.80;
award 0.81-0.90 only if it is ALSO exemplary (name the extra quality); 0.91-1.00
only if flawless and exceptional (rare). Subtract ~0.2 for each <specific error>.
If <the skill's core requirement> is missing, cap at 0.3.

Anchoring at 0.80 (not 1.0) leaves headroom so a competent run is not pinned at the ceiling, and a later regression has room to show.

Graded difficulty - avoid saturation

Target a with_skill mean of roughly 0.7–0.9, not 1.0. A suite where every case scores 1.0 is saturated: it has no room to detect drift, because a model can get meaningfully worse and the number won't move. Mix a few straightforward cases with harder edge cases that have real headroom.

Why sampling and bands exist

The 0.05 effect floor

Band separation alone is necessary but not sufficient. The judge quantizes scores to a coarse ~0.05–0.1 grid, so a confident grade often collapses to a zero-width "point band," and two point bands one quantum apart (0.60 vs 0.64) are technically separated yet mean no real change. So the mean must also move by at least the effect floor of 0.05 (one judge-quantization step). Separated-but-trivial moves are reported as within noise (below effect floor). Band separation plus a floor-sized delta - never either alone - triggers a verdict.

What makes a case unfair

A case is unfair when it can fail for reasons the skill's author never signed up for:

• Gotchas / trick questions - the "right" answer depends on a twist nothing in the SKILL.md prepares for.
• Undocumented expectations - the rubric demands behaviour the skill never claims. If the rubric asks for it, the SKILL.md must say it (the grounding policy: every gradable criterion traces to SKILL.md text; rubrics may not extrapolate).
• Provider- or model-specific trivia - answers that hinge on one model's quirks, version-specific behaviour, or knowledge cutoffs rather than the skill's content.
• Saturated or degenerate cases - trivially passed by any model with or without the skill, or impossible for both. Either measures nothing.
• Ambiguous rubrics - if two careful graders would disagree on 0.80-vs-0.50, tighten the rubric with concrete, checkable criteria and explicit caps.

If you wouldn't defend a case to the skill's author with "your own docs promise this," cut it.

How to read the receipt

• comparison.delta - the skill's measured lift (with_skill − baseline). Read it through the effect floor: ≥ 0.05 is a real lift, |delta| < 0.05 is within noise.
• results.aggregates - mean_score ± stddev per mode. A with_skill mean of ~0.95+ across the suite suggests saturation.
• Per-case outcome - pass / fail, or borderline when the threshold sits inside the band (the run can't confidently call it).
• run.judge - how grading was done (samples, temperature, surface). On api the judge is pinned to temperature 0; on claude-cli sampling is surface-controlled and the receipt says so.
• Hashes - content_hash / suite_hash / receipt_hash are reproducible; the self-hash makes a hand-edit detectable. driftproof validate <receipt> checks both.

To compare two runs across a model release, driftproof diff A.json B.json applies the band + floor rule per case and summarizes the per-case verdicts into a headline.

Contributing a suite for a public skill

Grounded in the docs, anchored at 0.80, graded for headroom, sampled and banded, floored at 0.05 - that is a suite whose verdict you can act on.