Methodology

For each eval case, Driftproof runs the model with the skill (SKILL.md supplied) and without it (bare baseline). Each response is graded by the fixed judge several times (n = 5), so a case carries a mean ± stddev band, not one fragile number. The skill's lift (delta) is the with-skill mean minus the baseline mean.

Diffing two receipts yields a per-case verdict. A case counts as regressed or improved only when both conditions hold:

• Band separation. The two with_skill confidence bands (mean ± stddev over judge samples) do not overlap.
• Effect floor. The mean also moves by at least 0.05 — one judge quantization step. A separated-but-tiny move is downgraded to within noise (below effect floor).

Bands that touch or overlap, or moves below the floor, are reported as within noise — never as drift. A tool that cries wolf on judge noise is worse than useless, so band separation plus a real-sized delta, not either alone, is what triggers a verdict. The judge is deliberately fixed and cheap; see the judge policy.

A suite case may declare optional checks[] — structural or regex assertions on the model output that simply hold or don't, with no LLM judgment (e.g. "the output contains a Conventional-Commits subject line", "the output mentions the 429 status the task described"). These run alongside the judge and are reported as a separate column.

Post-checks are supplementary evidence only. They are not folded into the case outcome or the band-overlap verdict — they corroborate or contradict the judge with a cheap, unambiguous signal, they do not replace it. Authors add them where they are natural and in-text-groundable; they are never required, and a suite without them is unaffected. Each result is recorded in the receipt as { name, kind, pass } (spec v0.3.1).

A skill is not free: its SKILL.md is prepended to every generation. A +0.10 lift for 400 skill tokens is a very different proposition from +0.10 for 6,000. So Driftproof reports each skill's lift both raw and normalized per 1,000 skill tokens: delta ÷ (skill_tokens ÷ 1000).

Method. skill_tokens is a coarse ceil(chars ÷ 4) estimate of the SKILL.md size — the standard rough proxy for English text, not a model tokenizer (no tiktoken dependency). It is a consistent yardstick for comparing skills to one another, not a billing figure, and it is recorded in the receipt (skill.tokens, spec v0.3.1). Value-per-token surfaces the skills that earn their context budget.

Driftproof's provider layer is two-axis: provider (anthropic / openai) × surface (api / cli), i.e. four concrete lanes. The same suites and the same fixed judge run across providers; only the target model varies. Every receipt records the provider and surface it ran on, the API surface is preferred for published runs, and a subscription CLI surface is disclosed whenever used.

Because one Claude-family judge grades every provider's output, a cross-provider report leads with the with/without delta — which cancels most judge style-affinity — and treats absolute cross-provider scores as context, never a ranking. Report #002 additionally holds the surface type constant: both substrates run on the vendor's own first-party subscription CLI (claude -p and codex exec), with each surface's harness overhead disclosed (Codex prepends a fixed ~12–15k-token preamble; claude -p a smaller first-party context). The full policy is on the neutrality page.

Skill Drift Is Contract Violation (Fan et al., arXiv:2605.10990) treats skill decay as a violation of environment contracts — external services and APIs moving under a skill — and repairs against them; that environmental axis is complementary to the substrate axis measured here (the model itself moving under a skill). The verification levels our receipts carry (UNVERIFIED / DECLARED / TESTED / FORMAL) follow the lattice introduced in Skills as Verifiable Artifacts (Metere, arXiv:2605.00424), whose companion, Methods for Formal Verification of Agent Skills (arXiv:2605.23951), works toward the FORMAL level our schema still marks reserved.

The receipt is not proprietary: the JSON Schema is served at its canonical id (/spec/receipt.schema.json), any harness is encouraged to emit receipts, and working converters import results from neighboring eval tools — with the DECLARED vs TESTED distinction enforced structurally (imported declarations are recorded faithfully but never earn drift verdicts). The schema, the import mappings, and the minimal summary-json interchange are on the interop page.