Is the gap your skill makes real, or noise? Did it hold on the last model release?

A SKILL.md teaches an AI coding agent how you like things done. A skill's own tests passing is one answer. Driftproof asks two more: whether the scores with the skill and without it separate beyond their spread, and whether that held when the model changed. Each answer is a dated, hash-verified receipt, and when there were too few draws to tell, the receipt says so.

Install in Claude Code

claude plugin marketplace add driftproofhq/driftproof claude plugin install driftproof@driftproofhq

Then run /driftproof:start on a skill in your repository. For CI, npx and the GitHub Action are further down.

See a receipt Read the findings Read the latest report

A piece by Maverick: quoted AI benchmark gaps, checked against their own sampling noise.

A receipt, in full

Verdict Passing

Model claude-haiku-4-5

Date 2026-09-17

Skill commit-message-conventions

Range 0.434 without, 0.822 with

the two ranges do not overlap: separated

Receipt hash fad883d1018d3a4b3c75bc60b1eaec50032256cbf69fba3299999edf7978589c

14 reports published

216 receipts published

11 models measured on

Comparing two benchmark scores? The gap calculator says whether the gap is bigger than the benchmark's own sampling noise.

What a receipt can say

Passing. A case separated upward under the band rule: the two ranges do not overlap and the move clears the effect floor. That is a separation detected under the rule, not proof that the skill moved the score.
Regressed. A case separated downward under the same rule.
No separation detected. The ranges overlap at this sample size, which is not evidence that nothing changed.
Not enough draws. Not enough draws to conclude at this effect floor. The receipt prints how many draws per arm would have been needed, or that no number of draws reaches the floor at the spreads it measured.
More cases needed. More cases, not more draws, are needed to conclude at this effect floor. The cases score too far apart from each other for more draws to be expected to bring the comparison band under the floor, and the receipt prints the band with each case's draw-to-draw spread taken out.
Not measured. The receipt carries no verdict, and its page names every reason, such as a run that does not record that a model answered it.

Every receipt has its own page, with its verdict, its two arms and a way to verify it yourself. How this is built describes how the tool itself is checked.

How it works

Run your skill's tests twice: with the skill, and without it.
Draw each answer several times and score each draw several times, so every result is a range, not one fragile number.
Call it a separation only when the ranges don't overlap and the move clears the floor. When the ranges overlap, say no separation detected at this sample size; when the draws were too few to tell, say so.

That range is a band. The dated file that records all of it is a receipt. What changes between two receipts is drift.

Two horizontal ranges, drawn apart. — Separated The two ranges do not overlap, and the move clears the floor. A separation is detected under the rule, which is not proof that the skill moved.

Two horizontal ranges, drawn crossing each other. — Overlapping The ranges share space. We say no separation detected at this sample size, and we say it plainly.

Two horizontal ranges, drawn with one hatched because nothing was measured. — Refused The run could not stand behind a number, so it reports no result instead of guessing one.

The same two ranges, drawn from the bundled example receipt.

Move the floor and change what counts as a band to see the verdict change. This is the bundled example receipt's own per-case data.

Skills don't break. Models move.

14reports published

216receipts published

11models measured on

1032eval cases scored

0refusals published

165runs with no separation detected at their sample size

On the report that measured it, draw-to-draw spread reaches sd 0.355 on a single case (semver-hidden-breaking-change). That is the across-draw spread of the model writing a different answer, not of the scorer re-reading one, and it is wider than the scorer's.

We publish refusals instead of guesses, and we publish our own instrument defects, including what outside audits found. Read the methodology, or read what we will not claim.

Who it's for

Skill authors

See whether it helps before you publish, and on which model.

Catalog maintainers

See which of your skills stopped separating when the model changed.

Platform teams

An evidence trail for the skills your agents run in production.

Install in Claude Code

claude plugin marketplace add driftproofhq/driftproof

claude plugin install driftproof@driftproofhq

That installs /driftproof:start, /driftproof:init, /driftproof:run and /driftproof:badge. The plugin hands your arguments to the pinned runner and writes the receipt that runner would have written; /driftproof:run spends your Claude Code subscription rather than an API key. It measures and never changes a skill's instructions. The one file it adds is a new evals/evals.json, from /driftproof:start or /driftproof:init, to a skill that has none, after you say yes in the run.

Start with a skill you already have. In Claude Code, from the folder above it, in a git repository, run /driftproof:start. Claude lists your skills and asks which to check. If the skill has no test cases, Claude reads your SKILL.md and drafts test cases with you, aimed at outcomes you would check. It writes only after you say yes, then runs a quick check and opens a results page: your first receipt. Drafted cases are labelled as drafted from the skill and approved by you. A quick check is a first look and cannot produce a verdict. For a direct measurement, use /driftproof:run path/to/your-skill. On a skill with no test cases, it offers the same draft. For a new skill, scaffold one with npx driftproof init my-skill, as the next section shows.

Driftproof is tested on Linux and macOS. In an outside retest on macOS the shipped gate failed only in the publishing helper, which needs GNU realpath -m. Windows is untested: from Node's source the plugin would not find npx there, but that failure has never been observed. A CI matrix across Linux, macOS and Windows is planned, and the findings page has the figures.

In CI: npx and the GitHub Action

You need Node 22 or newer and an ANTHROPIC_API_KEY. From a clean checkout to a receipt for a new skill, on the command line:

npx driftproof init my-skill          # scaffold SKILL.md + evals/evals.json + .driftproofrc
# edit the 3 example cases so each is grounded in a claim your SKILL.md makes
export CLAUDE_PROVIDER=api
read -rsp "Anthropic API key: " ANTHROPIC_API_KEY && export ANTHROPIC_API_KEY
npx driftproof run my-skill --models claude-haiku-4-5
cat receipts/*.summary.md              # read the receipt + human summary

Wire it into a repository with the GitHub Action (uses: driftproofhq/driftproof@v0.23.1): it re-runs the suite on every push, uploads the receipts, and fails the job when a case separated downward. driftproof badge turns a receipt into a badge that carries the model and the date it was measured.

driftproof badge: passing on claude-haiku-4-5

Latest report

Capability-gap report

Report 015: Skills.sh snapshot 2026-10-17: text tasks

claude-sonnet-5-5 claude-opus-5-5

2026-10-18

All 14 reports

Writing: What happens to agent skills when the model changes, the launch essay reading all fourteen reports on what a moving model does to encoded expertise.

Get the next one

Your address goes to Buttondown, our newsletter provider, and nowhere else. One email per model release. Unsubscribe any time.

Every report is also in the Atom feed.

Common questions

Roadmap

Activation drift receipts compare recorded Skill-tool firing rates across one model or runtime version change. Fixed setup is operator-declared.
Planned, not scheduled: a sandboxed execution harness for tool-execution skills (document renderers, diagram and asset generators), which Report 001 scopes out.