Driftproof
Menu

Your skill passed. On which model? On what date?

A SKILL.md teaches an AI coding agent how you like things done. When a new model ships, the same file can stop helping, or start hurting. Driftproof re-runs the skill's tests on the new model and hands you a dated, hash-verified receipt saying whether it still helps.

npx driftproof init my-skill Read the latest report

A receipt, in full

Verdict PASSED

Model claude-haiku-4-5

Date 2026-07-27

Skill commit-message-conventions

Range 0.474 without, 1 with

commit-message-conventions-claude-haiku-4-5-20251001-2026-07-27. separated: the ranges do not overlap and the move clears the floor. Without the skill, 0.474 plus or minus 0.391. With the skill, 1.000 plus or minus 0.000. 0 0.25 0.5 0.75 1 separated without the skill with the skill

Receipt hash 4c2f6a92539b8dea167ea37ccb762ae0cad7143653c5777e095e5094340a89d1

How it works

  1. Run your skill's tests twice: with the skill, and without it.
  2. Score each answer several times, so every result is a range, not one fragile number.
  3. Only call it a win or a loss when the ranges don't overlap. Otherwise we say: nothing moved beyond the noise.

That range is a band. The dated file that records all of it is a receipt. What changes between two receipts is drift.

Two horizontal ranges, drawn apart.
Separated The two ranges do not overlap. Something moved, and it moved by more than the noise.
Two horizontal ranges, drawn crossing each other.
Overlapping The ranges share space. We say nothing moved beyond the noise, and we say it plainly.
Two horizontal ranges, drawn with one hatched because nothing was measured.
Refused The run could not stand behind a number, so it reports no result instead of guessing one.
The same two ranges, drawn from the bundled example receipt.

Move the floor and change what counts as a band to see the verdict change. This is the bundled example receipt's own per-case data.

Skills don't break. Models move.

8reports published
123receipts published
8models measured on
838eval cases scored
3refusals published
118runs where nothing moved beyond the noise

On the report that measured it, draw-to-draw spread reaches sd 0.355 on a single case (semver-hidden-breaking-change). That is the across-draw spread of the model writing a different answer, not of the scorer re-reading one, and it is wider than the scorer's.

We publish refusals instead of guesses, and we publish our own instrument defects on the page. Read the methodology, or read what we will not claim.

Who it's for

Skill authors

Prove it helps before you publish.

Catalog maintainers

Know which of your skills broke when the model changed.

Platform teams

An evidence trail for the skills your agents run in production.

Run it on your own skill

You need Node 22 or newer and an ANTHROPIC_API_KEY. From "found the repo" to a receipt for your own skill is about five steps:

npx driftproof init my-skill          # scaffold SKILL.md + evals/evals.json + .driftproofrc
# edit the 3 example cases so each is grounded in a claim your SKILL.md makes
export CLAUDE_PROVIDER=api
read -rsp "Anthropic API key: " ANTHROPIC_API_KEY && export ANTHROPIC_API_KEY
npx driftproof run my-skill --models claude-haiku-4-5
cat receipts/*.summary.md              # read the receipt + human summary

The verdict answers "does the skill still help on this model?". PASSED means the with-skill score beats the baseline by at least the effect floor, NO_EFFECT means it doesn't, REGRESSED means the skill hurts. Each case is judged several times so it carries its own range, and a case counts as moved only when the ranges don't overlap.

Verification in CI

Wire drift detection into a repo with the GitHub Action (uses: driftproofhq/driftproof@v0.9.0): it re-runs the suite on every push, uploads the receipt, and fails the job if the skill regressed. driftproof badge turns any receipt into a shields.io badge, so the claim on your README carries the model and the date it was measured.

driftproof badge: passing on claude-haiku-4-5

Latest report

All 8 reports

Writing: Three model releases later: what actually happens to agent skills, the launch essay, revised to read all eight reports together on what a moving model does to encoded expertise.

Get the next one

Your address goes to Buttondown, our newsletter provider, and nowhere else. One email per model release. Unsubscribe any time.

Every report is also in the Atom feed.

Roadmap