A skill that helped last month may not help today.
Skills are the SKILL.md files that teach AI agents your conventions: code-review discipline, commit style, writing rules. Tens of thousands are shared publicly, and almost none are re-tested when the model underneath them changes.
Driftproof re-runs an agent skill's own eval suite across model versions, samples the judge so every score carries a confidence band, and only claims a regression when the bands actually separate. The phenomenon is the story: verdicts age because the substrate moves.
Read Report #001 → View on GitHub
What it does
For each skill, Driftproof runs its eval suite with the skill and without it, on two model versions. Each response is judged several times, so a case carries a mean ± stddev band rather than one fragile number. Diffing two runs yields a drift report: a case only counts as regressed or improved when its two confidence bands do not overlap. Everything is recorded in a signed receipt — a dated, hash-verified record anyone can re-run — whose every hash is reproducible.
Why band-based
An LLM judge is noisy. A tool that cries wolf on that noise is worse than useless. Driftproof's core rule is that a verdict is claimed only on non-overlapping bands — so the honest answer is allowed to be "nothing moved beyond the noise." That restraint is the point.
Latest report
Sonnet 5 shipped June 30. We re-ran the eval suites of 10 popular public skills across the release: 9 of 10 moved beyond noise (3 regressed, 3 improved, 3 mixed). Report #001 — do skill verdicts hold across a Sonnet model release? Ten public, text-representable agent skills, run on a current-vs-previous Sonnet pair with sampled, band-based verdicts.
Roadmap
- Monthly drift reports, plus a run triggered on each new model release.
- Signed receipts — a key signature over the canonical form, so a receipt can be trusted, not just read.
- A sandboxed-execution harness for tool-execution skills (document renderers, diagram/asset generators) that Report #001 scopes out.