What is Driftproof?
Driftproof (driftproofhq): open-source agent skill evaluation and regression testing across model releases. It runs a skill's own tests with the skill and without it, scores every answer several times, and calls a change only when it is clearly bigger than the usual spread. Each result is a dated, hash-verified receipt.
Explore the results
Skill badges · Can you tell? Every skill · Did the AI change? · Inspect these instructions · The AI noticed the sugar pill · Which bar is the fake? · Every task we gave a skill · Try it yourself · Which section did the work? · Skill or sugar pill?
What it measures
For each task in a skill's own test set, Driftproof asks the model twice: once with the skill's instructions loaded, and once with the task alone. The model answers each way several times. What the skill adds is the average with the skill minus the average without it.
A receipt records what produced its numbers: the model, the tools, the skill and the tasks by their content, and the grader with its instructions. The stale command compares those with what would run today and says what has moved.
What it does not measure
A clear difference is what these tests saw, not proof that the skill itself changed. Raw scores across companies are background, never a ranking.
Any constant style preference the judge holds toward a provider is present in both the with-skill and the baseline run for that provider, so taking the difference reduces it, though it is not cancelled.
How it differs from a skill's own tests and from other eval tools
A skill's own tests passing is one answer. Driftproof asks two more: whether the scores with the skill and without it separate beyond their spread, and whether that held when the model changed.
What each other tool measures, and whether it compares a task with the skill and without it, as its own docs say. The full table: What each evaluation tool measures.
- claude plugin eval (Claude Code). What it measures: How reliably a plugin (including a skill shipped as a plugin) leads Claude to the right outcome on eval cases, and what it contributes against a no-plugin baseline (source, read 30 September 2026). With and without the skill: Yes, by default: a with-plugin arm and a no-plugin arm, reported as WITH, W/OUT and delta; --ablation none turns it off (source, read 30 September 2026).
- alibaba/skill-up. What it measures: A Skill's behaviour across declarative cases (response, tool calls and workspace changes), optionally compared with and without the Skill; it also evaluates agents and workspaces without a Skill (source, read 30 September 2026; source, read 30 September 2026). With and without the skill: Yes, opt-in: benchmark mode (benchmark.enabled: true or --baseline) runs each case with_skill and without_skill and reports the delta; disabled by default (source, read 30 September 2026; source, read 30 September 2026).
- NVIDIA SkillEvaluator. What it measures: Three tiers: static validation (safety, well-formedness), deduplication (repeated or overlapping guidance), and live evaluation of whether the skill helps a real agent, reported on five dimensions (security, correctness, discoverability, effectiveness, efficiency) (source, read 30 September 2026; source, read 30 September 2026; source, read 2 October 2026; source, read 2 October 2026). With and without the skill: Yes, by default in Tier 3: a with-skill arm and a baseline arm in a sandbox; --skip-baseline removes the baseline and the lift (source, read 30 September 2026; source, read 2 October 2026).
- agent-skills-eval. What it measures: Whether a SKILL.md makes a model better at the task on the skill's evals/evals.json prompts, judged against declared assertions (source, read 30 September 2026). With and without the skill: Yes, opt-in: --baseline (config baseline: true) runs each eval with_skill and without_skill; the default is false (source, read 30 September 2026; source, read 30 September 2026).
The published evidence
Every published report, newest first.
- Report 019: a same-length placebo text
- Report 018: a same-length placebo text
- Report 017: a same-length placebo text
- Report 016: Popular rules files vs. a placebo: most made Claude write more
- Report 015: Skills.sh snapshot 2026-10-17: text tasks
- Report 014: Claude Haiku 5.5 on release day, three skills
- Report 013: Claude Sonnet 5.5, the day after its release
- Report 011: Claude Opus 5.5, the day after its release, three skills
- Report 010: repeated ADR evaluations across two Claude Code configurations
- Report 009: three skills under two eval harnesses
- Report 008: Claude Fable 5.1, no separation detected across the release in either arm
- Report 007: the baseline is the noisy arm
- Report 006: the skill text moves, the substrate holds still
- Report 005: what does a skill cost to run?
- Report 004: does encoded expertise still lift output on the frontier tier?
- Report 003: do skill verdicts hold across a model release?
- Report 002: does a skill's benefit hold across substrates?
- Report 001: do skill verdicts hold across a model release?
The paper
Reported, Not Measured: An Empirical Study of Measurement Defects in LLM and Agent Evaluation Tools. Version 2, 11 November 2026. Preprint, not peer reviewed. DOI 10.5281/zenodo.24136677.
The software
Upstream fixes and independent mentions
Fixes merged into other evaluation tools
Driftproof's maintainer, Maverick (mavericksea-ai on GitHub), reads the scoring and gating code of other evaluation tools and sends fixes upstream. Twelve are merged. Current GitHub release observations are listed on the maintainer page. Among the merged fixes:
- MLflow: relative-change gates now use the baseline's magnitude, so a negative baseline no longer passes a worse model and fails a better one (#26252).
- NVIDIA SkillEvaluator: a script earns execution credit only with evidence that it was run, not just read or searched (#154).
- Agent Skills: grader results are bound to the expectations they answer (#576); stale grading is cleared and each run records its identity (#587); the ADR eval grades what the skill asks for (#598); the floor-guard reference catches untracked files, deleted tests and raised limits (#600, #614). Two more fixes there are not about evaluation (#578, #615).
Open pull requests: search GitHub. Nine of the defects behind them are written up in the paper. More on who maintains Driftproof: Maintainer.
Mentions by people outside the project:
- JeanJeanLeM/GhostFrame pull request #3. The repository owner opened it on 28 September 2026; it cites a Driftproof DEV post as its source.
- addyosmani/agent-skills issue #433. A repository collaborator mentioned Driftproof in a comment on 27 August 2026.
- addyosmani/agent-skills issue #433. A repository collaborator mentioned Driftproof in a comment on 28 August 2026.
The name
This Driftproof, the project at driftproofhq, is unrelated to other projects that use the name.