Driftproof

What is Driftproof?

Driftproof (driftproofhq): open-source agent skill evaluation and regression testing across model releases. It runs a skill's own tests with the skill and without it, scores every answer several times, and calls a change only when it is clearly bigger than the usual spread. Each result is a dated, hash-verified receipt.

Explore the results

Skill badges · Can you tell? Every skill · Did the AI change? · Inspect these instructions · The AI noticed the sugar pill · Which bar is the fake? · Every task we gave a skill · Try it yourself · Which section did the work? · Skill or sugar pill?

What it measures

For each task in a skill's own test set, Driftproof asks the model twice: once with the skill's instructions loaded, and once with the task alone. The model answers each way several times. What the skill adds is the average with the skill minus the average without it.

A receipt records what produced its numbers: the model, the tools, the skill and the tasks by their content, and the grader with its instructions. The stale command compares those with what would run today and says what has moved.

What it does not measure

A clear difference is what these tests saw, not proof that the skill itself changed. Raw scores across companies are background, never a ranking.

Any constant style preference the judge holds toward a provider is present in both the with-skill and the baseline run for that provider, so taking the difference reduces it, though it is not cancelled.

How it differs from a skill's own tests and from other eval tools

A skill's own tests passing is one answer. Driftproof asks two more: whether the scores with the skill and without it separate beyond their spread, and whether that held when the model changed.

What each other tool measures, and whether it compares a task with the skill and without it, as its own docs say. The full table: What each evaluation tool measures.

The published evidence

Every published report, newest first.

The paper

Reported, Not Measured: An Empirical Study of Measurement Defects in LLM and Agent Evaluation Tools. Version 2, 11 November 2026. Preprint, not peer reviewed. DOI 10.5281/zenodo.24136677.

The software

Upstream fixes and independent mentions

Fixes merged into other evaluation tools

Driftproof's maintainer, Maverick (mavericksea-ai on GitHub), reads the scoring and gating code of other evaluation tools and sends fixes upstream. Twelve are merged. Current GitHub release observations are listed on the maintainer page. Among the merged fixes:

  • MLflow: relative-change gates now use the baseline's magnitude, so a negative baseline no longer passes a worse model and fails a better one (#26252).
  • NVIDIA SkillEvaluator: a script earns execution credit only with evidence that it was run, not just read or searched (#154).
  • Agent Skills: grader results are bound to the expectations they answer (#576); stale grading is cleared and each run records its identity (#587); the ADR eval grades what the skill asks for (#598); the floor-guard reference catches untracked files, deleted tests and raised limits (#600, #614). Two more fixes there are not about evaluation (#578, #615).

Open pull requests: search GitHub. Nine of the defects behind them are written up in the paper. More on who maintains Driftproof: Maintainer.

Mentions by people outside the project:

The name

This Driftproof, the project at driftproofhq, is unrelated to other projects that use the name.

More answers

Words with a dotted line link to the glossary, where each is explained in plain words.