What happens to agent skills when the model changes

By the Driftproof maintainer · published 2026-08-11 · revised 2026-10-07 to read the whole published record · retitled 2026-10-01 for what it is about. It launched as “Three model releases later: what actually happens to agent skills”.

Sonnet 5 shipped June 30. Opus 5 followed July 24. Most teams updated a model string and moved on. Almost nobody re-ran their skills.

Skills (the SKILL.md files that teach AI agents your conventions: code-review discipline, commit style, writing rules) are becoming how expertise gets encoded for the agent era. SkillsBench, the benchmark that indexes them, counts 47,000+ skills across 6,300+ public repositories (SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks, arXiv:2602.12670, read 2026-09-01). Almost none are tested once, let alone re-tested when the model underneath them changes. Every release silently re-prices every skill. I wanted to know by how much, so I built the checker.

Driftproof is a small open-source tool with one job: run a skill's eval suite with and without the skill, then say honestly whether anything moved. There are now seven report types, and they differ in what moves underneath the skill: the model version (release drift), a different model behind a different vendor's CLI (substrate durability), a higher tier of the same provider (capability gap), nothing at all, the axes widening instead from accuracy to accuracy, cost and latency (value), the skill's own text while everything under it is held still (revision drift), nothing outside the measuring device itself, when a corrected instrument re-runs cells an earlier report already published (instrument re-measurement), or two instruments measuring the same skills on the same pinned inputs (instrument comparison). Every score is judged five times, so it carries a band rather than a single fragile number: a descriptive spread, the mean plus or minus one sample standard deviation. A verdict is claimed only when bands fully separate AND the mean moves at least 0.05, one judge quantization step. Overlapping bands are reported as no separation detected at that sample size, never as drift and never as proof that nothing changed. Every run emits a dated, hash-verified receipt anyone can verify.

Thirteen reports in, here is what the records say.

Report #001 (sonnet-4-6 to sonnet-5, 10 popular public skills): 9 of 10 showed a separation detected under the rule, and the tally, as it stands after the v1.2 amendment described below, is 3 improved, 2 regressed, 4 mixed, 1 with no separation detected. Same files, same suites, same judge; only the model changed.

Report #002 (the same 10 suites, Claude against gpt-5.6-sol, one fixed judge): 3 durable, 4 substrate-dependent, 2 regressed, 1 no effect. The strangest rows are sign flips: a skill-authoring skill lifts output quality on Claude and reduces it on GPT; a code-review skill is the exact mirror. A skill tuned on one substrate can be dead weight, or worse, on another, and nothing warns you when you switch.

Report #003 (opus-4-8 to opus-5): 4 improved, 2 regressed, 0 mixed, 4 with no separation detected. Both regressions are single cases, on skills where no other case separated under the rule, one sitting exactly on the 0.05 floor. The sharpest signal is another sign flip, this time across versions rather than vendors: a skill that was net-negative on opus-4-8 is net-positive on opus-5, while three others went quiet because the new model does the thing without them.

Report #004 asks the question one tier up: does encoded expertise still lift output on the frontier tier of the same provider, claude-opus-5 against claude-fable-5? Over 10 skills: 3 durable, 5 tier-dependent, 0 regresses, 2 no effect. Encoded expertise survives the better model. Where lift had collapsed, it had collapsed on the flagship first, which is the opposite of the story I expected to write.

Report #005 stops asking only whether a skill helps and asks what it costs: the same suites on 3 substrates, with accuracy, cost and latency shown side by side and never folded into one score. In 20 of 30 skill by substrate pairs at least one case cleared the floor with separated bands. 14 cleared the floor on aggregate, of which 10 carry a price and 4 report a saving, having improved quality while reducing cost. The dominant cost driver turned out not to be the skill's own text but the input it causes the model to pull in: across those 30 cells the input delta tracks cost at r = +0.92 while the skill's own length tracks it at only r = +0.33, and one 738-token skill drew 34 times its own size in extra input. That cost-driver finding still stands and the page says so. Three of its cells no longer stand as published: they carry a v1.2 amendment naming their lifts as single-draw, judge-spread figures, for the reason the sixth and seventh reports established.

Then the sixth report refused itself. I built Report #006 to ask whether a skill's own text drifts, holding the model still and moving the words. To keep it cheap I reused #005's measurements as the fixed half of each comparison, on the argument that the ground beneath had not moved. My own control disagreed. It re-measured that ground, found it did not reproduce, and refused every cell before any of them produced a verdict: 3 cells, 0 measured, 3 refused. I had spent the budget and had nothing to report about revisions.

My first explanation was that the models had shifted under me. I had no evidence for it; it simply fit. So I measured instead: 120 calls, two cases, ten fresh draws each. The scores turned out to be single draws from wide distributions — one case landing anywhere from 0.300 to 0.808 — and the variation between generations ran three to seven times larger than the variation between judge samples. I sample the judge five times. I had been sampling the generation once. The error bars were on the smaller number.

So the receipt spec gained generation sampling, and the seventh report is that spec turned on the same three cells. Report #007 re-ran them with the generation sampled adaptively rather than once, and with a call timeout that finally did what the code had been declaring: a 300 s policy for this surface had been shadowed by a 120000 literal since 2026-07-27, four days before the policy that it outranked was even written. Under the corrected instrument the three cell lifts are +0.055, -0.002 and +0.131, every one of them smaller than its own band, and the 21 cases read 3 improved · 0 regressed · 18 no effect · 0 not measured. Nothing separates. Two of the three comparisons against the archive were refused outright by their own baseline control, and the skill text had moved upstream in the meantime, so none of the lower numbers is a correction and the report does not offer them as one.

The eighth report is the first one where the model moved and nothing else did. Report #008 re-measured two of the seventh report's cells on claude-fable-5-1 against claude-fable-5, with the skill text and the suite asserted byte-identical before the first call and with both sides generation-sampled rather than drawn once. Neither cell showed a separation detected under the rule: 0 improved · 0 regressed · 14 with no separation detected · 0 not measured. One case is counted there because of the effect floor rather than band overlap, and the report names it rather than leaving it in a table. No separation detected is not a finding that nothing changed; it is a statement about what this instrument resolves at the sample size it ran at, and the report says so in its own words.

The ninth report moves the ruler instead. Report #009 measured three skills from one plugin twice, once with Claude Code’s native plugin eval and once with Driftproof, on the same SKILL.md bytes, task prompts and rubrics, one case each. The two tools apply different treatments and grade differently, so the report reads them side by side and ranks neither. Driftproof’s own arms read 3 separated · 0 with no separation detected under the rule, each on one task. Both tools judged with claude-opus-5, which departs from the judge policy, so that tally is not comparable with the tallies for Reports #001 to #008. The native eval’s largest delta came on a skill whose Skill tool was not called in any of its with-plugin runs. It is the first instrument comparison here, and what it reports is what each instrument can and cannot establish on the same inputs, not which one is right.

The tenth report measures no skill with Driftproof at all. Report #010 ran one skill’s own behavioural eval, from the skill’s upstream repository, ten times at one commit with the same skill text, eval case and executor model, five times under each of two Claude Code configurations: 8 passed and 2 failed, both failures on one expectation that the skill’s ADR instructions do not state. No Driftproof judge ran, and the two configurations differ in version, settings and random draw, so the report does not say what caused the variation. What it shows is narrower: one run of that harness is one draw, and its verdict should be read with its configuration and its draw count beside it.

The eleventh report goes back to a model release. Report #011 measured Claude Opus 5.5 the day after its release, on the three skills and cases of Report #009, one case each, beside a fresh Claude Opus 5 arm on the same Claude Code version. Read by the runner’s own comparison of the with-skill arms, the three skills read 1 with no separation detected · 2 with not enough draws to conclude at the effect floor, which is not evidence that nothing changed. Every draw was judged with claude-opus-5, which departs from the judge policy, so that tally is not comparable with the tallies of the reports judged under it. A second table sets Report #009’s own Claude Opus 5 receipts beside this run’s: between them the Claude Code version, its host build and the date all changed, which the release drift type does not describe.

The next report is a release again, run three times. Report #013 measured Claude Sonnet 5.5 the day after its release on the same three skills and cases, one case each, beside fresh Claude Sonnet 5 and Claude Opus 5.5 arms on one Claude Code version, and ran every arm three times, because one run’s figure moved in the next. Read by the runner’s own comparison of the with-skill arms, Claude Sonnet 5 against Claude Sonnet 5.5 across the three runs reads 2 separated upward · 3 with no separation detected · 4 with not enough draws to conclude at the effect floor, which is not evidence that nothing changed. Beside Claude Opus 5.5 the rule separates Claude Sonnet 5.5 on no skill in any run, nine results in all, and five of those nine had too few answers to tell; what the skill adds differs where the two models’ scores without it differ. Every draw was judged with claude-opus-5, which departs from the judge policy; its tally compares with those of Reports #009 and #011, not with the reports judged under that policy.

The newest report is a release too, again run three times. Report #014 measured Claude Haiku 5.5 on its release day on the same three skills and cases, one case each, beside fresh Claude Haiku 4.5 arms on one Claude Code version, and ran every arm three times; Claude Sonnet 5.5 and Claude Opus 5.5 appear as Report #013’s own receipts. Each result is a receipt’s own reading of what the skill added to the model that answered. Across the two Haiku models’ results the tally reads 5 clearly helped · 4 with no clear difference · 9 with too few answers to tell, which is not evidence that the skill did nothing. Documentation clearly helped Claude Haiku 4.5 in two of three runs and Claude Haiku 5.5 in none; code review and git workflow never clearly helped either model in more than one run. Every draw was judged with claude-opus-5, which departs from the judge policy; its tally compares with those of Reports #009, #011 and #013, not with the reports judged under that policy.

The part I did not expect is what the broken run had done to the error bars. The first run lost 25 draws to that timeout, 24 of them in a single cell, and I re-ran that cell clean. The truncated run had drawn more and measured less, and it reported a maximum variance ratio of 1.55x where the clean run reports 5.88x. A timeout takes the long generations first, and the long generations are the ones carrying the spread, so the failure did not lose data at random: it truncated the distribution from above and biased the variance estimate downward, which is the direction that makes an instrument look more precise than it is. It also cost the study the one apparent result it had. Both runs are published, the broken one as evidence.

None of this is new to anyone who works on evaluation; the statistics have been in the literature for years, and the seventh report cites the work that says so. What I can say is that I turned it on my own published figures rather than someone else's. Three cells in #005 now carry a v1.2 amendment naming their published lifts as single-draw, judge-spread figures. #006 gained an amendment of its own: a cell reported as six cases against six turned out to be six rows over two different case sets, each having lost a different arm to the same timeout, and it reads +0.031 rather than +0.055 once the pairing is fixed. Its verdict did not change. Neither amendment edits a published figure; both sit beside them, versioned and dated.

Some honesty about the instrument, because a measurement tool that is not honest about itself is worthless. I wrote the eval suites; that authorship is disclosed, and every rubric criterion must trace to text the skill's own SKILL.md contains. When my own re-audit found one criterion importing outside knowledge, I tightened the standard, re-ran, and republished with a visible amendment; the maintainer of the largest affected skill pack was auditing the suites at the time, and the stricter standard went further than he asked. One of those amendments deleted a false regression. Another exposed a real one the earlier rubric had missed. The reports also carry low-resolution flags where the judge quantizes, and Report #001 is on its third published version because corrections belong in daylight. This page is a correction too: it carried Report #001's v1.1 tally, which v1.2 had superseded twelve days before this essay was published, and it stayed up that way for seventeen days. That is the staleness the tool exists to catch, on the one page nothing was checking. There is a check now, and it fails the build if a figure here disagrees with the report it cites.

What I take from thirteen reports: skills are not static assets. They are bets on a moving substrate, re-priced by every release, every provider switch and every tier change. And re-priced in the literal sense as well, which is the half I was missing when I first wrote this: a verdict without a price is half an answer, so the value report puts the dollars and the latency next to the lift instead of behind it. The newest lesson is about the ruler rather than the thing it measures: an instrument that loses data quietly does not merely lose it, it flatters itself, and the only reason I can say so is that both runs are published. The tenth report’s lesson is that a harness verdict is one draw, and it travels with its configuration. Measured, disclosed, and in the strangest cases, still unexplained: the receipts show the sign flips; the mechanism is future work.

The spec, runner, receipts, and reports are open (Apache-2.0) at driftproofhq.com. There is an npx quickstart, a GitHub Action that fails your CI when a skill regresses on a new model, and a badge. If you maintain a skill, "tested once" is not a durable answer, and now there is a way to know when it stops being true.

Verdicts age because the substrate moves.