Three model releases later: what actually happens to agent skills

By the Driftproof maintainer · published 2026-08-11 · revised 2026-09-03 to read the whole published record. It keeps the title it launched under; the record has outgrown it, which is rather the point.

Sonnet 5 shipped June 30. Opus 5 followed July 24. Most teams updated a model string and moved on. Almost nobody re-ran their skills.

Skills (the SKILL.md files that teach AI agents your conventions: code-review discipline, commit style, writing rules) are becoming how expertise gets encoded for the agent era. SkillsBench, the benchmark that indexes them, counts 47,000+ skills across 6,300+ public repositories (SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks, arXiv:2602.12670, read 2026-09-01). Almost none are tested once, let alone re-tested when the model underneath them changes. Every release silently re-prices every skill. I wanted to know by how much, so I built the checker.

Driftproof is a small open-source tool with one job: run a skill's eval suite with and without the skill, then say honestly whether anything moved. There are now six report types, and they differ in what moves underneath the skill: the model version (release drift), a different model behind a different vendor's CLI (substrate durability), a higher tier of the same provider (capability gap), nothing at all, the axes widening instead from accuracy to accuracy, cost and latency (value), the skill's own text while everything under it is held still (revision drift), or nothing outside the measuring device itself, when a corrected instrument re-runs cells an earlier report already published (instrument re-measurement). Every score is judged five times, so it carries a confidence band rather than a single fragile number. A verdict is claimed only when bands fully separate AND the mean moves at least 0.05, one judge quantization step. Overlapping bands are reported as within noise, never as drift. Every run emits a dated, hash-verified receipt anyone can verify.

Eight reports in, here is what the receipts say.

Report #001 (sonnet-4-6 to sonnet-5, 10 popular public skills): 9 of 10 moved beyond noise, and the tally, as it stands after the v1.2 amendment described below, is 3 improved, 2 regressed, 4 mixed, 1 within noise. Same files, same suites, same judge; only the model changed.

Report #002 (the same 10 suites, Claude against gpt-5.6-sol, one fixed judge): 3 durable, 4 substrate-dependent, 2 regressed, 1 no effect. The strangest rows are sign flips: a skill-authoring skill lifts output quality on Claude and reduces it on GPT; a code-review skill is the exact mirror. A skill tuned on one substrate can be dead weight, or worse, on another, and nothing warns you when you switch.

Report #003 (opus-4-8 to opus-5): 4 improved, 2 regressed, 0 mixed, 4 within noise. Both regressions are single cases on skills that are otherwise fine, one sitting exactly on the 0.05 floor. The sharpest signal is another sign flip, this time across versions rather than vendors: a skill that was net-negative on opus-4-8 is net-positive on opus-5, while three others went quiet because the new model does the thing without them.

Report #004 asks the question one tier up: does encoded expertise still lift output on the frontier tier of the same provider, claude-opus-5 against claude-fable-5? Over 10 skills: 3 durable, 5 tier-dependent, 0 regresses, 2 no effect. Encoded expertise survives the better model. Where lift had collapsed, it had collapsed on the flagship first, which is the opposite of the story I expected to write.

Report #005 stops asking only whether a skill helps and asks what it costs: the same suites on 3 substrates, with accuracy, cost and latency shown side by side and never folded into one score. In 20 of 30 skill by substrate pairs at least one case cleared the floor with separated bands. 14 cleared the floor on aggregate, of which 10 carry a price and 4 report a saving, having improved quality while reducing cost. The dominant cost driver turned out not to be the skill's own text but the input it causes the model to pull in: across those 30 cells the input delta tracks cost at r = +0.92 while the skill's own length tracks it at only r = +0.33, and one 738-token skill drew 34 times its own size in extra input. That cost-driver finding still stands and the page says so. Three of its cells no longer stand as published: they carry a v1.2 amendment naming their lifts as single-draw, judge-spread figures, for the reason the sixth and seventh reports established.

Then the sixth report refused itself. I built Report #006 to ask whether a skill's own text drifts, holding the model still and moving the words. To keep it cheap I reused #005's measurements as the fixed half of each comparison, on the argument that the ground beneath had not moved. My own control disagreed. It re-measured that ground, found it did not reproduce, and refused every cell before any of them produced a verdict: 3 cells, 0 measured, 3 refused. I had spent the budget and had nothing to report about revisions.

My first explanation was that the models had shifted under me. I had no evidence for it; it simply fit. So I measured instead: 120 calls, two cases, ten fresh draws each. The scores turned out to be single draws from wide distributions — one case landing anywhere from 0.300 to 0.808 — and the variation between generations ran three to seven times larger than the variation between judge samples. I sample the judge five times. I had been sampling the generation once. The error bars were on the smaller number.

So the receipt spec gained generation sampling, and the seventh report is that spec turned on the same three cells. Report #007 re-ran them with the generation sampled adaptively rather than once, and with a call timeout that finally did what the code had been declaring: a 300 s policy for this surface had been shadowed by a 120000 literal since 2026-07-27, four days before the policy that it outranked was even written. Under the corrected instrument the three cell lifts are +0.055, -0.002 and +0.131, every one of them smaller than its own band, and the 21 cases read 3 improved · 0 regressed · 18 no effect · 0 not measured. Nothing separates. Two of the three comparisons against the archive were refused outright by their own baseline control, and the skill text had moved upstream in the meantime, so none of the lower numbers is a correction and the report does not offer them as one.

The eighth report is the first one where the model moved and nothing else did. Report #008 re-measured two of the seventh report's cells on claude-fable-5-1 against claude-fable-5, with the skill text and the suite asserted byte-identical before the first call and with both sides generation-sampled rather than drawn once. Both cells came back within noise: 0 improved · 0 regressed · 14 within noise · 0 not measured. One case is held inside that verdict by the effect floor rather than by band overlap, and the report names it rather than leaving it in a table. A within-noise verdict is not a finding that nothing changed; it is a statement about what this instrument resolves at the sample size it ran at, and the report says so in its own words.

The part I did not expect is what the broken run had done to the error bars. The first run lost 25 draws to that timeout, 24 of them in a single cell, and I re-ran that cell clean. The truncated run had drawn more and measured less, and it reported a maximum variance ratio of 1.55x where the clean run reports 5.88x. A timeout takes the long generations first, and the long generations are the ones carrying the spread, so the failure did not lose data at random: it truncated the distribution from above and biased the variance estimate downward, which is the direction that makes an instrument look more precise than it is. It also cost the study the one apparent result it had. Both runs are published, the broken one as evidence.

None of this is new to anyone who works on evaluation; the statistics have been in the literature for years, and the seventh report cites the work that says so. What I can say is that I turned it on my own published figures rather than someone else's. Three cells in #005 now carry a v1.2 amendment naming their published lifts as single-draw, judge-spread figures. #006 gained an amendment of its own: a cell reported as six cases against six turned out to be six rows over two different case sets, each having lost a different arm to the same timeout, and it reads +0.031 rather than +0.055 once the pairing is fixed. Its verdict did not change. Neither amendment edits a published figure; both sit beside them, versioned and dated.

Some honesty about the instrument, because a measurement tool that is not honest about itself is worthless. I wrote the eval suites; that authorship is disclosed, and every rubric criterion must trace to text the skill's own SKILL.md contains. When my own re-audit found one criterion importing outside knowledge, I tightened the standard, re-ran, and republished with a visible amendment; the maintainer of the largest affected skill pack was auditing the suites at the time, and the stricter standard went further than he asked. One of those amendments deleted a false regression. Another exposed a real one the earlier rubric had missed. The reports also carry low-resolution flags where the judge quantizes, and Report #001 is on its third published version because corrections belong in daylight. This page is a correction too: it carried Report #001's v1.1 tally, which v1.2 had superseded twelve days before this essay was published, and it stayed up that way for seventeen days. That is the staleness the tool exists to catch, on the one page nothing was checking. There is a check now, and it fails the build if a figure here disagrees with the report it cites.

What I take from eight reports: skills are not static assets. They are bets on a moving substrate, re-priced by every release, every provider switch and every tier change. And re-priced in the literal sense as well, which is the half I was missing when I first wrote this: a verdict without a price is half an answer, so the value report puts the dollars and the latency next to the lift instead of behind it. The newest lesson is about the ruler rather than the thing it measures: an instrument that loses data quietly does not merely lose it, it flatters itself, and the only reason I can say so is that both runs are published. Measured, disclosed, and in the strangest cases, still unexplained: the receipts show the sign flips; the mechanism is future work.

The spec, runner, receipts, and reports are open (Apache-2.0) at driftproofhq.com. There is an npx quickstart, a GitHub Action that fails your CI when a skill regresses on a new model, and a badge. If you maintain a skill, "tested once" is not a durable answer, and now there is a way to know when it stops being true.

Verdicts age because the substrate moves.