Report #006 — the skill text moves, the substrate holds still

• The comparison is text vs text, on one substrate. Each cell reuses #005's receipt as the pinned-text arm and runs a fresh current-text arm on the same model, provider, surface, suite and judge. The verdict rule is the one every Driftproof report shares and this one did not touch: a case counts only when its with_skill bands do not overlap and the mean moves at least 0.05.
• The reused arm is a tested prediction, not an assumption. A receipt records run.model_release_date as null, so id equality alone cannot prove the substrate held still while the text moved — a provider that re-points a concrete id at a new snapshot is invisible to it. Every fresh run also emits a baseline arm, which contains no skill text and which the revision cannot touch by construction, so comparing it against #005's baseline re-measures exactly that, at no extra cost. A cell whose baselines do not reproduce is NOT MEASURED, and no verdict is published for it. The per-cell controls are below, and each one is a committed record, not a log line.
• One substrate per skill, chosen by a rule stated in advance. The substrate where #005 measured that skill's largest effect — where a revision has the most room to show, and where a null is most informative. For writing-plans that rule picks claude-fable-5 over claude-sonnet-5 by one thousandth (+0.177 against +0.176), far inside its own band. That is a tie, not a choice: running the sonnet-5 cell instead would have been an equally defensible application of the same rule, and the reader is entitled to know it.
• Not a claim that revisions are large. The report type's claim is that revision drift is measurable and separable from release drift. Two nulls and one signal is a result.
• Not a quality judgement of the upstream maintainers. A revision that does not move this suite may be doing something this suite does not measure — and in one cell below, that is explicitly the case.

• What this run cost, and what the guard believed. The three fresh cell-arms were pre-registered at a measured basis of $26.88, recomputed from #005's own economics blocks for exactly these cells, and cost $18.50 metered-equivalent. The cost guard's own projection for the same calls was $2.4553: it is a fixed token table that does not model the fixed CLI harness preamble, so --max-usd 2 per cell bounded call volume, not dollars. That under-projection is a standing property of lib/cost.js, visible because #005 finally measured the truth; it is filed, not fixed here. Actual metered spend was $0 — these are subscription CLI surfaces, and the dollar figures are the if-it-had-been-metered equivalent, counted against the caps identically because subscription attention is not free.
• The pinned arm was free, and that is a claim this report tests. Zero fresh pinned-text runs. The whole saving rests on the baseline-reproduction control above, which is why the control is reported per cell rather than summarised.
• Only revised skills appear. The other seven skills #005 measured are byte-identical to their pins and have no revision to measure; running one would spend money to produce a tautology.
• Third-party skill text is not committed. The current-text arm runs against an untracked workdir staged from the pinned raw URLs and verified by sha256 before the first model call. This repository records repo_sha, raw_url and content_sha256 only.
• Comparability metadata lags this report by one step. suites/manifest.json's comparable_reports still lists #001–#005. Adding #006 before it had receipts would have put a comparability claim in the manifest for a report that had measured nothing; it is recorded as a known divergence and resolves at publish time.

The control proves non-reproduction. It cannot say why. That distinction is the whole of this section. An earlier draft of this report asserted that the substrate had moved — an explanation invented to fit the shape of the failure, with nothing measuring it. This replaces that assertion with 120 model calls.

Design. Two of the six non-reproducing cases, 10 fresh draws each, 5 judge samples per draw — 120 calls, $0 metered on the same subscription surface. Each draw is a new generation of the baseline arm, which carries no skill text. Same judge (claude-haiku-4-5, n=5), same surface (claude-cli), and the rubric hashes are asserted equal to the report's: a953d281f6a9… and c17f2f214cec…. Held in evidence/probe-baseline-stability-20260828.json, with per-draw rows in evidence/probe-baseline-stability-20260828.draws.jsonl. It is marked not_a_receipt: true and is exploratory: it grades no skill and attests no verdict.

1. "The substrate moved." The first reading, from the control alone. It fit every observation and rested on nothing.
2. "The scores are bimodal." After the first draws showed a low mode and a high mode, this looked like a clean two-state story.
3. "Case 1 is a broad continuous spread; case 2 is tight with a rare high mode." Where 20 draws actually land. Case 1 runs 0.300–0.808 with draws at 0.506, 0.626, 0.686, 0.690, 0.732, 0.734, 0.774 — not two modes, a spread. Case 2 sits between 0.214 and 0.300 in nine draws and jumped to 0.856 in one.

The probe session retracted its own claim mid-run. With eight tight draws on case 2 in hand, it computed this report's re-measured 0.852 as roughly 122 standard deviations from the mean — a number that reads as impossible. Draw 9 then returned 0.856, landing squarely on the high mode and demolishing that statistic. The "122 sigma" figure was an artifact of estimating a standard deviation from a sample that had not yet met the tail. It is recorded here because a retraction that leaves no trace is not a retraction; the draw-by-draw sequence is in evidence/probe-baseline-stability-20260828.progress.txt.

• Neither number is authoritative at the tails. #005's pinned values and this report's re-measured values are both single draws from these spreads. Case 2 is the proof: #005's 0.282 is the modal value, and this report's 0.852 was the roughly 1-in-10 outcome. The fresh measurement can be the fluke. Nothing on this page presents a re-measured lift as a corrected value.
• The honest-labeling caveat. Today's draws measure noise now. Ruling out luck for the gap between the two runs — 2026-08-18 to 2026-08-28, both receipt-attested — assumes the noise level did not itself change over that interval, and #005's one-generation-per-cell design cannot confirm that either way.
• Rule (b) fired on both cases: generation-level noise explains every baseline gap examined. No substrate movement is needed to account for what the control saw. That is not the same as proving none occurred.
• Two cases, not six. The other four non-reproducing cases were not probed. Their gaps are unexplained rather than explained.

Sampling variance in LLM evaluation is established ground, not our discovery. Miller (2024), arXiv:2411.00640, sets out resampling and error bars for exactly this class of measurement; arXiv:2502.08943 quantifies how badly single-sample evaluation understates variance in benchmark scores. Both describe what the probe re-derived the hard way.

What we claim is narrower and, as far as we can tell, unoccupied: applying it to agent-skill verification, and turning it on our own published numbers. The literature says single-draw benchmarks are unreliable. This report is a verification project discovering that its own receipts inherited the defect, publishing the draws that show it, and changing the receipt specification because of it — rather than leaving the finding as a caveat about someone else's work.

Current-text arms run 2026-08-28 19:38 UTC, receipt-attested. 3 fresh receipts (3 skills × 1 substrate each, with/without × n=5, judged by claude-haiku-4-5), 252 model calls. Pinned-text arms reused from Report #005 at no cost. Surfaces: claude-fable-5 → claude-cli, claude-sonnet-5 → claude-cli. Metered spend $0 (vendor subscription CLIs); metered-equivalent $18.50 against a pre-registered basis of $26.88. Verification level: TESTED, the weakest among the receipts summed. Upstream revision scan: state/skill-version-check.json, the 2026-08-19 record #005's disclosure was written from — cited by path and date, not linked: it is kept in the source repository and build-public.sh excludes it from the published tree.

This report exists because #005's fairness disclosure named the exposure before anything had measured it: eight of ten skills were still byte-identical to their pins on 2026-08-19, and two were not. A third, git-workflow-and-versioning, was revised upstream after that check was made — #005's check was complete when it was made, and the timing is disclosed here rather than being allowed to look like an omission.

No cell in this report was measured, so nothing here amends a #005 figure — and nothing here confirms one either. A within-noise result would have said the pins were still representative. That is not what happened: every cell was refused by its own baseline control before a verdict existed, so this report has no evidence about the revisions in either direction.

It does raise a question about #005, and the probe above sharpens it without answering it. #005's per-cell lifts rest on baselines measured with one generation each, and two of those baselines are now shown to sit in spreads wide enough that a single draw does not pin them. This report's re-measured lifts are single draws from the same spreads and are not replacements for #005's figures. On commit-message-conventional-type the direction of the doubt actually runs toward this report: #005's 0.282 is the modal value across ten draws, and the 0.852 measured here was the roughly 1-in-10 outcome. The question of what #005's lifts would be under proper generation sampling is left open, because nothing run so far can close it.

v1.1 · 2026-09-01. The writing-plans cell's aggregate was computed over two different case sets, and is corrected here. This page publishes that cell at +0.055. Under correct pairing it is +0.031, and its band moves from ± 0.194 to ± 0.038.

The mechanism, because the counts hid it. The two arms were reported as 6 against 6, which looks paired and is not: two different cases each lost one arm to the same 120 s call timeout, task-right-sizing-testable-deliverable (baseline) and full-small-plan-header-and-tasks (with_skill). Filtering each arm's failures independently left six rows on each side over different case sets, so the aggregate subtracted a mean of one set of cases from a mean of another. Excluding a case from both arms when either arm is unmeasured gives 5 against 5, genuinely paired. The counts matching at 6 against 6 was a coincidence, which is why nothing on this page or in the receipt suggested anything was wrong. run.failed_case_count reads 2 before and after; what the receipt did not carry was the list, and excluded_cases now names it.

The band moves further than the lift does: ± 0.194 to ± 0.038, a fivefold narrowing, because the two cases that dropped out were the two widest. The corrected lift of +0.031 against ± 0.038 is still inside its own band, narrowly, as +0.055 against ± 0.194 was comfortably.

The verdict does not change, and neither does the control. This cell's classification stays NOT MEASURED. The baseline-reproduction control that produced it is driven by per-case movement, and it is recorded in a separate control record whose per-case table pairwise exclusion does not touch: moved_cases is still bite-sized-tdd-steps, interfaces-exact-signatures, two cases past the band and the 0.05 floor, and the reason string is unchanged. Exclusion changes aggregates only, and results.cases is left intact, so no per-case comparison moves.

One disclosed diagnostic inside the control block does move, and is stated rather than left to be found. aggregate_baseline_delta is computed over the baseline arm's aggregate, and that aggregate goes from 0.754 over 6 cases to 0.820 over 5. The figure therefore moves from +0.099 to +0.165. It is a diagnostic and not the verdict basis, since the reason string counts cases rather than reading the aggregate, but it is a published-adjacent number that this correction touches, and leaving it unmentioned would be the same omission in miniature.

No figure on this page has been edited, and the archived receipt is byte-unchanged. The corrected figures above are rebuilt from that receipt's own case rows through the current aggregate path. Filed by Report 007, which measured the timeout that caused both losses.

v1.0 · 2026-08-29 — First publication.