Report 007: the baseline is the noisy arm

• Surface. The surface is claude-cli with local machine context, one box, held constant across all cells and all reports.
• Scope. The instrument measures skill text in context on the append-system-prompt surface. Out of scope: routing via description:, progressive disclosure, and tool execution.
• What moved. Nothing underneath the skill. Same suites, same substrates, same fixed judge (claude-haiku-4-5, five samples per draw) as Report 005. The instrument moved instead: receipt spec v0.5 samples the generation adaptively, a minimum of three draws per arm and more until the across-draw spread settles, and a call now gets the timeout its surface policy declares.
• The set. Three cells, 21 cases, 144 draws, 143 measured. Every verdict below is verifiable from the four receipts linked at the foot of this page: three published, one retained as defect evidence.

Two of the eighteen are the ones a reader will argue with. bite-sized-tdd-steps lifts +0.120 and interfaces-exact-signatures lifts +0.104, each more than twice the 0.05 floor, and each reads no effect. The reason sits on the same row: their baseline bands are ± 0.155 and ± 0.144. A lift twice the effect floor is not a finding when the baseline it is measured against will not sit still. Exactly one of the eighteen fails on the floor rather than on overlap: self-review-spec-coverage, at +0.028.

A declared 300 s timeout had never executed. lib/provider.js declares a per-surface retry policy, and for a claude-cli surface it declares 300000 ms, with a written rationale about cold-start-dominated subprocesses. lib/run.js then set its own default as a numeric literal, 120000, and the provider layer documents that an explicit caller value wins. So every run this project has ever made used 120 s on every surface, and the CLI policy was dead text from the day it was written.

The dates are the uncomfortable part. The literals landed on 2026-07-27 with the runner skeleton. The policy that they shadowed was written on 2026-07-31, four days later, and read as authoritative from then on. Nothing failed, nothing logged, and no gate reached it until preparing this report did. It was fixed on 2026-08-31.

Run 1 lost 25 draws to it, 24 of them in writing-plans@claude-fable-5. Every loss carries the same recorded reason, and the string names the number: provider(claude-cli) timed out after 120000ms. Run 2 re-ran that one cell under the corrected timeout and measured 55 of 55, with nothing lost.

Both runs are published. Run 1's three receipts are committed at 7468e9e and run 2's at 7050fd8. Run 1's writing-plans@claude-fable-5 receipt is retained as defect evidence and is not part of the published set; the other two cells of run 1 are the published receipts for their skills, and were never re-run.

The truncated run drew more, measured less, and looked calmer. 71 draws against 55, 5.07 per arm against 3.93, and a maximum variance ratio of 1.55× where the clean run reports 5.88×. 1 of 13 arms sat above 1× in run 1; 5 of 14 do in run 2.

The mechanism is the ordering. A 120 s ceiling removes the long generations first, and the long scattered generations are the ones carrying the across-draw spread. So the timeout did not merely lose data at random: it truncated the distribution from above, and biased the variance estimate downward. That is the direction that makes an instrument look more precise than it is. Silent truncation flatters stability.

It also cost this study its one apparent result. Run 1's writing-plans@claude-fable-5 lift of +0.116 ± 0.115 sat outside its own band by about a thousandth: the only separation anywhere in Report 007. Run 2's +0.131 ± 0.157 does not. Run 1 reported three improvements out of six measurable cases and could not measure the seventh at all; run 2, with every draw measured, reports two out of seven, and the case run 1 lost, full-small-plan-header-and-tasks, turns out to carry the largest case lift in the whole study at +0.482. The one result the study appeared to have was an artifact of the draws it lost.

One arm of run 1 lost every draw it took, so its variance ratio is null with variance_ratio_unavailable: "no_measured_draws" rather than a fabricated zero. That is why run 1 reports 13 ratios over 14 arms and run 2 reports 14.

The one comparison that passed its precondition. writing-plans@claude-fable-5, Report 005 to Report 007, --mode revision: WITHIN NOISE — the revision moved no case beyond its confidence band; the pinned text and the current text measure the same.

with_skill mean moved +0.002 (0.831 ± 0.092 → 0.833 ± 0.074; band = suite dispersion). Per-case band-overlap verdicts: 0 regression(s), 0 improvement(s), 7 within noise.

So on the one cell where the control held, a lift of +0.177 and a lift of +0.131 are the same measurement as far as this instrument can tell. That is the honest reading of the mapping table above, and it is why none of the three lower lifts is presented as a correction.

2 of 3 cells are cheaper with the skill than without it, at -$110.884 per thousand calls on code-review-and-quality@claude-fable-5 and -$117.997 per thousand calls on writing-plans@claude-fable-5. The mechanism is visible in the tokens rather than asserted: mean input falls by 8,557 and 17,843 tokens per call, which is the baseline's sprawl becoming unnecessary. The one cell that costs more, git-workflow-and-versioning@claude-sonnet-5, costs +$12.264 per thousand calls, and it is also the cell with no measured benefit at all: a lift of -0.002 and zero improved cases. Latency falls in 3 of 3.

A caution the tables carry and a reader should not skip: call counts are draw counts, and draw counts differ per arm. writing-plans@claude-fable-5's baseline drew 31 calls against with_skill's 24, because the baseline was unstable enough to keep drawing. Per-call means are therefore computed over differently sized draw sets in the two arms. That is correct, since a draw is a call and each measured draw contributes one usage row, but it means the cost delta and the score delta are averages over unequal n, and a reader who assumes matched n will misread both.

Two mechanical notes on the absolute columns. A CLI call carries a fixed harness preamble of roughly 25k input tokens that is identical in both arms and cancels in the delta, which is why the incremental row is the one to read. Cached input tokens are costed at the list input rate, which over-estimates; cached_tokens is recorded per call so a reader can recompute on other assumptions.

The published set is not fully clean, and one absorbed draw sits inside it. code-review-and-quality@claude-fable-5's severity-labeled-findings baseline arm drew 5 and measured 4: one generation call was lost to the 120 s timeout, and the band for that arm was computed from the 4 that survived. It produced no failed_timeout, no exclusion and no failed_case_count, because 4 measured draws is a legal draw set. That is silent absorption, and it is the same failure mode this report's instrument section describes, sitting in a cell this report publishes. It is one draw of 144 and it does not move the verdict, which reads improved at +0.332 on a wide baseline band either way. But "run 2 fixed it" is true of writing-plans@claude-fable-5 only. The other two cells were never re-run, and this one still carries the absorbed draw.

Run 1's receipts were sealed with null economics, and the reader was fixed after the sealing. Both run-1 receipts record call_count: 0 and null in every arm field. generationUsages read only the case-level usage, and a v0.5 receipt puts usage on the draws, so the arm figures came out empty while the draws carried complete usage the whole time. The judge block was never affected, because judge_usage stayed at case level, and that asymmetry is what located the defect. The economics tables above are recomputed from the same committed receipts through the corrected armEconomics, with no re-run and no edit to any receipt. writing-plans@claude-fable-5's run-2 receipt was sealed after the fix, and its recorded block reproduces field for field under recomputation, which is the control on the other two.

The per-case table above is computed by a path with no command behind it. driftproof diff compares two receipts. There is no shipped command that renders within-report, per-case, skill-against-baseline verdicts: a receipt carries one aggregate comparison.delta, and lib/verdict.js reads that aggregate through the effect floor with no band test at all. This page therefore calls the shipped rule functions directly, bandOf from lib/reuse.js and the band-and-floor rule from lib/diff.js, over each receipt's own case rows. No number is re-derived by hand, but the headline per-case table has no command a reader can run to get it, and no assertion over that command. It is filed as a gap in the tool, disclosed here rather than after someone notices.

Surface. The surface is claude-cli with local machine context, one box, held constant across all cells and all reports.

Scope. The instrument measures skill text in context on the append-system-prompt surface. Out of scope: routing via description:, progressive disclosure, and tool execution.

5 of the seven cases have a baseline wider than the with-skill arm, three of them by more than nineteen times. code-review-and-quality@claude-fable-5 shows the same shape more weakly (5 of seven wider, maximum 8.00×) and git-workflow-and-versioning@claude-sonnet-5 weakly again (5 of seven, maximum 2.73×).

What the shape suggests, without this report having tested it, is that the skill's most visible effect on this corpus is on consistency rather than on the mean. That would also explain why so few cases separate: a wide baseline band is precisely what prevents a real mean difference from clearing an overlap test. This is a hypothesis the design of Report 007 cannot settle. A mean-difference test is not an instrument for a variance-reduction effect, and nothing here was pre-registered as a variance claim, so the shape is recorded and left open.

Why we looked, not what we confirmed. Two lines of prior work motivated these two tables and neither is evidence for them. Prompt rankings are unstable under ordinary evaluation variability, and ranking by a lower confidence bound rather than by a point estimate is a documented response: On the Stability of Prompt Ranking in Large Language Model Evaluation, arXiv:2606.24381. A wide baseline band is exactly the condition under which that selection rule matters, and Driftproof does not currently apply one. The variance-component framing above is CyclicJudge, arXiv:2603.01865, cited again here for the same decomposition.

• This report has not corrected Report 005. Two of the three archive comparisons were refused on baseline non-reproduction, so for those two cells nothing here is comparable to what that page published.
• The skill text changed between the two runs on all three cells. Every 005-to-007 pair differs in skill.content_hash, so even the one comparison that passed its control is a revision pair and not a straight re-measurement of the same text.
• The across-draw bands are small-budget estimates, uncorrected. Three to six draws per arm is a finite repeat budget, and the literature cited below shows estimators in this regime carry bias whose direction depends on what is being estimated. No debiasing is applied on this page; the bands are the plain sample spreads the receipts record.
• Cost means are over unequal n. Draw counts differ per arm, so a cell's two per-call means are averages over differently sized call sets.
• One absorbed draw remains in the published set, in code-review-and-quality@claude-fable-5, disclosed above.
• The judge remains single-model (claude-haiku-4-5, five samples), and judge affinity is not measured here.
• Three cells is three cells. Twenty-one cases on two substrates is not a corpus, and nothing on this page generalises past the skills named on it.
• Every verdict is verifiable from the receipts, which are linked below rather than described.

v1.1 · 2026-09-02. Two sentences in this page's economics section read a mechanism out of a basis this page did not control for, and are qualified here. They are quoted rather than paraphrased. The first: “2 of 3 cells are cheaper with the skill than without it.” The second: “The mechanism is visible in the tokens rather than asserted: mean input falls by 8,557 and 17,843 tokens per call, which is the baseline's sprawl becoming unnecessary.” Both figures are correct as arithmetic over the receipts. Neither sentence is a safe reading of them.

What the basis actually was. The incremental column is one per-call mean minus another, and the two means are taken over call sets that differ in size and in composition: 21 draws with the skill against 22 without, allocated 3,3,3,3,3,3,3 against 4,3,3,3,3,3,3 on code-review-and-quality@claude-fable-5; 23 draws with the skill against 22 without, allocated 3,3,3,3,3,5,3 against 3,3,3,3,3,4,3 on git-workflow-and-versioning@claude-sonnet-5; 24 draws with the skill against 31 without, allocated 3,3,3,3,6,3,3 against 4,5,3,4,6,6,3 on writing-plans@claude-fable-5. Adaptive stopping chose those allocations per arm, so the two arms of a cell are averages over different case mixes. On top of that, every input token in both means is priced at the list input rate, cached tokens included, which is what the tables' own caption says and what computeEconomics does. On this surface cached context runs 89.8% to 95.1% of each call's input. The largest term in each mean is therefore harness context that neither arm chose and that the skill cannot move, priced at full rate, and the difference of two such means over unequal call sets is what the headline sentence read a mechanism out of. This page's Limits already stated the first half of that, “Cost means are over unequal n”; what it did not state is that the headline sentence rested on it.

The same draws, re-priced with cached input excluded. Nothing is re-run and no receipt is touched: cached_tokens is recorded per draw so that a reader can do exactly this, and the receipts say so in economics.notes.cache_pricing. Fresh input means input minus cached, at the same frozen rates, over the same measured draws.

2 of the 3 cells change sign, and they are not the ones the sentence would lead a reader to expect. On the fresh basis the cheaper cells are code-review-and-quality@claude-fable-5 and git-workflow-and-versioning@claude-sonnet-5, and the dearer one is writing-plans@claude-fable-5. The count of two survives; the identities do not. writing-plans@claude-fable-5, named in the sentence as one of the two cheaper cells at -$0.117997 per call, is +$0.040498 on fresh input, the dearest of the three. git-workflow-and-versioning@claude-sonnet-5, named as the one cell that costs more, is -$0.002212, cheaper. Only code-review-and-quality@claude-fable-5 keeps its sign, and its magnitude falls to 26.0% of the published figure.

The mechanism claim is retracted as written. “The baseline's sprawl becoming unnecessary” was offered as the explanation of an input-token fall, and the fall is not in the tokens the model generated. Of the 8,557-token fall on code-review-and-quality@claude-fable-5, 8,204 tokens, 95.9%, are cached; of the 17,843-token fall on writing-plans@claude-fable-5, 15,850 tokens, 88.8%, are cached. On fresh input the two falls are 354 and 1,994 tokens per call. A skill's text cannot make a harness preamble shorter, and an arm without the skill prepended consuming more input than the arm with it is a fact about cache warmth and call ordering, not about sprawl.

What survives, restated to its actual axis. On the fresh basis every cell's incremental is carried by output length, not input. code-review-and-quality@claude-fable-5 is cheaper because the skilled arm emits 506 fewer output tokens per call, worth -$0.025312 against -$0.003537 from fresh input. git-workflow-and-versioning@claude-sonnet-5 is cheaper on the same axis, -$0.002671 from 178 fewer output tokens. writing-plans@claude-fable-5 is dearer on that axis for the same reason in reverse: +1,209 output tokens per call, worth +$0.060434, which more than repays its fresh-input fall. So the defensible sentence is narrower than the published one and points somewhere else: where this skill changes cost at all, it changes it by changing how much the model writes, and the direction is not the same in every cell. Two cells shorter, one cell longer, on differences that are small in absolute terms and were never metered.

Neither basis is the true one, and that is the point. Cached input is billed below list rate on metered surfaces and at nothing at all here, where the metered spend is $0.00; pricing it at list is an over-estimate the receipts declare, and excluding it entirely is an under-estimate. The published tables use the first, this amendment adds the second, and the honest statement is that the sign of this metric on this surface is a property of which one a reader picks. That is a fact about the measurement and not about the skill, and it should have been on the page before a mechanism was asserted from either.

No figure above has been edited and no table is withdrawn. Every number in the economics tables re-derives from the committed receipts through computeEconomics at each receipt's own frozen snapshot, and the receipts are byte-unchanged. The two bases are computed from the same recorded fields, so this amendment adds a column beside the published one rather than replacing it. Filed by Report 008, which found the same two sentences waiting to be written about a second model pair and did not write them.

Report 005 to v1.2. The three cells' published lifts are named as single-draw, judge-spread figures, re-measured here with no cell-level separation. No figure on that page is edited.

Report 006 to v1.1. That page's writing-plans cell is corrected under pairwise exclusion, from +0.055 to +0.031, and its 6 against 6 aggregate is restated as the 5 against 5 it should have been. The cell's verdict does not change and its baseline-reproduction control does not move.

Both amendments are versioned records appended to the pages they concern. Constitution invariant 4: a published report is amended visibly, never edited silently.

Run 1: 2 published cells plus one evidence cell, 2026-08-31 07:59 UTC, committed at 7468e9e. Run 2: writing-plans@claude-fable-5 re-run under the corrected timeout, 2026-08-31 19:52 UTC, committed at 7050fd8. Published set: 144 draws, 143 measured, 1 absorbed. Judge claude-haiku-4-5 at five samples per draw. Surfaces: claude-fable-5 to claude-cli, claude-sonnet-5 to claude-cli. Metered spend $0.00 on vendor subscription CLIs; every dollar on this page is the if-it-had-been-metered equivalent at each receipt's own frozen snapshot. Verification level TESTED on all four receipts.