Report #005 — what does a skill cost to run?

• Accuracy lift (band). with_skill mean ± stddev over 5 judge samples, against the no-skill baseline on the same substrate. A lift is only called real when a per-case band separation clears the 0.05 effect floor — the same anti-cry-wolf rule as every earlier report. Not a model ranking: the comparison is always with-skill vs without-skill within one substrate.
• Δ tokens (cost). The change in the whole call's token footprint with the skill loaded — input and output. The skill's own text is one part of the input delta and usually the smaller part: across these 30 cells the input delta tracks cost at r = +0.92 while the skill's own length tracks it at only r = +0.33, and one 738-token skill drew 34× its own size in extra input. What moves the number is how the skill changes the path the model takes, multiplied by the substrate's price per token — the same skill at near-identical token deltas costs 3.3× more on claude-fable-5 than on claude-sonnet-5, which is exactly their input-rate ratio in the frozen snapshot — the rates differ, not the behaviour. The change is not always upward: in 7 of 30 cells the skill made the call cheaper, and in 8 it pulled in less input — not the same set, since a skill can read less and still write more. This is the durable cost fact: it is what the skill actually consumed, and it does not change when a vendor reprices. The dollar figure beneath is a derived view of exactly these tokens at the rates frozen into the receipt. Not a bill: on subscription surfaces actual metered spend is $0 and the dollars are metered-equivalent. Not a total cost of ownership — it is the skill's marginal cost, the number that changes when you adopt it. Output length is not a quality signal in either direction; longer is not better.
• Δ latency. Median wall-clock added, observed on subscription CLI surface, indicative. Not a serving-latency benchmark: it includes CLI cold starts and the vendor's own harness, so it describes what a user of that surface experiences, not the model's speed.
• Cost / benefit. What one unit of measured benefit costs — dollars per 0.01 lift, derived from the two columns to its left. The denominator is the cell's aggregate lift, not the lift of the single case that drove it. The cost is paid on every case in the suite, so the benefit it buys must be averaged over those same cases; pricing a whole-suite cost against one case's lift would mix populations and read 2.5–6× cheaper than the measurement supports. Four states, and only one of them is a price: a floor-clearing positive lift is priced; a cell whose drivers cleared the floor while its aggregate did not reads n/a (driver-only), with those drivers named under Verdict basis; a cell with no separated driver at all reads n/a (within noise); and a skill that measurably hurt prices nothing. Where a skill improved quality and reduced cost, the cell states the saving instead of a price — there is no cost per unit of benefit when the benefit is free.
• There is no composite score. The three axes have different units, different error bars, and different owners. Collapsing them into one number would manufacture a figure no reader could trace to evidence, so the report shows the three and leaves the weighing to you.

• Tokens are the measurement; dollars are derived. The cost column leads with the token delta — what the skill actually consumed, which does not change when a vendor reprices. The dollar figure beneath it is a derived view of exactly those tokens at the rates frozen into the receipt, and every one of them re-derives as (input/1e6 × input_rate) + (output/1e6 × output_rate).
• Pricing snapshot. Metered-equivalent, computed from each receipt's own frozen run.pricing_snapshot — priced as frozen 2026-08-18 (registry prices at run time) — never from the live registry, so these numbers do not change meaning when a vendor changes prices. Cached input tokens are costed at the list input rate (an over-estimate where caching is heavy); cached_tokens is recorded per call so a reader can recompute.
• Surfaces. claude-sonnet-5 → claude-cli (subscription; metered spend $0); claude-fable-5 → claude-cli (subscription; metered spend $0); gpt-5.6-sol → openai-cli (subscription; metered spend $0). Disclosed per the neutrality policy.
• The judge is excluded — and measurement is the expensive half in time. Judging consumed 18.26 of the run's 20.47 compute hours (89%), against 2.21 hours of generation (n=5 judge calls per generation). In dollars it was $87.57 of $192.79 — cheaper than generation in aggregate. It cost more than the work it graded on claude-sonnet-5 (judge $29.37 vs generation $28.93, 1.5% more, in 7 of its 10 receipts) and gpt-5.6-sol (judge $28.97 vs generation $17.12, 69.3% more, in 9 of its 10 receipts), and 16 of 30 receipts overall — so "grading costs more" is substrate-dependent, and on claude-sonnet-5 the margin is thin enough that a small pricing move would flip it. Judge tokens are recorded separately on every receipt as judge_usage and enter no value figure — the receipt schema pins economics.judge_excluded to true. That spend is ours, for measuring; it is not a cost of running the skill.
• What this run cost us, and what the guard did. The run was projected at $20.89 (the figure produced at run time, frozen here — the token constants behind it are being replaced, and a later re-render must not silently restate what was projected on the day) and cost $192.79 metered-equivalent — 9.2× the projection — against a $40.00 guard that never fired, because it checked the projection rather than accrued spend. Actual metered spend was $0 (subscription surfaces), so nothing was billed. The projection's token constants did not model the fixed CLI harness preamble; the guard is being rebuilt to check accrual before the next run. We publish the measured figure, not the estimate: it is the same argument this report makes about skills, applied to us.
• Skill versions are pinned, not current. Every skill was measured at the commit pinned in suites/manifest.json. Checked against upstream on 2026-08-19: 8 of 10 are still byte-identical; code-review-and-quality and writing-plans have been revised upstream. No upstream revision responds to a Driftproof finding — no commit message or diff references this project. The influence has run the other way: a maintainer's fairness audit produced our in-text grounding policy, amended 2 of 10 suites, and changed a published verdict (Report #001 v1.2).
• Ratios are floor-gated. The cost-per-benefit cell prices one unit of measured benefit — dollars per 0.01 lift. Where a lift did not clear the effect floor, it reads n/a (within noise) rather than a number: dividing noise by a real cost produces a precise-looking figure with nothing under it.

Run started 2026-08-18 19:01 UTC — receipt-attested, identical on all 30 receipts. Compute: 20.47 h (2.21 h generation over 420 calls, 18.26 h judging over 420 case rows). At the run's concurrency of 2 that is ≈10.2 h elapsed, a derived finish of ≈2026-08-19 05:15 UTC — derived, not attested: a receipt records per-call wall-clock, not a run finish, and concurrency is a launch parameter rather than receipt evidence. 30 receipts (10 skills × 3 substrates, with/without × n=5, judged by claude-haiku-4-5). Surfaces: claude-sonnet-5 → claude-cli, claude-fable-5 → claude-cli, gpt-5.6-sol → openai-cli. Metered spend $0 (vendor subscription CLIs); metered-equivalent $192.79 ($105.22 generation + $87.57 judge measurement, derived from the receipts at their frozen rates; verification level TESTED, the weakest among the 30 receipts summed). The $40 guard did not fire: it checked the $20.89 projection, not accrued spend — see the disclosure above. All receipts complete. [resume: 0 restored, 30 skipped]

v1.2 · 2026-09-01. The three cells v1.1 flagged have been re-measured, and their published lifts are named here as what they are: single-draw, judge-spread figures. Each of the three lifts on this page rests on one generation per arm, and its band is the spread of the judge re-scoring that single response. Report 007 re-ran all three cells with the generation sampled adaptively, three draws per arm minimum and more until the across-draw spread settled, and found lower lifts in every one: code-review-and-quality on claude-fable-5 +0.103 to +0.055; git-workflow-and-versioning on claude-sonnet-5 +0.116 to -0.002; writing-plans on claude-fable-5 +0.177 to +0.131. At the cell level none of the three separates from its own band.

These are not corrections, and Report 007 does not present them as such. Two of the three comparisons were refused before any verdict was formed, by the same baseline-reproduction control that refused every cell of Report 006: the no-skill arm did not reproduce the arm this page measured. On the third, writing-plans on claude-fable-5, the control passed and the comparison rendered WITHIN NOISE, which says the two measurements are the same as far as the instrument can tell. All three cells also changed skill.content_hash upstream between the two runs, so none of the pairs is a straight re-measurement of the same text.

v1.1's "no cause is asserted" holds, for the same reason and one more. Sampling noise accounts for the observations without needing a cause; whether anything else also changed is not established either way. Report 007 adds a second, independent instrument change to the same gap: a declared 300 s call timeout on this surface had never executed, because a numeric literal at the call site shadowed it from 2026-07-27 onward. Every run behind this page used 120 s. The question stays open.

What this qualifies on the page above, in that page's own words. Two sentences already published here are the ones this amendment bears on, and they are quoted rather than paraphrased. The first is the cost-per-benefit rule: “The denominator is the cell’s aggregate lift, not the lift of the single case that drove it. The cost is paid on every case in the suite, so the benefit it buys must be averaged over those same cases; pricing a whole-suite cost against one case’s lift would mix populations and read 2.5–6× cheaper than the measurement supports.” Every $ per 0.01 lift figure on this page divides by a cell's aggregate lift, and two of those denominators are the lifts under revision here. The band this page prints beside each cell is the with_skill band, not the lift band, and the two are different quantities: where this page reads 0.889 ± 0.023 the receipt's delta_uncertainty is 0.216857. Nothing above has been swapped for anything else.

The second is this page's own fairness disclosure: “Checked against upstream on 2026-08-19: 8 of 10 are still byte-identical; code-review-and-quality and writing-plans have been revised upstream.” That sentence is why the 005 to 007 comparison is a revision pair on all three cells rather than a re-measurement of the same text, and why v1.2 cannot present Report 007's lower lifts as this page's figures measured again.

Unaffected, and unchanged from v1.1. The cost-driver correlation, the substrate-disagreement result and the three-axis presentation do not rest on single-cell baselines and are not qualified here. No figure on this page has been edited; every number, count and verdict above is as published and re-derives from the same 30 receipts.

v1.1 · 2026-08-29 — Three cells rest on single-draw baselines now shown unstable. code-review-and-quality on claude-fable-5, git-workflow-and-versioning on claude-sonnet-5, and writing-plans on claude-fable-5 each report a lift measured against a baseline arm generated once. A 20-draw stability probe run for Report #006 shows two of those baselines sit in distributions wide enough that a single draw does not pin them: one spans 0.300–0.808 across ten draws, the other sits between 0.214 and 0.300 in nine draws and reaches 0.856 in one. Generation-level variation exceeds judge-level variation by 3.2× and 7.5× on those cases, and this report sampled the judge five times and the generation once.

What this does not say. It does not say these lifts are wrong, and it does not supply corrected ones. Report #006 re-measured the same three cells and got lower lifts, but those re-measurements are equally single-draw and are not replacements — on commit-message-conventional-type the published 0.282 is the modal value across ten draws while #006’s 0.852 was the roughly 1-in-10 outcome, so the fresh number is the likelier fluke of the two. No cause is asserted. Sampling noise accounts for the observations without needing one; whether anything else also changed is not established either way, and the one-generation design of this report cannot settle it. The question is recorded as open.

Unaffected. The headline findings of this report do not rest on single-cell baselines and are not qualified here: the cost-driver correlation (input-token delta tracks cost at r = +0.92 against skill length at r = +0.33), the substrate-disagreement result, and the three-axis presentation stand as published. Every figure above is left legible and unedited. Methods: Report #006, stability probe.

v1.0.1 · 2026-08-19 — Publication-chrome correction. No measured value changed: every figure, count and verdict on this page is identical to v1.0, and all 30 receipts are unchanged. At first publication the page’s <title> element and its footer both still carried the draft marker. The publication switch removed the noindex robots tag and the draft banner, but was never applied to those two places, so a page that was published in every other respect still announced itself as unpublished in the browser tab. Both are now rendered from the same switch, and the repo gate asserts the property rather than the two known places: no published report page may carry that marker anywhere, checked across every published report, so the defect cannot recur silently on a later one.

v1.0 · 2026-08-19 — First publication.

v1.3 · 2026-09-14. This entry corrects how the page’s band wording is read; it follows the entries recorded before it, no earlier text is changed, and no figure, verdict token, table value or receipt reference changes. The lift column marks cells (within noise), the cost-per-benefit cells and the method notes read n/a (within noise), and the entry of 2026-09-01 says a comparison “rendered WITHIN NOISE, which says the two measurements are the same as far as the instrument can tell”. The ten cells marked within noise have no case separated under the rule at the sample size used: no separation detected, which is not evidence of equivalence and not evidence that nothing changed. The comparison that entry quotes, Report 005 against Report 007 on writing-plans, had none of its seven with_skill cases separated, so it did not show the two measurements are the same: it detected no separation between them. The lift column’s mark and the cost-per-benefit label are left as published on this page; for the cost-per-benefit cell the renderer now writes n/a (no separation detected), and it still writes the lift column’s (within noise) mark as published. How to read this page. A case is separated under the rule when its two bands do not overlap and its mean moved by at least the 0.05 effect floor; bands that do not overlap by a smaller move are not separated. No cell’s aggregate lift on this page is separated: in every cell the with_skill and baseline aggregate bands overlap. A cell’s separations are its cases’. Wherever the page reads a result with no separation detected as settled, as within noise, dividing noise by a real cost, or a figure with nothing under it, read it as no separation detected at the sample size used, which is not evidence that nothing is there; of the sixteen cells whose aggregate lift is below the floor, six have a case separated and read n/a (driver-only). Sampling noise that accounts for the observations, and a draw that is the likelier fluke, are causes offered for baseline cases that Reports 006 and 007 found separated, from draws to which no separation rule was applied: an observation, not proof of a cause. Wherever it reads a separation in a cell’s cases as a settled property of the skill, as a lift called real, the effect is a clean grid step, or The verdict stands, read it as a separation detected under the rule in one or more cases, which is not proof that the skill moved the score. Wherever it reads a saving as a settled improvement, as having improved quality while reducing cost or the benefit is free, read it as a floor-clearing aggregate lift that is not separated under the rule, in a cell with a separation detected in one or more cases, which is not proof that the skill raised quality or cut cost. The no-skill arm that did not reproduce, in the entry of 2026-09-01, reads a baseline control that found baseline cases whose bands did not overlap, which is not proof that a baseline changed. The page’s a skill that measurably hurt names a rendering for a floor-clearing negative aggregate lift, which no cell on this page has. The baselines the entry of 2026-08-29 calls now shown unstable rest on the spread of Report 006’s draws, to which no separation rule was applied. Labels, tallies and reason strings are read the same way and are left as published. The bands. The band printed beside each cell, under the header lift (band) and in the legend’s Accuracy lift (band), is not a lift band: it is the cell’s with_skill mean plus or minus the sample standard deviation of its seven per-case means, which is suite dispersion, and the legend’s mean ± stddev over 5 judge samples and the lede’s the accuracy lift with its bands read as that. The per-case bands the Verdict basis and the low-resolution note describe are a case’s mean plus or minus the sample standard deviation of its five judge samples. The lift band the entry of 2026-09-01 names, delta_uncertainty, combines the two arms’ suite dispersions in quadrature; that entry’s its band is the spread of the judge re-scoring that single response describes neither the band printed beside a cell nor the lift band, and its none of the three separates from its own band compares a lift with its lift band, which is not the separation rule. The different error bars of the axes are different descriptive spreads. Each band is a descriptive spread, not a confidence interval, with no coverage probability. Filed under the wording rules of the repository's spec 031, amendment A-031-20.

v1.4 · 2026-10-03. This entry adds a dated note under the headline; no earlier text is changed, and no figure, verdict token, table value or receipt reference changes. The note says the headline counts on this page include receipts whose receipt pages mark them Not measured, and that corrected counts follow in spec 159. It was written after each of the 30 receipts this page links was read with the badge's own rule, and all 30 read Not measured.