Report #005 — what does a skill cost to run?

• Accuracy lift (band). with_skill mean ± stddev over 5 judge samples, against the no-skill baseline on the same substrate. A lift is only called real when a per-case band separation clears the 0.05 effect floor — the same anti-cry-wolf rule as every earlier report. Not a model ranking: the comparison is always with-skill vs without-skill within one substrate.
• Δ tokens (cost). The change in the whole call's token footprint with the skill loaded — input and output. The skill's own text is one part of the input delta and usually the smaller part: across these 30 cells the input delta tracks cost at r = +0.92 while the skill's own length tracks it at only r = +0.33, and one 738-token skill drew 34× its own size in extra input. What moves the number is how the skill changes the path the model takes, multiplied by the substrate's price per token — the same skill at near-identical token deltas costs 3.3× more on claude-fable-5 than on claude-sonnet-5, which is exactly their input-rate ratio in the frozen snapshot — the rates differ, not the behaviour. The change is not always upward: in 7 of 30 cells the skill made the call cheaper, and in 8 it pulled in less input — not the same set, since a skill can read less and still write more. This is the durable cost fact: it is what the skill actually consumed, and it does not change when a vendor reprices. The dollar figure beneath is a derived view of exactly these tokens at the rates frozen into the receipt. Not a bill: on subscription surfaces actual metered spend is $0 and the dollars are metered-equivalent. Not a total cost of ownership — it is the skill's marginal cost, the number that changes when you adopt it. Output length is not a quality signal in either direction; longer is not better.
• Δ latency. Median wall-clock added, observed on subscription CLI surface, indicative. Not a serving-latency benchmark: it includes CLI cold starts and the vendor's own harness, so it describes what a user of that surface experiences, not the model's speed.
• Cost / benefit. What one unit of measured benefit costs — dollars per 0.01 lift, derived from the two columns to its left. The denominator is the cell's aggregate lift, not the lift of the single case that drove it. The cost is paid on every case in the suite, so the benefit it buys must be averaged over those same cases; pricing a whole-suite cost against one case's lift would mix populations and read 2.5–6× cheaper than the measurement supports. Four states, and only one of them is a price: a floor-clearing positive lift is priced; a cell whose drivers cleared the floor while its aggregate did not reads n/a (driver-only), with those drivers named under Verdict basis; a cell with no separated driver at all reads n/a (within noise); and a skill that measurably hurt prices nothing. Where a skill improved quality and reduced cost, the cell states the saving instead of a price — there is no cost per unit of benefit when the benefit is free.
• There is no composite score. The three axes have different units, different error bars, and different owners. Collapsing them into one number would manufacture a figure no reader could trace to evidence, so the report shows the three and leaves the weighing to you.

• Tokens are the measurement; dollars are derived. The cost column leads with the token delta — what the skill actually consumed, which does not change when a vendor reprices. The dollar figure beneath it is a derived view of exactly those tokens at the rates frozen into the receipt, and every one of them re-derives as (input/1e6 × input_rate) + (output/1e6 × output_rate).
• Pricing snapshot. Metered-equivalent, computed from each receipt's own frozen run.pricing_snapshot — priced as frozen 2026-08-18 (registry prices at run time) — never from the live registry, so these numbers do not change meaning when a vendor changes prices. Cached input tokens are costed at the list input rate (an over-estimate where caching is heavy); cached_tokens is recorded per call so a reader can recompute.
• Surfaces. claude-sonnet-5 → claude-cli (subscription; metered spend $0); claude-fable-5 → claude-cli (subscription; metered spend $0); gpt-5.6-sol → openai-cli (subscription; metered spend $0). Disclosed per the neutrality policy.
• The judge is excluded — and measurement is the expensive half in time. Judging consumed 18.26 of the run's 20.47 compute hours (89%), against 2.21 hours of generation (n=5 judge calls per generation). In dollars it was $87.57 of $192.79 — cheaper than generation in aggregate. It cost more than the work it graded on claude-sonnet-5 (judge $29.37 vs generation $28.93, 1.5% more, in 7 of its 10 receipts) and gpt-5.6-sol (judge $28.97 vs generation $17.12, 69.3% more, in 9 of its 10 receipts), and 16 of 30 receipts overall — so "grading costs more" is substrate-dependent, and on claude-sonnet-5 the margin is thin enough that a small pricing move would flip it. Judge tokens are recorded separately on every receipt as judge_usage and enter no value figure — the receipt schema pins economics.judge_excluded to true. That spend is ours, for measuring; it is not a cost of running the skill.
• What this run cost us, and what the guard did. The run was projected at $20.89 (the figure produced at run time, frozen here — the token constants behind it are being replaced, and a later re-render must not silently restate what was projected on the day) and cost $192.79 metered-equivalent — 9.2× the projection — against a $40.00 guard that never fired, because it checked the projection rather than accrued spend. Actual metered spend was $0 (subscription surfaces), so nothing was billed. The projection's token constants did not model the fixed CLI harness preamble; the guard is being rebuilt to check accrual before the next run. We publish the measured figure, not the estimate: it is the same argument this report makes about skills, applied to us.
• Skill versions are pinned, not current. Every skill was measured at the commit pinned in suites/manifest.json. Checked against upstream on 2026-08-19: 8 of 10 are still byte-identical; code-review-and-quality and writing-plans have been revised upstream. No upstream revision responds to a Driftproof finding — no commit message or diff references this project. The influence has run the other way: a maintainer's fairness audit produced our in-text grounding policy, amended 2 of 10 suites, and changed a published verdict (Report #001 v1.2).
• Ratios are floor-gated. The cost-per-benefit cell prices one unit of measured benefit — dollars per 0.01 lift. Where a lift did not clear the effect floor, it reads n/a (within noise) rather than a number: dividing noise by a real cost produces a precise-looking figure with nothing under it.

Run started 2026-08-18 19:01 UTC — receipt-attested, identical on all 30 receipts. Compute: 20.47 h (2.21 h generation over 420 calls, 18.26 h judging over 420 case rows). At the run's concurrency of 2 that is ≈10.2 h elapsed, a derived finish of ≈2026-08-19 05:15 UTC — derived, not attested: a receipt records per-call wall-clock, not a run finish, and concurrency is a launch parameter rather than receipt evidence. 30 receipts (10 skills × 3 substrates, with/without × n=5, judged by claude-haiku-4-5). Surfaces: claude-sonnet-5 → claude-cli, claude-fable-5 → claude-cli, gpt-5.6-sol → openai-cli. Metered spend $0 (vendor subscription CLIs); metered-equivalent $192.79 ($105.22 generation + $87.57 judge measurement, derived from the receipts at their frozen rates; verification level TESTED, the weakest among the 30 receipts summed). The $40 guard did not fire: it checked the $20.89 projection, not accrued spend — see the disclosure above. All receipts complete. [resume: 0 restored, 30 skipped]

v1.2 · 2026-09-01. The three cells v1.1 flagged have been re-measured, and their published lifts are named here as what they are: single-draw, judge-spread figures. Each of the three lifts on this page rests on one generation per arm, and its band is the spread of the judge re-scoring that single response. Report 007 re-ran all three cells with the generation sampled adaptively, three draws per arm minimum and more until the across-draw spread settled, and found lower lifts in every one: code-review-and-quality on claude-fable-5 +0.103 to +0.055; git-workflow-and-versioning on claude-sonnet-5 +0.116 to -0.002; writing-plans on claude-fable-5 +0.177 to +0.131. At the cell level none of the three separates from its own band.

These are not corrections, and Report 007 does not present them as such. Two of the three comparisons were refused before any verdict was formed, by the same baseline-reproduction control that refused every cell of Report 006: the no-skill arm did not reproduce the arm this page measured. On the third, writing-plans on claude-fable-5, the control passed and the comparison rendered WITHIN NOISE, which says the two measurements are the same as far as the instrument can tell. All three cells also changed skill.content_hash upstream between the two runs, so none of the pairs is a straight re-measurement of the same text.

v1.1's "no cause is asserted" holds, for the same reason and one more. Sampling noise accounts for the observations without needing a cause; whether anything else also changed is not established either way. Report 007 adds a second, independent instrument change to the same gap: a declared 300 s call timeout on this surface had never executed, because a numeric literal at the call site shadowed it from 2026-07-27 onward. Every run behind this page used 120 s. The question stays open.

What this qualifies on the page above, in that page's own words. Two sentences already published here are the ones this amendment bears on, and they are quoted rather than paraphrased. The first is the cost-per-benefit rule: “The denominator is the cell’s aggregate lift, not the lift of the single case that drove it. The cost is paid on every case in the suite, so the benefit it buys must be averaged over those same cases; pricing a whole-suite cost against one case’s lift would mix populations and read 2.5–6× cheaper than the measurement supports.” Every $ per 0.01 lift figure on this page divides by a cell's aggregate lift, and two of those denominators are the lifts under revision here. The band this page prints beside each cell is the with_skill band, not the lift band, and the two are different quantities: where this page reads 0.889 ± 0.023 the receipt's delta_uncertainty is 0.216857. Nothing above has been swapped for anything else.

The second is this page's own fairness disclosure: “Checked against upstream on 2026-08-19: 8 of 10 are still byte-identical; code-review-and-quality and writing-plans have been revised upstream.” That sentence is why the 005 to 007 comparison is a revision pair on all three cells rather than a re-measurement of the same text, and why v1.2 cannot present Report 007's lower lifts as this page's figures measured again.

Unaffected, and unchanged from v1.1. The cost-driver correlation, the substrate-disagreement result and the three-axis presentation do not rest on single-cell baselines and are not qualified here. No figure on this page has been edited; every number, count and verdict above is as published and re-derives from the same 30 receipts.

v1.1 · 2026-08-29 — Three cells rest on single-draw baselines now shown unstable. code-review-and-quality on claude-fable-5, git-workflow-and-versioning on claude-sonnet-5, and writing-plans on claude-fable-5 each report a lift measured against a baseline arm generated once. A 20-draw stability probe run for Report #006 shows two of those baselines sit in distributions wide enough that a single draw does not pin them: one spans 0.300–0.808 across ten draws, the other sits between 0.214 and 0.300 in nine draws and reaches 0.856 in one. Generation-level variation exceeds judge-level variation by 3.2× and 7.5× on those cases, and this report sampled the judge five times and the generation once.

What this does not say. It does not say these lifts are wrong, and it does not supply corrected ones. Report #006 re-measured the same three cells and got lower lifts, but those re-measurements are equally single-draw and are not replacements — on commit-message-conventional-type the published 0.282 is the modal value across ten draws while #006’s 0.852 was the roughly 1-in-10 outcome, so the fresh number is the likelier fluke of the two. No cause is asserted. Sampling noise accounts for the observations without needing one; whether anything else also changed is not established either way, and the one-generation design of this report cannot settle it. The question is recorded as open.

Unaffected. The headline findings of this report do not rest on single-cell baselines and are not qualified here: the cost-driver correlation (input-token delta tracks cost at r = +0.92 against skill length at r = +0.33), the substrate-disagreement result, and the three-axis presentation stand as published. Every figure above is left legible and unedited. Methods: Report #006, stability probe.

v1.0.1 · 2026-08-19 — Publication-chrome correction. No measured value changed: every figure, count and verdict on this page is identical to v1.0, and all 30 receipts are unchanged. At first publication the page’s <title> element and its footer both still carried the draft marker. The publication switch removed the noindex robots tag and the draft banner, but was never applied to those two places, so a page that was published in every other respect still announced itself as unpublished in the browser tab. Both are now rendered from the same switch, and the repo gate asserts the property rather than the two known places: no published report page may carry that marker anywhere, checked across every published report, so the defect cannot recur silently on a later one.

v1.0 · 2026-08-19 — First publication.