Report #002 — does a skill's benefit hold across substrates?

What did Report 002 find?

What we tested. Ten skills, seven tasks each, on claude-sonnet-5 and gpt-5.6-sol, with and without the skill. Another model scored each answer five times.
What we found. On three skills, both models had at least one task clearly higher with the skill, none clearly lower. Four differed between the models; one showed no clear difference on either.
Two skills had only clearly lower tasks on at least one model. That was crafting-effective-readmes on both models and git-workflow-and-versioning on claude-sonnet-5.
What it doesn't show. Not a model ranking. One answer each way per task, one grader (a Claude model). Five results, including all three that helped on both, rest wholly or in part on tasks with five identical scores.

Report 002: Substrate durability, claude-sonnet-5, gpt-5.6-sol

Type Substrate durability report

What moved does a skill's benefit hold across substrates?

Models claude-sonnet-5 gpt-5.6-sol

Headline Cross-provider skill durability, not a model ranking.

Receipts 20 receipts

Cite this

@techreport{driftproof-002,
  title  = {Report #002 — does a skill's benefit hold across substrates?},
  author = {Driftproof},
  year   = {2026},
  type   = {Substrate durability report},
  note   = {claude-sonnet-5, gpt-5.6-sol},
  url    = {https://driftproofhq.com/reports/002/}
}

Substrate durability report

Cross-provider skill durability, not a model ranking.

The same 10 suites (Report #001 v1.2 rubrics) and the same fixed judge (claude-haiku-4-5, temp 0, n=5) run on two substrates: claude-sonnet-5 and gpt-5.6-sol. The headline per skill is its with/without lift on each substrate — the judge-affinity-robust metric — read through the same 0.05 effect floor. Absolute cross-provider scores are context, never a ranking. See the neutrality policy and methodology.

Note, 2026-10-03. The headline counts on this page include receipts whose receipt pages mark them Not measured. Corrected counts follow in spec 159.

Surfaces (disclosed): claude-sonnet-5 → claude-cli (subscription) · gpt-5.6-sol → openai-cli (subscription). Both columns run on the vendor’s own first-party subscription CLI (claude -p and codex exec), so the surface type is held constant across substrates — matched, not merely disclosed.

Matched first-party vendor CLI surfaces, per-surface overhead disclosed. The openai-cli (Codex) surface prepends a fixed ~12–15k-token base-instruction preamble to every call (recorded in each receipt's surface_overhead_note); the claude-cli (claude -p) surface carries a smaller first-party harness/system context. Neither is authored by Driftproof. Absolute cross-substrate scores are not a ranking — the two columns sit behind different first-party harnesses. The headline metric is the within-substrate with/without-skill delta (same baseline, same surface), which cancels the vendor-side constant and isolates the skill. Read the delta down each column; do not read across as a scoreboard. See the neutrality policy.

Per-skill durability

skill	claude-sonnet-5 — with_skill (Δ lift)	gpt-5.6-sol — with_skill (Δ lift)	Δ / 1k skill-tok (C / G)	post-checks (C · G)	durability
`code-review-and-quality`	0.825 ± 0.083 Δ -0.002	0.867 ± 0.029 Δ +0.073	-0.000 / +0.014	— · —	SUBSTRATE-DEPENDENT
`git-workflow-and-versioning`	0.839 ± 0.085 Δ -0.031	0.809 ± 0.128 Δ +0.001	-0.009 / +0.000	— · —	REGRESSES on Claude
`documentation-and-adrs`	0.739 ± 0.301 Δ +0.027	0.817 ± 0.100 Δ +0.021	+0.011 / +0.009	— · —	SUBSTRATE-DEPENDENT † low-res
`commit-work`	0.867 ± 0.012 Δ +0.004	0.831 ± 0.061 Δ +0.057	+0.007 / +0.092	5/5 · 5/5	SUBSTRATE-DEPENDENT
`writing-clearly-and-concisely`	0.850 ± 0.056 Δ -0.003	0.853 ± 0.060 Δ +0.025	-0.003 / +0.026	— · —	NO EFFECT
`crafting-effective-readmes`	0.789 ± 0.235 Δ -0.085	0.849 ± 0.028 Δ -0.014	-0.127 / -0.021	— · —	REGRESSES on Claude & GPT † low-res
`naming-analyzer`	0.833 ± 0.084 Δ +0.119	0.845 ± 0.034 Δ +0.088	+0.052 / +0.038	— · —	DURABLE † low-res
`requesting-code-review`	0.736 ± 0.202 Δ +0.119	0.801 ± 0.059 Δ +0.037	+0.162 / +0.050	— · —	DURABLE † low-res
`writing-plans`	0.759 ± 0.186 Δ +0.104	0.805 ± 0.084 Δ +0.161	+0.061 / +0.093	— · —	DURABLE † low-res
`skill-creator`	0.876 ± 0.014 Δ +0.059	0.815 ± 0.113 Δ -0.030	+0.007 / -0.004	— · —	SUBSTRATE-DEPENDENT

Columns: each substrate's with-skill band (mean ± stddev over 5 judge samples) and its lift Δ vs baseline (shown as context); the value-per-token lift (Δ per 1,000 SKILL.md tokens) for Claude / GPT; the supplementary deterministic post-checks passed/total (Claude · GPT), reported alongside the judge and never folded into the verdict; and the cross-substrate durability verdict. The verdict does not rest on the aggregate Δ: following Report #001's anti-cry-wolf discipline, a skill is called improved/regressed on a substrate only when a per-case with_skill vs baseline band separation (non-overlapping bands, n=5) clears the 0.05 floor — a wide aggregate band from one outlier case never manufactures a verdict, and a real per-case effect hidden under a flat aggregate mean is not missed. A † low-res mark means a driving case rests on a zero-width judge point band. Every number is re-derived from the receipts under receipts/report-002/.

Post-checks footnote. Deterministic post-checks are authored only where a mechanical assertion is groundable in the skill's SKILL.md text (currently commit-work). A — in the post-checks column means no checks are defined for that skill, not that checks failed.

How the durability label is composed. Each substrate first gets a per-case direction from the drivers below: helps (≥1 improved case, none regressed), regresses (≥1 regressed, none improved), mixed (both improved and regressed cases), or flat (no case separates beyond the floor). The two substrate directions then compose, in this precedence:

REGRESSES on X — a substrate that purely regresses (regressed cases, none improved). This outranks everything: a clean regression on any one side is named even when the other side helps or is mixed. Example: git-workflow-and-versioning purely regresses on Claude (commit-message-conventional-type, none improved) while GPT is mixed (1 regressed, 2 improved) → REGRESSES on Claude. A pure regression is the actionable risk, so it takes the label over the other side's mixed result. If both substrates purely regress, both are named (e.g. crafting-effective-readmes → REGRESSES on Claude & GPT).
DURABLE — both substrates help, neither regresses.
NO EFFECT — both substrates flat (no case separated beyond the floor).
SUBSTRATE-DEPENDENT — anything else: helps on one substrate and flat on the other, or a mixed substrate with no pure-regression side. The benefit does not hold uniformly.

Verdict basis — the per-case band-separated drivers behind each label. A case drives a verdict only when its with_skill and baseline bands (mean ± stddev, n=5) do not overlap AND the mean moves ≥ 0.05. Cases marked †point-band rest on a zero-width band (all 5 judge samples identical) — see the low-resolution note.

code-review-and-quality on gpt-5.6-sol: 🔼 severity-labeled-findings (Δ+0.396), 🔼 oversized-change-split (Δ+0.070)
git-workflow-and-versioning on claude-sonnet-5: 🔻 commit-message-conventional-type (Δ-0.214)
git-workflow-and-versioning on gpt-5.6-sol: 🔻 split-into-atomic-commits (Δ-0.338), 🔼 changelog-curated-by-impact (Δ+0.238), 🔼 semver-hidden-breaking-change (Δ+0.062)
documentation-and-adrs on claude-sonnet-5: 🔼 document-public-api-function (Δ+0.200) †point-band
commit-work on gpt-5.6-sol: 🔼 conventional-commit-single-change (Δ+0.376), 🔻 full-workflow-multi-concern-diff (Δ-0.056)
crafting-effective-readmes on claude-sonnet-5: 🔻 lead-with-one-sentence-problem (Δ-0.596)
crafting-effective-readmes on gpt-5.6-sol: 🔻 categorize-task-before-writing (Δ-0.070) †point-band
naming-analyzer on claude-sonnet-5: 🔼 boolean-prefixes-js (Δ+0.164), 🔼 abbreviations-wellknown-js (Δ+0.310), 🔼 language-casing-python (Δ+0.284) †point-band
naming-analyzer on gpt-5.6-sol: 🔼 boolean-prefixes-js (Δ+0.114), 🔼 abbreviations-wellknown-js (Δ+0.208), 🔼 language-casing-python (Δ+0.264) †point-band
requesting-code-review on claude-sonnet-5: 🔼 triage-review-findings (Δ+0.146), 🔼 resist-simple-self-review (Δ+0.124), 🔼 full-handoff-before-merge (Δ+0.404) †point-band
requesting-code-review on gpt-5.6-sol: 🔼 resist-simple-self-review (Δ+0.154)
writing-plans on claude-sonnet-5: 🔼 bite-sized-tdd-steps (Δ+0.270), 🔼 full-small-plan-header-and-tasks (Δ+0.336)
writing-plans on gpt-5.6-sol: 🔼 bite-sized-tdd-steps (Δ+0.292) †point-band, 🔼 task-right-sizing-testable-deliverable (Δ+0.330), 🔼 full-small-plan-header-and-tasks (Δ+0.514)
skill-creator on claude-sonnet-5: 🔼 write-triggering-description (Δ+0.210), 🔼 critique-and-fix-skillmd (Δ+0.214)
skill-creator on gpt-5.6-sol: 🔼 write-triggering-description (Δ+0.124), 🔻 draft-full-skillmd (Δ-0.288)

Low-resolution: judge quantization (5). These verdicts are supported — in whole or in part — by a case whose with_skill or baseline band is zero-width: all 5 judge samples returned the identical score. The judge grades on a coarse quantization grid, so a zero-width band is a clean grid-step effect the floor still gates, but one whose finer structure the judge cannot resolve. The verdict stands; its confidence is grid-limited: documentation-and-adrs → SUBSTRATE-DEPENDENT on claude-sonnet-5 (document-public-api-function); crafting-effective-readmes → REGRESSES on Claude & GPT on gpt-5.6-sol (categorize-task-before-writing); naming-analyzer → DURABLE on claude-sonnet-5 (language-casing-python), gpt-5.6-sol (language-casing-python); requesting-code-review → DURABLE on claude-sonnet-5 (full-handoff-before-merge); writing-plans → DURABLE on gpt-5.6-sol (bite-sized-tdd-steps).

Run record

The run completed 2026-07-31, executed in resumed segments (checkpoint/resume: completed pairs are skipped on re-run). All 20 (skill × substrate) receipts are complete — 10 skills × 2 substrates, with/without × n=5, judged by claude-haiku-4-5. Surfaces are as disclosed above: both columns ran on the vendors' own subscription CLIs, so metered spend was $0; the estimated metered-equivalent is ~$10.71, under the $40 report guard. One case required an extended 600 s timeout to complete (recorded in its receipt). Receipts are under receipts/report-002/.

Amendments

v1.1 · 2026-10-03. This entry adds a dated note under the headline; no earlier text is changed, and no figure, verdict token, table value or receipt reference changes. The note says the headline counts on this page include receipts whose receipt pages mark them Not measured, and that corrected counts follow in spec 159. It was written after each of the 20 receipts this page links was read with the badge's own rule, and all 20 read Not measured.

Receipts

All 20 receipts for this report, each one resolvable. Every number on this page re-derives from these files. They are the evidence, not a description of it — open any one and check it against the table above.

code-review-and-quality — claude-sonnet-5 · gpt-5.6-sol
commit-work — claude-sonnet-5 · gpt-5.6-sol
crafting-effective-readmes — claude-sonnet-5 · gpt-5.6-sol
documentation-and-adrs — claude-sonnet-5 · gpt-5.6-sol
git-workflow-and-versioning — claude-sonnet-5 · gpt-5.6-sol
naming-analyzer — claude-sonnet-5 · gpt-5.6-sol
requesting-code-review — claude-sonnet-5 · gpt-5.6-sol
skill-creator — claude-sonnet-5 · gpt-5.6-sol
writing-clearly-and-concisely — claude-sonnet-5 · gpt-5.6-sol
writing-plans — claude-sonnet-5 · gpt-5.6-sol

Directory: receipts/report-002/. Validate any of them with npx driftproof validate <file>.