Driftproof

Report #002 — does a skill's benefit hold across substrates?

Substrate durability report

Cross-provider skill durability, not a model ranking.

The same 10 suites (Report #001 v1.2 rubrics) and the same fixed judge (claude-haiku-4-5, temp 0, n=5) run on two substrates: claude-sonnet-5 and gpt-5.6-sol. The headline per skill is its with/without lift on each substrate — the judge-affinity-robust metric — read through the same 0.05 effect floor. Absolute cross-provider scores are context, never a ranking. See the neutrality policy and methodology.

Surfaces (disclosed): claude-sonnet-5 → claude-cli (subscription) · gpt-5.6-sol → openai-cli (subscription). Both columns run on the vendor’s own first-party subscription CLI (claude -p and codex exec), so the surface type is held constant across substrates — matched, not merely disclosed.

Matched first-party vendor CLI surfaces, per-surface overhead disclosed. The openai-cli (Codex) surface prepends a fixed ~12–15k-token base-instruction preamble to every call (recorded in each receipt's surface_overhead_note); the claude-cli (claude -p) surface carries a smaller first-party harness/system context. Neither is authored by Driftproof. Absolute cross-substrate scores are not a ranking — the two columns sit behind different first-party harnesses. The headline metric is the within-substrate with/without-skill delta (same baseline, same surface), which cancels the vendor-side constant and isolates the skill. Read the delta down each column; do not read across as a scoreboard. See the neutrality policy.

Per-skill durability

skill claude-sonnet-5 — with_skill (Δ lift) gpt-5.6-sol — with_skill (Δ lift) Δ / 1k skill-tok (C / G) post-checks (C · G) durability
code-review-and-quality 0.825 ± 0.083 Δ -0.002 0.867 ± 0.029 Δ +0.073 -0.000 / +0.014 — · — SUBSTRATE-DEPENDENT
git-workflow-and-versioning 0.839 ± 0.085 Δ -0.031 0.809 ± 0.128 Δ +0.001 -0.009 / +0.000 — · — REGRESSES on Claude
documentation-and-adrs 0.739 ± 0.301 Δ +0.027 0.817 ± 0.100 Δ +0.021 +0.011 / +0.009 — · — SUBSTRATE-DEPENDENT † low-res
commit-work 0.867 ± 0.012 Δ +0.004 0.831 ± 0.061 Δ +0.057 +0.007 / +0.092 5/5 · 5/5 SUBSTRATE-DEPENDENT
writing-clearly-and-concisely 0.850 ± 0.056 Δ -0.003 0.853 ± 0.060 Δ +0.025 -0.003 / +0.026 — · — NO EFFECT
crafting-effective-readmes 0.789 ± 0.235 Δ -0.085 0.849 ± 0.028 Δ -0.014 -0.127 / -0.021 — · — REGRESSES on Claude & GPT † low-res
naming-analyzer 0.833 ± 0.084 Δ +0.119 0.845 ± 0.034 Δ +0.088 +0.052 / +0.038 — · — DURABLE † low-res
requesting-code-review 0.736 ± 0.202 Δ +0.119 0.801 ± 0.059 Δ +0.037 +0.162 / +0.050 — · — DURABLE † low-res
writing-plans 0.759 ± 0.186 Δ +0.104 0.805 ± 0.084 Δ +0.161 +0.061 / +0.093 — · — DURABLE † low-res
skill-creator 0.876 ± 0.014 Δ +0.059 0.815 ± 0.113 Δ -0.030 +0.007 / -0.004 — · — SUBSTRATE-DEPENDENT

Columns: each substrate's with-skill band (mean ± stddev over 5 judge samples) and its lift Δ vs baseline (shown as context); the value-per-token lift (Δ per 1,000 SKILL.md tokens) for Claude / GPT; the supplementary deterministic post-checks passed/total (Claude · GPT), reported alongside the judge and never folded into the verdict; and the cross-substrate durability verdict. The verdict does not rest on the aggregate Δ: following Report #001's anti-cry-wolf discipline, a skill is called improved/regressed on a substrate only when a per-case with_skill vs baseline band separation (non-overlapping bands, n=5) clears the 0.05 floor — a wide aggregate band from one outlier case never manufactures a verdict, and a real per-case effect hidden under a flat aggregate mean is not missed. A † low-res mark means a driving case rests on a zero-width judge point band. Every number is re-derived from the receipts under receipts/report-002/.

Post-checks footnote. Deterministic post-checks are authored only where a mechanical assertion is groundable in the skill's SKILL.md text (currently commit-work). A — in the post-checks column means no checks are defined for that skill, not that checks failed.

How the durability label is composed. Each substrate first gets a per-case direction from the drivers below: helps (≥1 improved case, none regressed), regresses (≥1 regressed, none improved), mixed (both improved and regressed cases), or flat (no case separates beyond the floor). The two substrate directions then compose, in this precedence:
Verdict basis — the per-case band-separated drivers behind each label. A case drives a verdict only when its with_skill and baseline bands (mean ± stddev, n=5) do not overlap AND the mean moves ≥ 0.05. Cases marked †point-band rest on a zero-width band (all 5 judge samples identical) — see the low-resolution note.
Low-resolution: judge quantization (5). These verdicts are supported — in whole or in part — by a case whose with_skill or baseline band is zero-width: all 5 judge samples returned the identical score. The judge grades on a coarse quantization grid, so a zero-width band is a clean grid-step effect the floor still gates, but one whose finer structure the judge cannot resolve. The verdict stands; its confidence is grid-limited: documentation-and-adrs → SUBSTRATE-DEPENDENT on claude-sonnet-5 (document-public-api-function); crafting-effective-readmes → REGRESSES on Claude & GPT on gpt-5.6-sol (categorize-task-before-writing); naming-analyzer → DURABLE on claude-sonnet-5 (language-casing-python), gpt-5.6-sol (language-casing-python); requesting-code-review → DURABLE on claude-sonnet-5 (full-handoff-before-merge); writing-plans → DURABLE on gpt-5.6-sol (bite-sized-tdd-steps).

Run record

The run completed 2026-07-31, executed in resumed segments (checkpoint/resume: completed pairs are skipped on re-run). All 20 (skill × substrate) receipts are complete — 10 skills × 2 substrates, with/without × n=5, judged by claude-haiku-4-5. Surfaces are as disclosed above: both columns ran on the vendors' own subscription CLIs, so metered spend was $0; the estimated metered-equivalent is ~$10.71, under the $40 report guard. One case required an extended 600 s timeout to complete (recorded in its receipt). Receipts are under receipts/report-002/.