Report #001 — do skill verdicts hold across a model release?

node scripts/fetch-skills.js
node scripts/run-report-001.js --concurrency 5
node scripts/build-report-001.js

v1.2 · 2026-07-30 — In-text grounding policy adopted and applied across all 10 suites. Headline: v1.1 read 3 regressed / 3 improved / 3 mixed / 1 within noise; v1.2 reads 2 regressed / 3 improved / 4 mixed / 1 within noise. This supersedes v1.1.

• Policy adopted (codified, applied unilaterally). Every gradable rubric criterion must trace to text present in the skill's SKILL.md at the pinned SHA. Criteria that import domain knowledge the skill does not state are not gradable, regardless of how standard that knowledge is. Claims may summarize; rubrics may not extrapolate. The grading standard does not depend on the audited party's tolerance. Full sweep of all 10 suites (70 cases): reports/rubric-sweep.md.
• Criteria amended (2 of 10 suites; the other 8 were grounded and unchanged). git-workflow-and-versioning/semver-clean-bump required "the highest-precedence bump governs when changes are combined / reset PATCH to 0" — a rule the SKILL.md never states (it defines only MAJOR/MINOR/PATCH). That criterion was removed; the case now grades only the stated MAJOR/MINOR/PATCH definitions. commit-work (all 7 cases) required a specific Conventional Commits type token (e.g. chore(deps) vs feat) and, in one case, a subject <=72-chars / no-trailing-period / imperative-mood rule — none of which its SKILL.md states (it names Conventional Commits and shows the type(scope): summary shape but does not enumerate the type vocabulary). Type grading is now shape-only; the length/period/mood rule was removed. The logical-split, patch-staging, and secret/debug checks the SKILL.md does state remain fully graded.
• Re-run. Only the two amended suites were re-run (both models, n=5, claude-cli, transcripts retained); the amendment changes only those two suite_hashes. The grounding reference now recorded on every case is excluded from suite_hash, so annotating the other eight suites did not disturb their receipts.
• Verdict changes (stated plainly, not softened). The amended case git-workflow-and-versioning/semver-clean-bump flips regression → within noise (v1.1 Δ−0.054 on separated bands; v1.2 Δ−0.004). The v1.1 "regression" was driven by the imported precedence/reset criterion; grading only the MAJOR/MINOR/PATCH definitions the SKILL.md actually states, the new model shows no drift on this case. But re-running a suite re-rolls all seven of its cases (one receipt covers the whole suite), and the fresh n=5 measurement surfaced, on two unamended, well-grounded cases, a real regression and a real improvement: commit-message-conventional-type now regresses (Δ−0.256 — the new model wrote feat: instead of fix: for a bug fix; all five judge samples scored 0.60 with that same reason, transcript-verified) and semver-hidden-breaking-change now improves (Δ+0.062). Net, the skill moves REGRESSED(1) → MIXED. commit-work is unchanged at IMPROVED(1): its driver (two-sentence-describability-test, Δ+0.168) held, and its amended cases (specific type token → shape, and the removed ≤72-char/no-period/imperative-mood rule) remain within noise. The other eight skills were not re-run and keep their v1.1 labels.

v1.1 · 2026-07-28 — Pre-launch QA corrections. Net headline change: v1.0 read 4 regressed / 3 improved / 3 mixed / 0 within noise; v1.1 reads 3 regressed / 3 improved / 3 mixed / 1 within noise. Two causes, below.

• (1) Minimum-effect floor of 0.05. A case is now called regressed/improved only when its two with_skill bands do not overlap AND the mean moved ≥ 0.05 (one judge-quantization step) — band separation alone is no longer sufficient, because the judge quantizes to a coarse grid and a confident grade often collapses to a zero-width point band. This reclassified 3 hair-trigger cases (|Δ| 0.014–0.024) to “within noise (below effect floor)”: code-review-and-quality/security-axis-sql-injection (which flips that whole skill REGRESSED → WITHIN NOISE), naming-analyzer/misleading-name-mutation-js (MIXED 2r/3i → 2r/2i), and requesting-code-review/identify-base-head-shas (IMPROVED 2 → 1).
• (2) Two artifact-contaminated receipts re-run. A transcript audit found new-model bands containing a judge parse-fallback score of exactly 0.0 (an unparseable judge reply defaults to 0, per lib/judge.js). Both __claude-sonnet-5 receipts were re-run on the same claude-cli surface, n=5 (old-model receipts, which were clean, were kept as-is): writing-clearly-and-concisely and documentation-and-adrs.
• What the re-run showed. active-voice-logging-passage’s v1.0 “regression” was entirely the artifact — the clean re-run scores 0.862 (vs old 0.852), i.e. within noise. match-existing-adr-convention reconfirmed as a genuine new-model failure (~0.08 again on an independent run). Because a receipt covers all 7 of a skill’s cases, re-running also refreshed the other cases honestly: comment-intent-not-implementation’s v1.0 regression did not reproduce (now within noise), while two different, real regressions surfaced — documentation-and-adrs/surface-conflicting-adr-conventions (Δ−0.268) and writing-clearly-and-concisely/concrete-language-incident-summary (Δ−0.152). Both skills keep their v1.0 labels (MIXED and REGRESSED-1 respectively); only the driving cases changed.
• Full per-case transcript evidence and REAL-vs-artifact classifications are in reports/report-001-audit.md.

v1.3 · 2026-09-14. This entry corrects how the report’s band wording is read. It follows the entries recorded before it, no earlier text in the report is changed, and no figure, verdict token, table value or receipt reference changes.

• The headline. This page says “9 of 10 skills moved beyond noise on this model pair.” Each of those nine skills has at least one case whose two bands do not overlap and whose mean moved by at least the 0.05 effect floor: a separation detected under the rule, which is not proof that the skill’s behaviour moved. No skill’s aggregate with_skill band separated. As amended:

9 of 10 skills showed a separation detected under the rule on this model pair.

• How to read this page. A case is separated under the rule when its two bands do not overlap and its mean moved by at least the 0.05 effect floor; bands that do not overlap by a smaller move are not separated. The four rows marked within noise (below floor) are of that kind, and what the headline rule calls bands that separate by less than the floor are bands that do not overlap. Wherever the page reads a result that was not separated as settled, as within noise, never as drift, no drift on this case, hair-trigger, a regression that did not reproduce, or a cause confirmed because a later run was not separated, as the entry of 2026-07-30 does of the imported criterion and the entry of 2026-07-28 of the artifact, read it as no separation detected at the sample size used, which is not evidence of equivalence, not evidence that nothing changed, and not proof of the cause. Wherever it reads a separation as settled, as a real regression and a real improvement, real regressions surfaced, reconfirmed as a genuine new-model failure, or a driver that held, read it as a separation detected under the rule, which is not proof that the model’s behaviour moved. The Methodology’s no meaningful change is a hypothetical that explains the floor, and the footer’s Verdicts age because the substrate moves names a cause that no separation here proves. Labels, tallies and headings, the summary card’s What moved among them, are read the same way and are left as published.
• The bands. Each band on this page is a mean plus or minus one sample standard deviation, of two kinds: a case’s band, over its five judge samples, and a skill’s with_skill and baseline bands, over its seven per-case means, which is suite dispersion. Each band is a descriptive spread, not a confidence interval, with no coverage probability.
• Where the amended headline appears. The site’s report index and feed quote the amended headline; this page’s summary card, description and social card image keep the headline as published. Filed under the wording rules of the repository's spec 031, amendment A-031-20.