Which section did the work?

What we did

A skill is a file of instructions that people add to an AI assistant. A long file raises a fair question: which lines do anything? Driftproof's compare command runs two versions of a file on the same tasks and says whether the scores clearly moved between them.

This page is a worked example. We took three skills Driftproof had already measured in Report 018, removed from each the longest stretch of text under one heading by the rule below, and ran compare: the whole file against the file without that text.

Claude Haiku 5.5 answered every task and Claude Haiku 5.5 graded each answer 5 times, the same arrangement as Report 018. For each skill the whole file ran first, then the reduced file. The run was made by Driftproof on 2026-11-08.

How the three were chosen, and what was removed

The skills are the first three, in Report 018's own order, whose result there read “Clearly helped”, that are not routers (a router's whole text hands the task to another skill), and whose file has at least two sections below its title. No skill was chosen by hand.

In each file we measured the text under every heading, from its heading line to the next heading line of any level, so a subsection is measured on its own and does not count toward the heading above it. A heading line starts with one to six # characters and a space. The title at the top and anything inside a code block do not count. The heading with the most bytes of its own text was removed, with that text. Every other byte stayed as it was.

This is not how the --list-sections option measures. It runs each section to the next heading of the same or a higher level, so a heading's subsections count toward its size. On two of the three files above it shows a section called Process as larger than the one removed, because Process holds it. The cards used the any-level measure. Each heading removed has no subsections, so naming it to the --without-section option on the pinned file gives the same reduced file.

What this can and cannot say

Read these before the readings below. They apply to every card.

• The three tasks were drafted by Driftproof from the skill's own description; they are not the author's tests.
• One section was removed, so sections that matter only together are not seen.
• A shorter file is also a shorter file, so length moved too.
• This model, this judge (the model grading itself), this day.
• The two runs were sequential, so time-of-day conditions were not identical.
• No clear difference is not proof the section does nothing.
• Across these nine tasks, chance alone can produce a difference like the ones below; this page does not say whether one came from the removal, from the time between the two runs or from chance.

The three skills

Each card gives the plain reading first, then the exact lines compare prints for the two folders, worked out again from the sealed receipts in today's wording. A reading is about these tasks on this day; it is not advice about the skill.

Three of three pairs completed.

- report: B (improve-codebase-architecture, the folder given second) scored clearly lower than A (improve-codebase-architecture, the folder given first)
- adr: too few answers
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case adr: the two arms' spreads sum to 0.332, at or above the 0.05 floor).
- candidate: too few answers
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case candidate: the two arms' spreads sum to 0.058, at or above the 0.05 floor).
B (improve-codebase-architecture, the folder given second) scored lower than A (improve-codebase-architecture, the folder given first) on the report task, and no other task separated. This is a separation detected under the rule (the two averages differ by more than their usual spread), not proof of its cause.

- render: too few answers
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case render: the two arms' spreads sum to 0.098, at or above the 0.05 floor).
- timeline: too few answers
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case timeline: the two arms' spreads sum to 0.050, at or above the 0.05 floor).
- loop: too few answers
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case loop: the two arms' spreads sum to 0.141, at or above the 0.05 floor).
No separation was detected between A (hyperframes-cli, the folder given first) and B (hyperframes-cli, the folder given second) on any task at this sample size; this is not evidence that the two files behave alike.

- monorepo: B (setup-matt-pocock-skills, the folder given second) scored clearly lower than A (setup-matt-pocock-skills, the folder given first)
- labels: too few answers
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case labels: the two arms' spreads sum to 0.121, at or above the 0.05 floor).
- local: too few answers
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case local: the two arms' spreads sum to 0.186, at or above the 0.05 floor).
B (setup-matt-pocock-skills, the folder given second) scored lower than A (setup-matt-pocock-skills, the folder given first) on the monorepo task, and no other task separated. This is a separation detected under the rule (the two averages differ by more than their usual spread), not proof of its cause.

Do this to your own file

The same two commands work on any Markdown rules file, such as a CLAUDE.md or AGENTS.md, with a tasks file beside it. The first lists the sections and their sizes without spending anything. The second removes one named section, runs both versions on your tasks, and writes the two receipts and the reading. Both count a heading's subsections as part of its section, unlike the rule the cards used.

npx driftproof compare FILE.md --list-sections
npx driftproof compare FILE.md --without-section "Heading" --suite tasks.json

For a skill folder, make a copy, remove the section from the copy's SKILL.md, keep the same evals/evals.json in both, and compare the two folders. To redo one of the three above, fetch the pinned file from the repository at the commit shown, check its hash, apply the removal rule, and check the reduced file's hash against the card.

npx driftproof compare skill-a skill-b --models claude-haiku-5-5 --samples 5

The removal rule in one line: find the heading below the title with the most bytes of its own text, from its heading line to the next heading line of any level, and remove those lines, leaving every other byte as it was.

Where the evidence is

The six receipts, the three readings and the manifest are in the public repository. Each receipt has its own page with the two arms, what was verified and how to check the file yourself.

The receipts, readings and manifest · Every receipt page · Every task we gave a skill · Report 018