How Driftproof tests a skill, step by step

1. One task, three ways

Driftproof gives an AI model a task Driftproof wrote to test the skill, and it gives the same task three ways. Here is a real task, for the handoff skill:

Do not use tools, access the network, write files, run commands or install anything. Answer only in text using the supplied facts. Write a handoff from these facts: parser fix drafted, tests not run, decision recorded in docs/parser-plan.md, next session verifies invalid input. Keep the existing plan as the detailed source.

2. With the skill, a placebo, or no skill text

One way with the skill, one with a placebo, one with no skill text added.

A placebo is a text exactly as long as the skill, built from sentences chosen without looking at any task.

The skill is 940 bytes long and its placebo 940, about a byte per letter. Each text's first sentence (the skill's header lines are left out):

Read the whole skill, at the version tested. The whole placebo text is not shown on this page.

3. A separate judge scores each answer

The judge is a second AI model, here Claude Haiku 5.5, that scores each answer against a guide written for the task.

A score runs from 0, no point of the guide met, to 1, every point met.

The answers came from Claude Sonnet 5.5. The judge gets the task, one answer and the guide, and is not told which way the answer was made; an answer can still show which way it was made.

This task's guide gives 0.25 for each point met:

1. State current status including unrun tests
2. Reference the plan instead of duplicating it
3. Name the next verification task
4. Suggest a relevant next-session skill without claiming it ran

The first answer each way, shown as plain text:

4. Repeat, then read the case rule

Each way answers the task at least 3 times and at most 10, stopping once the scores settle, and each answer is called a draw.

On this task there were 3 draws with the skill, 3 with the placebo and 3 with no skill text, and the judge scored each draw 5 times.

The spread measures how much the scores of repeated draws differ from one another.

The case rule reads each case, meaning each task, on its own, and the effect floor, 0.05, is the smallest gap it counts.

Clearly helped means: on at least one test task, the average with the skill was higher, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.

Clearly hurt means: on at least one test task, the average with the skill was lower, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.

If any task is clearly lower, the reading is Clearly hurt, whatever the others show.

Above the placebo means: case rule: at least one case scored above the same-length placebo, and none below.

Below the placebo means: the same case rule, measured against the placebo instead of no skill text: a task clearly lower.

Scope: 3 tasks, written by Driftproof; answers in text only, with tools forbidden; Claude Sonnet 5.5 answering, Claude Haiku 5.5 judging; 1 Nov 2026.

5. What a result does not tell you

• It covers only these 3 tasks, Claude Sonnet 5.5 answering and Claude Haiku 5.5 judging, on 1 Nov 2026.
• Too few answers to tell, which the case rule writes as Not enough draws to conclude at this effect floor, means there were too few answers, or they varied too much, to call it either way. It does not mean the skill does nothing.
• Beating the placebo does not mean beating no skill text.
• A placebo text can still bear on a task by chance.
• The judge is an AI reading a guide, not a person.
• A reading under the rule is not proof.

Each reading comes from a receipt, a sealed record whose fingerprint stops matching if any figure in it changes.

Check it: the receipt's page, its file, the kept answers, the tasks and guides, and Report 017.