Driftproof

Skill or sugar pill, on Fable 5.2? Twenty skills against a same-length placebo

What did Report 019 find?

  • What we tested. Twenty skills on Claude Fable 5.2, three tasks each, with the skill, with no skill and with a same-length fake.
  • What we found. Of the 20 skills, 10 beat the fake, 4 did not beat it, and for 6 we can't tell yet. Not beating the fake does not show a skill is useless.
  • How it was graded. Another model judged the fourteen. The six run with tools on were graded on what Claude Code did, by a check written before the run.
  • What it doesn't show. Beating the fake is not beating no skill. It does not show how these skills do beyond three tasks each, or how Fable 5.2 compares with other models.

Did these 20 skills beat a same-length fake?

10 beat it. 4 did not beat it (1 scored below it, 3 showed no clear difference). For 6 we can't tell yet.

On Claude Fable 5.2, three tasks per skill. Another model judged the answers of the fourteen; the six run with tools on were graded on what Claude Code did, by a check written before the run.

The skill The fake No skill Beat the fake Did not beat the fake Can't tell yet

SkillAverage scoreWith the skillWith the fakeWith no skill
Beat the fake 10
improve-codebase-architecture0.8260.7140.790
tdd1.0000.9170.981
hyperframes-cli0.9630.6920.738
handoff0.8750.8280.833
triage0.9890.7350.711
prototype0.8970.4180.448
grilling0.8720.8080.817
hyperframes-animation0.8720.5440.640
design-mobile-apps0.9170.8470.820
hyperframes-core0.9170.6610.658
Did not beat the fake 4 1 scored below it, 3 showed no clear difference
reddit-automation0.8060.9390.958
grill-with-docs tools on1.0001.0000.833
agent-browser tools on1.0001.0001.000
hyperframes tools on1.0001.0001.000
Can't tell yet 6
setup-matt-pocock-skills0.9170.9110.907
frontend-design0.9450.9580.980
media-use0.9810.9010.942
find-skills tools on0.9260.9440.926
grill-me tools on0.9861.0001.000
hyperframes-registry tools on0.8520.8520.889

Each mark is an average over all of a skill's tasks. The verdict is decided task by task, so a mark can point the other way.

Each skill

Open a skill for its tasks, its scores and its receipt.

Beat the fake

improve-codebase-architecture mattpocock/skills Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: too few answers to tell.

Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO

improve-codebase-architecture mattpocock/skills

Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below

  • candidate (separated-up): with the skill 0.728571 plus or minus 0.090633; with the same-length placebo 0.4875 plus or minus 0.047871.
  • adr (underpowered): with the skill 1 plus or minus 0; with the same-length placebo 0.9375 plus or minus 0.075.
  • report (underpowered): with the skill 0.75 plus or minus 0; with the same-length placebo 0.716667 plus or minus 0.028868.

Against no skill: Too few answers to tell: 0.826 with the skill and 0.790 without, but the scores varied too much to call it.

Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case candidate: the two arms' spreads sum to 0.141, at or above the 0.05 floor).

  • candidate (underpowered): with the skill 0.728571 plus or minus 0.090633; without it 0.75 plus or minus 0.05.
  • adr (underpowered): with the skill 1 plus or minus 0; without it 0.92 plus or minus 0.115109.
  • report (underpowered): with the skill 0.75 plus or minus 0; without it 0.7 plus or minus 0.079057.

The placebo against no skill, shown here and not counted: PLACEBO_BELOW_BASELINE

Receipt c44af9665c99f35e673fe83c158205785c7743bc13cddba2289ab17170341c83

tdd mattpocock/skills Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: too few answers to tell.

Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO

tdd mattpocock/skills

Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below

  • test (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
  • coupling (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
  • agreement (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.75 plus or minus 0.

Against no skill: Too few answers to tell: 1.000 with the skill and 0.981 without, but the scores varied too much to call it.

Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case agreement: the two arms' spreads sum to 0.102, at or above the 0.05 floor).

  • test (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
  • coupling (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
  • agreement (underpowered): with the skill 1 plus or minus 0; without it 0.941667 plus or minus 0.102062.

The placebo against no skill, shown here and not counted: PLACEBO_BELOW_BASELINE

Receipt 383f4db201b9416ec333a321d25f324dcb07fd22b024a2189fb80591bffad092

hyperframes-cli heygen-com/hyperframes Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: clearly helped.

Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO

hyperframes-cli heygen-com/hyperframes

Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below

  • loop (separated-up): with the skill 0.89 plus or minus 0.108397; with the same-length placebo 0.625 plus or minus 0.103682.
  • timeline (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.75 plus or minus 0.
  • render (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.7 plus or minus 0.114018.

Against no skill: The skill clearly helped: 0.963 with it and 0.738 without, on 3 test tasks.

0.963 with the skill and 0.738 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.

  • loop (separated-up): with the skill 0.89 plus or minus 0.108397; without it 0.733333 plus or minus 0.028868.
  • timeline (separated-up): with the skill 1 plus or minus 0; without it 0.75 plus or minus 0.
  • render (separated-up): with the skill 1 plus or minus 0; without it 0.73 plus or minus 0.057009.

The placebo against no skill, shown here and not counted: UNDERPOWERED

Receipt f68117a83f4e32cc4844353587b188f1b4cb61e6862f9a103cf0f5b25330158a

handoff mattpocock/skills Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: clearly helped.

Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO

handoff mattpocock/skills

Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below

  • resume (not-separated): with the skill 0.75 plus or minus 0; with the same-length placebo 0.75 plus or minus 0.
  • privacy (underpowered): with the skill 0.875 plus or minus 0.144338; with the same-length placebo 0.983333 plus or minus 0.028868.
  • destination (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.75 plus or minus 0.

Against no skill: The skill clearly helped: 0.875 with it and 0.833 without, on 3 test tasks.

0.875 with the skill and 0.833 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.

  • resume (not-separated): with the skill 0.75 plus or minus 0; without it 0.75 plus or minus 0.
  • privacy (underpowered): with the skill 0.875 plus or minus 0.144338; without it 1 plus or minus 0.
  • destination (separated-up): with the skill 1 plus or minus 0; without it 0.75 plus or minus 0.

The placebo against no skill, shown here and not counted: UNDERPOWERED

Receipt d2a21cc18ae1423f3cfb6f033cc56ec25b457421004829e3f5c67c8d6eeb33df

triage mattpocock/skills Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: clearly helped.

Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO

triage mattpocock/skills

Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below

  • conflict (underpowered): with the skill 1 plus or minus 0; with the same-length placebo 0.885714 plus or minus 0.128174.
  • evidence (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
  • ready (separated-up): with the skill 0.966667 plus or minus 0.028868; with the same-length placebo 0.32 plus or minus 0.067082.

Against no skill: The skill clearly helped: 0.989 with it and 0.711 without, on 3 test tasks.

0.989 with the skill and 0.711 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.

  • conflict (separated-up): with the skill 1 plus or minus 0; without it 0.733333 plus or minus 0.028868.
  • evidence (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
  • ready (separated-up): with the skill 0.966667 plus or minus 0.028868; without it 0.4 plus or minus 0.108012.

The placebo against no skill, shown here and not counted: UNDERPOWERED

Receipt 6ee5ccb7d05d791a9e90f5ece0fcd8bffd092a0f837ea76dc33287dc1cd16d01

prototype mattpocock/skills Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: clearly helped.

Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO

prototype mattpocock/skills

Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below

  • logic (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.525 plus or minus 0.05.
  • ui (separated-up): with the skill 0.93 plus or minus 0.057009; with the same-length placebo 0.33 plus or minus 0.175357.
  • unclear (separated-up): with the skill 0.7625 plus or minus 0.131498; with the same-length placebo 0.4 plus or minus 0.094868.

Against no skill: The skill clearly helped: 0.897 with it and 0.448 without, on 3 test tasks.

0.897 with the skill and 0.448 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.

  • logic (separated-up): with the skill 1 plus or minus 0; without it 0.5 plus or minus 0.
  • ui (separated-up): with the skill 0.93 plus or minus 0.057009; without it 0.492857 plus or minus 0.171825.
  • unclear (separated-up): with the skill 0.7625 plus or minus 0.131498; without it 0.35 plus or minus 0.

The placebo against no skill, shown here and not counted: UNDERPOWERED

Receipt e8e18fbaec9a403354b12b1761968687e8213bbf57350def1a6164eccf6226aa

grilling mattpocock/skills Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: clearly helped.

Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO

grilling mattpocock/skills

Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below

  • frontier (separated-up): with the skill 0.983333 plus or minus 0.028868; with the same-length placebo 0.75 plus or minus 0.
  • settled (underpowered): with the skill 0.9 plus or minus 0.106066; with the same-length placebo 0.925 plus or minus 0.093541.
  • finish (underpowered): with the skill 0.733333 plus or minus 0.028868; with the same-length placebo 0.75 plus or minus 0.

Against no skill: The skill clearly helped: 0.872 with it and 0.817 without, on 3 test tasks.

0.872 with the skill and 0.817 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.

  • frontier (separated-up): with the skill 0.983333 plus or minus 0.028868; without it 0.75 plus or minus 0.
  • settled (underpowered): with the skill 0.9 plus or minus 0.106066; without it 1 plus or minus 0.
  • finish (underpowered): with the skill 0.733333 plus or minus 0.028868; without it 0.7 plus or minus 0.05.

The placebo against no skill, shown here and not counted: UNDERPOWERED

Receipt 3400122dda9e97243aa7457a193e9514fe2a9bd8ada484ad00898d9e34f14b1a

hyperframes-animation heygen-com/hyperframes Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: clearly helped.

Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO

hyperframes-animation heygen-com/hyperframes

Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below

  • timeline (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
  • runtime (separated-up): with the skill 0.616667 plus or minus 0.028868; with the same-length placebo 0.033333 plus or minus 0.028868.
  • template (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.6 plus or minus 0.11547.

Against no skill: The skill clearly helped: 0.872 with it and 0.640 without, on 3 test tasks.

0.872 with the skill and 0.640 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.

  • timeline (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
  • runtime (separated-up): with the skill 0.616667 plus or minus 0.028868; without it 0.295 plus or minus 0.146154.
  • template (separated-up): with the skill 1 plus or minus 0; without it 0.625 plus or minus 0.144338.

The placebo against no skill, shown here and not counted: PLACEBO_BELOW_BASELINE

Receipt cb7fee140bf69efb5bd73549a8d01f49394e817be92b7ffe84885d6e72297cfe

design-mobile-apps designed-by-ai/skills Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: clearly helped.

Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO

design-mobile-apps designed-by-ai/skills

Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below

  • response (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.95 plus or minus 0.
  • shots (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.825 plus or minus 0.095743.
  • ids (underpowered): with the skill 0.75 plus or minus 0; with the same-length placebo 0.766667 plus or minus 0.028868.

Against no skill: The skill clearly helped: 0.917 with it and 0.820 without, on 3 test tasks.

0.917 with the skill and 0.820 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.

  • response (separated-up): with the skill 1 plus or minus 0; without it 0.891667 plus or minus 0.073598.
  • shots (separated-up): with the skill 1 plus or minus 0; without it 0.78 plus or minus 0.097468.
  • ids (underpowered): with the skill 0.75 plus or minus 0; without it 0.7875 plus or minus 0.047871.

The placebo against no skill, shown here and not counted: UNDERPOWERED

Receipt ee86c450b8a7c2d805f600b88fdc4e1f8ef83b6bcf9798c5a86ffbd09ce29b97

hyperframes-core heygen-com/hyperframes Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: clearly helped.

Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO

hyperframes-core heygen-com/hyperframes

Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below

  • root (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
  • transform (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.75 plus or minus 0.
  • exit (separated-up): with the skill 0.75 plus or minus 0; with the same-length placebo 0.233333 plus or minus 0.028868.

Against no skill: The skill clearly helped: 0.917 with it and 0.658 without, on 3 test tasks.

0.917 with the skill and 0.658 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.

  • root (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
  • transform (separated-up): with the skill 1 plus or minus 0; without it 0.725 plus or minus 0.05.
  • exit (separated-up): with the skill 0.75 plus or minus 0; without it 0.25 plus or minus 0.141421.

The placebo against no skill, shown here and not counted: UNDERPOWERED

Receipt e3ee1e3bd056d398f04c8610b9067a0fd5af11dafdd9cb04cba1d740da547193

Did not beat the fake

reddit-automation flowkit-labs/skills Did not beat the fake Scored below the fake on at least one task. Compared with no skill: clearly hurt.

Label: Delete candidate Against the same-length placebo: below the fake BELOW_PLACEBO

reddit-automation flowkit-labs/skills

Against the same-length placebo: case rule: at least one case scored below the same-length placebo

  • rank (separated-down): with the skill 0.78 plus or minus 0.067082; with the same-length placebo 1 plus or minus 0.
  • reply (underpowered): with the skill 0.7625 plus or minus 0.062915; with the same-length placebo 0.816667 plus or minus 0.143759.
  • rules (underpowered): with the skill 0.875 plus or minus 0.144338; with the same-length placebo 1 plus or minus 0.

Against no skill: The skill clearly hurt: 0.806 with it and 0.958 without, on 3 test tasks.

0.806 with the skill and 0.958 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.

  • rank (separated-down): with the skill 0.78 plus or minus 0.067082; without it 0.983333 plus or minus 0.028868.
  • reply (underpowered): with the skill 0.7625 plus or minus 0.062915; without it 0.89 plus or minus 0.151658.
  • rules (underpowered): with the skill 0.875 plus or minus 0.144338; without it 1 plus or minus 0.

The placebo against no skill, shown here and not counted: UNDERPOWERED

Receipt ecc1cb963a360697df5142934d4390b53ae7bcc091303a8d0c6ad56555757ebe

grill-with-docs mattpocock/skills tools on Did not beat the fake No task differed from the fake by enough to count. Compared with no skill: clearly helped.

Label: Delete candidate Against the same-length placebo: no difference NO_DIFFERENCE

grill-with-docs mattpocock/skills

Graded on the end state: each draw's score is its task's check, written before any call. The judge's grade of the reply is secondary and enters no reading.

Against the same-length placebo: case rule: no case differed from the same-length placebo at this effect floor

  • gd-booking (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
  • gd-cart-order (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
  • gd-account-user (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.

Against no skill: The skill clearly helped: 1.000 with it and 0.833 without, on 3 test tasks.

1.000 with the skill and 0.833 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.

  • gd-booking (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
  • gd-cart-order (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
  • gd-account-user (separated-up): with the skill 1 plus or minus 0; without it 0.5 plus or minus 0.

Draws with a Skill call to grilling: 9 of 9 with the skill, 6 of 9 with no skill, 9 of 9 with the placebo.

Recorded cost USD 31.97 over 27 generations, judging included.

Receipt bb52fbbc6320aa3fc879ccaad9dcdc64f1680eb4ff4777fd4804f678efe9d2eb

agent-browser vercel-labs/agent-browser tools on Did not beat the fake No task differed from the fake by enough to count. Compared with no skill: no clear difference.

Label: Delete candidate Against the same-length placebo: no difference NO_DIFFERENCE

agent-browser vercel-labs/agent-browser

Graded on the end state: each draw's score is its task's check, written before any call. The judge's grade of the reply is secondary and enters no reading.

Against the same-length placebo: case rule: no case differed from the same-length placebo at this effect floor

  • ab-ticket (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
  • ab-status (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
  • ab-note (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.

Against no skill: No clear difference: 1.000 with the skill and 1.000 without, with enough answers to have seen a gap of 0.05.

there were enough answers to have seen a gap of 0.05, and none of the 3 test tasks showed one.

  • ab-ticket (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
  • ab-status (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
  • ab-note (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.

Recorded cost USD 79.80 over 27 generations, judging included.

Receipt 0b714814822b461a0fcc7bdb0f7e6ea45e195cc20977c10357596cfde49f3aa6

hyperframes heygen-com/hyperframes tools on Did not beat the fake No task differed from the fake by enough to count. Compared with no skill: no clear difference.

Label: Delete candidate Against the same-length placebo: no difference NO_DIFFERENCE

hyperframes heygen-com/hyperframes

Graded on the end state: each draw's score is its task's check, written before any call. The judge's grade of the reply is secondary and enters no reading.

Against the same-length placebo: case rule: no case differed from the same-length placebo at this effect floor

  • hf-title (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
  • hf-logo (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
  • hf-caption (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.

Against no skill: No clear difference: 1.000 with the skill and 1.000 without, with enough answers to have seen a gap of 0.05.

there were enough answers to have seen a gap of 0.05, and none of the 3 test tasks showed one.

  • hf-title (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
  • hf-logo (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
  • hf-caption (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.

Recorded cost USD 41.57 over 27 generations, judging included.

Receipt 2c4cb0f1a92b8c3e0c17849062ac709d62717436355b3bf25c07c81118c12f2e

Can't tell yet

setup-matt-pocock-skills mattpocock/skills Can't tell yet Not enough answers to tell it from the fake. Compared with no skill: too few answers to tell.

Label: Cannot tell Against the same-length placebo: not enough draws UNDERPOWERED

setup-matt-pocock-skills mattpocock/skills

Against the same-length placebo: case rule: Not enough draws to conclude at this effect floor

  • local (underpowered): with the skill 1 plus or minus 0; with the same-length placebo 0.983333 plus or minus 0.028868.
  • labels (not-separated): with the skill 0.75 plus or minus 0; with the same-length placebo 0.75 plus or minus 0.
  • monorepo (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.

Against no skill: Too few answers to tell: 0.917 with the skill and 0.907 without, but the scores varied too much to call it.

Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case monorepo: the two arms' spreads sum to 0.095, at or above the 0.05 floor).

  • local (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
  • labels (underpowered): with the skill 0.75 plus or minus 0; without it 0.783333 plus or minus 0.028868.
  • monorepo (underpowered): with the skill 1 plus or minus 0; without it 0.9375 plus or minus 0.094648.

The placebo against no skill, shown here and not counted: UNDERPOWERED

Receipt 2b2e2930934e39b9ce3013b2b963adb1f886f24a14e0f3678eb52bb1eb219706

frontend-design anthropics/skills Can't tell yet Not enough answers to tell it from the fake. Compared with no skill: too few answers to tell.

Label: Cannot tell Against the same-length placebo: not enough draws UNDERPOWERED

frontend-design anthropics/skills

Against the same-length placebo: case rule: Not enough draws to conclude at this effect floor

  • identity (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
  • brief (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
  • content (underpowered): with the skill 0.835714 plus or minus 0.114434; with the same-length placebo 0.875 plus or minus 0.112916.

Against no skill: Too few answers to tell: 0.945 with the skill and 0.980 without, but the scores varied too much to call it.

Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case content: the two arms' spreads sum to 0.169, at or above the 0.05 floor).

  • identity (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
  • brief (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
  • content (underpowered): with the skill 0.835714 plus or minus 0.114434; without it 0.94 plus or minus 0.054772.

The placebo against no skill, shown here and not counted: UNDERPOWERED

Receipt 03d93027ab744923fb42141d774614a030ab85728ab26e7b650a3b2c86a0c281

media-use heygen-com/hyperframes Can't tell yet Not enough answers to tell it from the fake. Compared with no skill: too few answers to tell.

Label: Cannot tell Against the same-length placebo: not enough draws UNDERPOWERED

media-use heygen-com/hyperframes

Against the same-length placebo: case rule: Not enough draws to conclude at this effect floor

  • reuse (underpowered): with the skill 0.941667 plus or minus 0.102062; with the same-length placebo 0.7375 plus or minus 0.103078.
  • logo (underpowered): with the skill 1 plus or minus 0; with the same-length placebo 0.966667 plus or minus 0.08165.
  • grade (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.

Against no skill: Too few answers to tell: 0.981 with the skill and 0.942 without, but the scores varied too much to call it.

Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case reuse: the two arms' spreads sum to 0.131, at or above the 0.05 floor).

  • reuse (underpowered): with the skill 0.941667 plus or minus 0.102062; without it 0.883333 plus or minus 0.028868.
  • logo (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
  • grade (underpowered): with the skill 1 plus or minus 0; without it 0.941667 plus or minus 0.102062.

The placebo against no skill, shown here and not counted: PLACEBO_BELOW_BASELINE

Receipt 593f226fd71b4f3ada583bd7dbb755a6a5e2ebd44c81274b763aa5ce5485df22

find-skills vercel-labs/skills tools on Can't tell yet Not enough answers to tell it from the fake. Compared with no skill: too few answers to tell.

Label: Cannot tell Against the same-length placebo: not enough draws UNDERPOWERED

find-skills vercel-labs/skills

Graded on the end state: each draw's score is its task's check, written before any call. The judge's grade of the reply is secondary and enters no reading.

Against the same-length placebo: case rule: Not enough draws to conclude at this effect floor

  • fs-changelog (underpowered): with the skill 0.777778 plus or minus 0.172133; with the same-length placebo 0.833333 plus or minus 0.19245.
  • fs-pdf (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
  • fs-none (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.

Against no skill: Too few answers to tell: 0.926 with the skill and 0.926 without, but the scores varied too much to call it.

Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case fs-changelog: the two arms' spreads sum to 0.344, at or above the 0.05 floor).

  • fs-changelog (underpowered): with the skill 0.777778 plus or minus 0.172133; without it 0.777778 plus or minus 0.172133.
  • fs-pdf (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
  • fs-none (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.

Recorded cost USD 116.14 over 34 generations, judging included.

Receipt 745ed6e2dc62f204c0ccafd7b57e545027926fdf0486e61d633e3552a67854bc

grill-me mattpocock/skills tools on Can't tell yet Not enough answers to tell it from the fake. Compared with no skill: too few answers to tell.

Label: Cannot tell Against the same-length placebo: not enough draws UNDERPOWERED

grill-me mattpocock/skills

Graded on the end state: each draw's score is its task's check, written before any call. The judge's grade of the reply is secondary and enters no reading.

Against the same-length placebo: case rule: Not enough draws to conclude at this effect floor

  • gm-report-app (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
  • gm-search-cache (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
  • gm-split-users (underpowered): with the skill 0.958333 plus or minus 0.102062; with the same-length placebo 1 plus or minus 0.

Against no skill: Too few answers to tell: 0.986 with the skill and 1.000 without, but the scores varied too much to call it.

Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case gm-split-users: the two arms' spreads sum to 0.102, at or above the 0.05 floor).

  • gm-report-app (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
  • gm-search-cache (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
  • gm-split-users (underpowered): with the skill 0.958333 plus or minus 0.102062; without it 1 plus or minus 0.

Draws with a Skill call to grilling: 12 of 12 with the skill, 9 of 9 with no skill, 9 of 9 with the placebo.

Recorded cost USD 27.86 over 30 generations, judging included.

Receipt b18ad267e07543695d8d1ca295187f051991fd065ba80d2e7707de5d4717042d

hyperframes-registry heygen-com/hyperframes tools on Can't tell yet Not enough answers to tell it from the fake. Compared with no skill: too few answers to tell.

Label: Cannot tell Against the same-length placebo: not enough draws UNDERPOWERED

hyperframes-registry heygen-com/hyperframes

Graded on the end state: each draw's score is its task's check, written before any call. The judge's grade of the reply is secondary and enters no reading.

Against the same-length placebo: case rule: Not enough draws to conclude at this effect floor

  • hr-screen (underpowered): with the skill 0.888889 plus or minus 0.272166; with the same-length placebo 1 plus or minus 0.
  • hr-grain (underpowered): with the skill 1 plus or minus 0; with the same-length placebo 0.888889 plus or minus 0.272166.
  • hr-chart (not-separated): with the skill 0.666667 plus or minus 0; with the same-length placebo 0.666667 plus or minus 0.

Against no skill: Too few answers to tell: 0.852 with the skill and 0.889 without, but the scores varied too much to call it.

Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case hr-screen: the two arms' spreads sum to 0.272, at or above the 0.05 floor).

  • hr-screen (underpowered): with the skill 0.888889 plus or minus 0.272166; without it 1 plus or minus 0.
  • hr-grain (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
  • hr-chart (not-separated): with the skill 0.666667 plus or minus 0; without it 0.666667 plus or minus 0.

Recorded cost USD 282.40 over 33 generations, judging included.

Receipt 78dfbfddb6ffd4d2ad88cf2fa09b12bb175cbb3d285ec09172ffb080417e4708

Keep in mind

  • The six skills marked tools on ran with tools on and were graded on what Claude Code did, not on its reply. That is not the same test as the other fourteen.
  • Beating the fake is not the same as beating no skill. Each skill's details show both.
  • Can't tell yet means there was not enough evidence to call it. It does not mean the skill had no effect.
  • This is one model, Claude Fable 5.2, and three tasks per skill. It says nothing about other models or other tasks.

Report 020: Fable 5.2 against Fable 5.1 on three skills

Three skills, each run on both models with the skill loaded. On Claude Fable 5.2, compared with Claude Fable 5.1:

  • code-review-and-quality: scored higher on Fable 5.2.
  • git-workflow-and-versioning: no clear difference.
  • documentation-and-adrs: scored lower on Fable 5.2.
The table, how it was run and its limits

Its own section, outside the twenty-skill headline. Three of Addy Osmani's skills, one case each, run fresh on Claude Fable 5.2 and Claude Fable 5.1, with the skill and without it, judged by claude-haiku-5-5 at five samples. Each verdict reads the two with-skill arms against each other, in Report 011's labels; upward means Fable 5.2 scored above Fable 5.1.

No effort setting was passed: Fable 5.2 ran at its default, medium, and Fable 5.1 at its default, high. The table reads the two models as users get them, not at the same effort.

Report 020's judge differs from Report 011's, so Report 020's figures are not set beside Report 011's as if comparable.

SkillFable 5.1, with the skillFable 5.2, with the skillVerdictWithout the skill, context only (Fable 5.1; Fable 5.2)
code-review-and-quality0.63 ± 0.13, 5 draws0.83 ± 0.03, 3 drawsseparation detected, upward0.21 ± 0.00, 3 draws; 0.62 ± 0.07, 5 draws
git-workflow-and-versioning0.84 ± 0.01, 3 draws0.82 ± 0.02, 3 drawsno separation detected0.84 ± 0.01, 3 draws; 0.30 ± 0.00, 3 draws
documentation-and-adrs0.75 ± 0.03, 3 draws0.50 ± 0.08, 4 drawsseparation detected, downward0.58 ± 0.14, 7 draws; 0.39 ± 0.05, 3 draws

The means without the skill are context; no verdict is read from them.

Limits

One case per skill. Nothing in the report describes how any of the three skills behaves on any other task, or how Fable 5.2 behaves in general.

Few draws. Each arm draws between 3 and 10 generations, and each case's band is the spread across its draws. With one case there is no spread across cases.

The judge is not Report 011's. The figures are comparable within Report 020 only.

The two models ran at their own default effort, medium and high.

It does not say which model is better at these skills in general, why a verdict reads as it does, or that no separation means the two models score alike. It does not compare any figure with Report 011 or with Reports 017, 018 or 019.

How we tested

The headline as fixed before the run, the full method, the judge, the price, the run record and the limits.

Placebo comparison report. Twenty skills on Fable 5.2, each against no skill and against a same-length text from a fixed corpus, judged by Haiku 5.5: fourteen graded on the reply, and six run with tools on and graded on what Claude Code left.

Report 019. Recorded on 11 Nov 2026. Answering model Claude Fable 5.2; judge Claude Haiku 5.5.

The headline, as fixed before the run

Of the 14 skills graded on the reply, 10 beat the fake on Fable 5.2.

Of the 6 skills graded on the end state with tools on, 0 beat the fake on Fable 5.2.

10 of 20 skills beat the fake on Fable 5.2: by the case rule, each scored above the same-length placebo on at least one of three tasks and below it on none. A skill that did not beat the fake is not shown to do no better than it.

Of the 20 skills counted: 10 beat the fake, 1 below the fake, 3 no difference, 0 inconclusive, 6 not enough draws, 0 not measured.

Of the 20: ABOVE_PLACEBO 10, BELOW_PLACEBO 1, NO_DIFFERENCE 3, INCONCLUSIVE 0, UNDERPOWERED 6, NOT_MEASURED 0. Beat the fake means ABOVE_PLACEBO only.

In plain labels: of the fourteen, Keep 10, Delete candidate 1, Cannot tell 3; of the six, Keep 0, Delete candidate 3, Cannot tell 3.

Six of these skills hand work to other skills or tools, so they ran with tools on, after the other fourteen, and each was scored on what Claude Code ended up doing.

The note fixed before the text-only run counted these six differently; Maverick's ruling of 11 Nov 2026 counts them by their tools-on reading.

The six are scored on the end state; the fourteen on the reply. The two are not the same test.

Where the fake scored below no skill, beating it does not mean beating no skill; each row shows both readings.

The judge is claude-haiku-5-5 at five samples, a different model from the target. For the six, its grade of the reply is a secondary score and enters no reading.

No effort setting was passed, so claude-fable-5-2 ran at its catalog default, medium.

The six, with tools on

What was installed, the same in every version of a skill: the called skills (grilling, domain-modeling and hyperframes-core), the command-line tools (git, npm and agent-browser), Chrome 155.0.8059.79, and local copies of the skills registry, an npm registry, the HyperFrames registry and a web app, each served on the draw's own loopback. There was no live internet: each draw's network was none or its own loopback. The router or pointer text was the only difference between the versions: the skill's text, no text, or a same-length placebo. The grade was the end state Claude Code left: files, installed skills, commands, app state and Skill calls, read by a check written before any call. The judge's grade of the reply was secondary and entered no reading.

Every version scored at or near the top on most tasks, so there was little room for a difference: 45 of 54 task means, one per task and version, were 0.9 or above.

Every version of the two pointer skills called grilling, because it was installed in all of them. Draws with a grilling call: grill-me 12 of 12 with the skill, 9 of 9 with no skill, 9 of 9 with the placebo; grill-with-docs 9 of 9 with the skill, 6 of 9 with no skill, 9 of 9 with the placebo.

Tools-on runs cost more than planned for 3 of the six. The plan projected USD 2.19 a generation and USD 0.01 a judged draw, from Report 016's 75th percentile at the Fable 5.2 estimate. Recorded cost a generation, judging included: find-skills USD 3.42, grill-me USD 0.93, grill-with-docs USD 1.18, agent-browser USD 2.96, hyperframes USD 1.54, hyperframes-registry USD 8.56. hyperframes-registry's first attempt stopped at its ceiling of USD 148.96 after USD 160.57 and was run again under a raised cap, with a ceiling of USD 662.08; both attempts' costs are counted.

The same six as text-only receipts

Published and linked as context. They enter no count and no headline line.

find-skills vercel-labs/skills (text-only, not counted) Did not beat the fake Scored below the fake on at least one task. Compared with no skill: no verdict, delegated skill not tested.

Label: Delete candidate Against the same-length placebo: below the fake BELOW_PLACEBO

find-skills vercel-labs/skills

These text tasks test the router's written response, not the delegated skill. Report 015 Amendment 1.

Graded on the reply, with tools off. The tools-on reading above is the one counted.

Against the same-length placebo: case rule: at least one case scored below the same-length placebo

  • search (separated-down): with the skill 0.733333 plus or minus 0.028868; with the same-length placebo 1 plus or minus 0.
  • missing (underpowered): with the skill 0.97 plus or minus 0.067082; with the same-length placebo 1 plus or minus 0.
  • scope (underpowered): with the skill 0.983333 plus or minus 0.028868; with the same-length placebo 1 plus or minus 0.

Against no skill: No verdict: delegated skill not tested.

The placebo against no skill, shown here and not counted: NO_DIFFERENCE

Receipt 3c132baf43df44ed15adbed42569afd59b606bf997dc80528ef8080c94758bb5

grill-me mattpocock/skills (text-only, not counted) Can't tell yet Not enough answers to tell it from the fake. Compared with no skill: too few answers to tell.

Label: Cannot tell Against the same-length placebo: not enough draws UNDERPOWERED

grill-me mattpocock/skills

About grill-me and grill-with-docs: each skill's whole text is an instruction to call another skill. These text tasks forbid tools, so each reading reflects that one-line hand-off, not the skill it calls. Both stay in the pre-registered counts.

Graded on the reply, with tools off. The tools-on reading above is the one counted.

Against the same-length placebo: case rule: Not enough draws to conclude at this effect floor

  • plan (underpowered): with the skill 0.683333 plus or minus 0.196638; with the same-length placebo 0.633333 plus or minus 0.10328.
  • constraint (underpowered): with the skill 0.6 plus or minus 0.106066; with the same-length placebo 0.483333 plus or minus 0.160208.
  • followup (underpowered): with the skill 0.9375 plus or minus 0.047871; with the same-length placebo 0.883333 plus or minus 0.028868.

Against no skill: Too few answers to tell: 0.740 with the skill and 0.717 without, but the scores varied too much to call it.

Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case plan: the two arms' spreads sum to 0.343, at or above the 0.05 floor).

  • plan (underpowered): with the skill 0.683333 plus or minus 0.196638; without it 0.685714 plus or minus 0.146385.
  • constraint (underpowered): with the skill 0.6 plus or minus 0.106066; without it 0.516667 plus or minus 0.028868.
  • followup (underpowered): with the skill 0.9375 plus or minus 0.047871; without it 0.95 plus or minus 0.05.

The placebo against no skill, shown here and not counted: UNDERPOWERED

Receipt f76798ed009f4f5811dd935ffd1c215ad6675f6e3336c3a79ce1c0cd2e08aa4d

grill-with-docs mattpocock/skills (text-only, not counted) Can't tell yet Not enough answers to tell it from the fake. Compared with no skill: too few answers to tell.

Label: Cannot tell Against the same-length placebo: not enough draws UNDERPOWERED

grill-with-docs mattpocock/skills

About grill-me and grill-with-docs: each skill's whole text is an instruction to call another skill. These text tasks forbid tools, so each reading reflects that one-line hand-off, not the skill it calls. Both stay in the pre-registered counts.

Graded on the reply, with tools off. The tools-on reading above is the one counted.

Against the same-length placebo: case rule: Not enough draws to conclude at this effect floor

  • decision (underpowered): with the skill 0.95 plus or minus 0.057735; with the same-length placebo 0.93 plus or minus 0.097468.
  • adr (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
  • revision (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.

Against no skill: Too few answers to tell: 0.983 with the skill and 0.976 without, but the scores varied too much to call it.

Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case decision: the two arms' spreads sum to 0.148, at or above the 0.05 floor).

  • decision (underpowered): with the skill 0.95 plus or minus 0.057735; without it 0.928571 plus or minus 0.090633.
  • adr (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
  • revision (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.

The placebo against no skill, shown here and not counted: UNDERPOWERED

Receipt adc67d2d503f41826a483cd7dbce047dfead9a30ab09a34acdb36e7bdad30b2f

agent-browser vercel-labs/agent-browser (text-only, not counted) Did not beat the fake Scored below the fake on at least one task. Compared with no skill: no verdict, delegated skill not tested.

Label: Delete candidate Against the same-length placebo: below the fake BELOW_PLACEBO

agent-browser vercel-labs/agent-browser

These text tasks test the router's written response, not the delegated skill. Report 015 Amendment 1.

Graded on the reply, with tools off. The tools-on reading above is the one counted.

Against the same-length placebo: case rule: at least one case scored below the same-length placebo

  • web (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
  • desktop (separated-down): with the skill 0.733333 plus or minus 0.028868; with the same-length placebo 0.92 plus or minus 0.09083.
  • dashboard (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.65 plus or minus 0.1.

Against no skill: No verdict: delegated skill not tested.

The placebo against no skill, shown here and not counted: UNDERPOWERED

Receipt 86257def619b1db544acda4b26b243fdb6ff419a7648936c22c4736bc54c8197

hyperframes heygen-com/hyperframes (text-only, not counted) Can't tell yet Not enough answers to tell it from the fake. Compared with no skill: no verdict, delegated skill not tested.

Label: Cannot tell Against the same-length placebo: not enough draws UNDERPOWERED

hyperframes heygen-com/hyperframes

These text tasks test the router's written response, not the delegated skill. Report 015 Amendment 1.

Graded on the reply, with tools off. The tools-on reading above is the one counted.

Against the same-length placebo: case rule: Not enough draws to conclude at this effect floor

  • existing (underpowered): with the skill 0.75 plus or minus 0; with the same-length placebo 0.8 plus or minus 0.093541.
  • usage (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
  • port (underpowered): with the skill 0.875 plus or minus 0.144338; with the same-length placebo 1 plus or minus 0.

Against no skill: No verdict: delegated skill not tested.

The placebo against no skill, shown here and not counted: UNDERPOWERED

Receipt d82a78f5d986f1822f98d56ad2f55370cdc51b3aad135d0f0bda156c6b319f38

hyperframes-registry heygen-com/hyperframes (text-only, not counted) Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: no verdict, delegated skill not tested.

Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO

hyperframes-registry heygen-com/hyperframes

These text tasks test the router's written response, not the delegated skill. Report 015 Amendment 1.

Graded on the reply, with tools off. The tools-on reading above is the one counted.

Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below

  • block (separated-up): with the skill 0.85 plus or minus 0.05; with the same-length placebo 0.483333 plus or minus 0.028868.
  • component (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.8375 plus or minus 0.118145.
  • search (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.516667 plus or minus 0.028868.

Against no skill: No verdict: delegated skill not tested.

The placebo against no skill, shown here and not counted: UNDERPOWERED

Receipt 64045ab20a9d60331a66f90956f3f7f8b1361fb4881bf101c7d272b8e448291e

Method

These sections are the pre-registered notes, as fixed before each run: plan (a)'s for the text-only twenty, then spec 786's for the six with tools on.

What each receipt shows

Each of the 20 frozen attempts runs one skill of Report 017's frozen roster, the same pinned skill bytes and the same committed three-task suite, on claude-fable-5-2, graded by claude-haiku-5-5, with three arms: the skill text, no skill, and a text of the skill's length in bytes from the revision 1 fixed corpus (spec 440). The runner is Driftproof 0.28.0 and the CLI is Claude Code 2.1.311, on the claude-cli subscription surface.

The report shows two readings for each receipt, each in lib/verdict.js's own words:

  • the verdict: receiptVerdict, the skill against no skill, in the receipt reader's words;
  • the placebo reading: placeboLine(placeboReading(receipt)), the skill against the same-length text.

The verdict shown is the Report 019 receipt's own. The verdict reads only the skill and no-skill arms, so the placebo arm's replies move it in no way. No figure other than these readings and their counts enters the headline.

The judge

The judge is claude-haiku-5-5, the judge the judge policy fixes for the reports Driftproof publishes: "That judge is fixed at claude-haiku-5-5 and changed only rarely" (/judge-policy/). Reports 017 and 018 were graded by the same judge. No judge policy or registry entry moves for this plan.

The judge is a different model from the target. lib/receipt.js records no same-model limitation for this pair, and the plan pins that as judge_relationship. The report says nothing about judge accuracy.

The effort setting

No effort setting is passed, so the catalog default for claude-fable-5-2, medium, applies. That is the default a user of the CLI gets. The report says so beside the readings, and does not present them as the model at its highest effort.

The price

claude-fable-5-2 has no published price in this repository. Every cost figure of this report prices its calls at an estimate, USD 10 input and USD 50 output per million tokens, the most expensive known tier, and says so where it is shown. Metered spend on the subscription surface is USD 0.

Report 019 stands alone

The page and every Report 019 claim show no Report 017 or Report 018 reading, verdict, count or figure. One methods sentence names the design and what changed: Report 017's same-length placebo design, the same twenty skills, suites and placebo corpus, with claude-fable-5-2 answering, graded by the same fixed judge, on runner 0.28.0. The judge policy holds that "results scored by different graders are not compared directly"; here the grader is the same, and the target and runner are not, so the report compares nothing across reports: it does not subtract one reading from another, does not count skills that moved, and does not say which model a skill helps more.

Which receipt is read

The report reads exactly one sealed receipt per attempt id. It is the receipt the campaign's manifest chain records for that attempt, read through the campaign ledger. The first completed receipt is the only one: an attempt with a sealed receipt is never run again, and a second receipt for the same attempt id refuses the whole chain. No receipt is chosen after its results are seen.

An attempt without a receipt may be run again in a later invocation of the same frozen plan, at most three runs in all, its earlier costs counted against the USD 375 cap. Every failure stays in the ledger. A failure carries no reading, so running it again chooses nothing.

The campaign has one root directory. It keeps the plan and this note as first started and an append-only log with one line per start. A second start with no prior chain is refused, and a resume must continue the log's latest manifest. The report reads only the chain that ends at the log's latest manifest, and refuses if the root holds any other manifest under this plan. A cost correction may be added after a run and is never rewritten: each manifest records the corrections in force when it started.

Terms

"Beat the fake" means exactly one thing: the placebo reading is ABOVE_PLACEBO. In the reader's words, "case rule: at least one case scored above the same-length placebo, and none below". The case rule reads each of the three tasks by its draw bands, so this says no more than that, on at least one of three tasks, the skill's band sat above the placebo's at the effect floor, and on none below.

No other reading beats the fake. BELOW_PLACEBO, NO_DIFFERENCE, INCONCLUSIVE, UNDERPOWERED and NOT_MEASURED do not. None of them shows that a skill does no better than the fake.

A missing receipt reads NOT_MEASURED: a failed, cancelled or budget-stopped attempt is counted as not read, never as a result. A stub receipt also reads NOT_MEASURED.

Secondary reading: the placebo against no skill

Beside each skill, and never in the headline, the report shows a second reading: the same case rule over the placebo's rows against the no-skill rows of the same receipt (spec 613's tally.cjs secondaryReading). Its tokens are PLACEBO_ABOVE_BASELINE, PLACEBO_BELOW_BASELINE, NO_DIFFERENCE, INCONCLUSIVE, UNDERPOWERED and NOT_MEASURED. It is read only where the placebo reading is read, and refused wherever that is refused. It is not counted, not subtracted from the placebo reading, and not combined with it.

With tools on: what runs

Six skills from Report 019's roster: grill-me, grill-with-docs, find-skills, hyperframes-registry, agent-browser and hyperframes. Each runs on claude-fable-5-2 through Claude Code 2.1.311 in the repository lane's tools-on mode, with Report 019's pinned skill bytes. No effort setting is passed, so the catalog default for claude-fable-5-2, medium, applies.

Each attempt has three tasks and three arms: the skill as system text, no system text, and a same-length placebo from Report 019's corpus. Every arm of an attempt gets the same installed world: the called skills, the pinned command-line tools and hashed local fixtures on the draw's own loopback. Nothing reaches the internet. Each arm draws at least 3 and at most 6 times per task, by the runner's sampling rule.

With tools on: how a draw is scored

The primary score of a draw is its task's check, frozen in the plan by sha256. The check reads what Claude Code ended up doing: files, tool calls, Skill calls and the fixtures' state. For the two pointer skills it also reads the form of the reply: numbered questions, each with a recommended answer. No check credits an action from the reply's words.

The secondary score is the fixed judge's grade of the reply and its captured files: claude-haiku-5-5 at five samples, as the judge policy fixes it ("That judge is fixed at claude-haiku-5-5 and changed only rarely"). The secondary score never enters a verdict.

A draw whose end state cannot be read is not measured. It is never scored 0.

With tools on: the readings

Each skill's verdict and placebo reading are lib/verdict.js's own, read over the primary scores in its own words. Only ABOVE_PLACEBO beats the fake. An attempt without a verified receipt reads "not measured".

Limits

The placebo text is not inert: on a task that supplies the facts its answer needs, a same-length text about landscape and village life can pull an answer off them, so the fake can score below no skill. The secondary reading shows where it did.

The projection in plan.json reprices Reports 017 and 018's recorded tokens at the claude-fable-5-2 estimate for generation and at Haiku 5.5 rates for judging. Fable 5.2 may answer at another length and draw another number of times, so the projection is a planning assumption, not a bound. The cap and each attempt's ceiling are the bounds.

It does not say why a skill helps, that a skill and the fake are the same, or that the fake carries no information about a task. It says nothing about a skill that was not run with the placebo arm. It does not combine the 20 readings into one score. It does not say how Fable 5.2, Sonnet 5.5 and Haiku 5.5 compare: no reading, verdict or count of Reports 015, 017 or 018 appears on the page or in any Report 019 claim.

  • Three tasks per skill, and at most six draws per arm.
  • The six are scored on the end state; the fourteen on the reply. The two are not the same test.
  • The tools-on attempts run from the source branch at the commit their campaign records, not from a release. The receipt's runner version is the one config.js records at that commit.
  • The cost is priced at an estimate for claude-fable-5-2, USD 10 and USD 50 per million tokens.

The placebo corpus was chosen without reference to any task, so a placebo text can still bear on a task by chance.

Unkeyed seals and retained outputs permit checking internal consistency; they do not authenticate an origin.

Run record

Spend, read from the records. Report 019 text-only USD 140.06: USD 135.55 in the chain read here and USD 4.52 in stopped chains recorded in specs/783-fable-5-2-scope/chains.json. With tools on USD 790.31: USD 740.32 in the campaign manifest, the stopped attempt included, and USD 49.99 in smoke runs recorded in specs/786-tools-on-routers/freeze.json. Report 020 USD 10.03. Recorded costs are API-equivalent estimates, not a subscription invoice, with claude-fable-5-2 priced at an estimate of USD 10 input and USD 50 output per million tokens. Metered spend on the subscription surface was USD 0.

The text-only twenty: plan SHA-256 e557c3b4a5849ccfff353352a55c28742becdb5cb51be68fe9b06d0ebb7171ce, pre-registered note SHA-256 76e1913b0ae3bec0f91ef72b72685922af2906e977320d35a595b77161676a94, run at 17cdf896f7cf7fab190f663d247051321f0c6b3f, runner 0.28.0, Claude Code 2.1.311.

  1. receipts/report-019/campaign-001/manifest.json, SHA-256 d8556d35959da7e3e884571f0377d45b8e2cae7eba06a11c24ff33fbd0ff6b11
  2. receipts/report-019/campaign-002/manifest.json, SHA-256 3ae34883ab2858618db65e0325cff9f87f044c60381c81446827eeb998b043cf
  3. receipts/report-019/campaign-003/manifest.json, SHA-256 5c4d439a3a1dd26354cd9c490d300f88adc862916e984f0db230b42487604bda

The six with tools on: plan SHA-256 f7968ae3753384d767b70873bc3f646857abc8afdc8c077483645d66f36808e0, pre-registered note SHA-256 25f0f51911413fc75ceca2020db64a5afc0cbaec4fef7f7e42dbdfc7f9af6f74, manifest receipts/report-019-tools-on/manifest.json, SHA-256 d06a8f03b59752815559e074f9ebd51f6e9424e7784750f2eb4a8bf6e39c0659. Each attempt ran from the source branch at the commit its start records, runner 0.28.0, Claude Code 2.1.311:

  1. commit 46beb63c22adae7ff5b735649695e21fd27be2d2, plan SHA-256 f77e92a516922e374457431cfc46f2240288e8094998b0b7f24fbee83aa5b3a1
  2. commit 7be8f9bdf0a1990c2858b3e39b9322110102289e, plan SHA-256 a311cf94cf1c7b3dce6a2ea9d420315d4267270086a001e5f48311f58d198145
  3. commit 7be8f9bdf0a1990c2858b3e39b9322110102289e, plan SHA-256 a311cf94cf1c7b3dce6a2ea9d420315d4267270086a001e5f48311f58d198145
  4. commit 2a6f9dc5d611fc61727d2faee907c64be5b83e15, plan SHA-256 c618cf2f86ed03769dcea43f15bf99e1856e673c1ddb62f91995f3f3e16bfeaa
  5. commit 2a6f9dc5d611fc61727d2faee907c64be5b83e15, plan SHA-256 c618cf2f86ed03769dcea43f15bf99e1856e673c1ddb62f91995f3f3e16bfeaa
  6. commit 4229a9db7969601e38c5b32e253e8af132a170bf, plan SHA-256 f7968ae3753384d767b70873bc3f646857abc8afdc8c077483645d66f36808e0

Report 020: plan SHA-256 40d30de73cafa2dbac9ae0303401f43d8ff50c4a55f2369ba4283f528d85e445.

  1. receipts/report-020/campaign-001/manifest.json, SHA-256 38a0cb2b90ffeac399cd5e9deb358473ee0816af217dadc74ec56cde6b3157d0, commit 8550f3b03c52052e2c5a681d570a20107b58480e

The retained answers of handoff's text-only receipt show the eval account's home directory as ~/ in 5 answers. The reader restores the original from the account name and checks it against the hash the receipt seals.

Read and built by:

  • scripts/prepare-report-019.js, SHA-256 b493a739bc93d18c71fe6def89661b28b1c7ab7658927460be4049257d9c5e51
  • scripts/report-019-reader.js, SHA-256 7764c59353736dca5f74d4d8265d2b1fb1e126e17fb0a95185e176b980a91d16
  • scripts/answer-redaction.js, SHA-256 d17de4ffeb1e120a7ad6df063d2016fd93f8d937095545abe9c5dfb9fa4f086c
  • specs/783-fable-5-2-scope/placebo/run-campaign.cjs, SHA-256 22dcd101c6319ec07fe5091d950e63bbe7fd6e749e2b6e728d21f530fb2c3a2c
  • specs/783-fable-5-2-scope/boundary/run-campaign.cjs, SHA-256 0fba7b1e86a5f8dc002448d40141fb26c66765d0bac463cb8f0a00380ce3d5ff
  • specs/613-placebo-skills-report/run-campaign.cjs, SHA-256 db070608736eb9401fb8acf6a15cf800deeefbd706b18a92383242ef6609af6a
  • specs/613-placebo-skills-report/tally.cjs, SHA-256 a7d334796f32591a9488baa2286e3ba9462305ff6a92b8b2f0c7695a96a415dd
  • specs/408-skill-measurements/ledger.cjs, SHA-256 263a3d3423feea70aaf90074a58bcaf48d4ddb8b0eea262ec634450b17b63aae
  • specs/786-tools-on-routers/world.cjs, SHA-256 e94047b18e0919ab60fde49503bca828af893bf33d0e36522ba7191775e9d9aa
  • lib/verdict.js, SHA-256 c09edd5e1a8c62c43fa0e0d1f1a268d5d6076e474846e2cf9247a1f0e35b5b6e
  • lib/plain.js, SHA-256 46f471b1487460b6101680f0b4c1191e358611020f326898e37f0e420011dbb9
  • lib/receipt.js, SHA-256 979be61ec82acca2e20c9764844188f2d1f025a34d7fbfd0365d7fed8fdef7a8
  • scripts/offer.js, SHA-256 dcc5adde7ce49b2f853708c35a7bfe671a38f38a908a01b2ecc12234c76adddb

Receipts

Skills measured: anthropics/skills: frontend-design; designed-by-ai/skills: design-mobile-apps; flowkit-labs/skills: reddit-automation; heygen-com/hyperframes: hyperframes, hyperframes-animation, hyperframes-cli, hyperframes-core, hyperframes-registry and media-use; mattpocock/skills: grill-me, grill-with-docs, grilling, handoff, improve-codebase-architecture, prototype, setup-matt-pocock-skills, tdd and triage; vercel-labs/agent-browser: agent-browser; vercel-labs/skills: find-skills; source not recorded: agent-browser, code-review-and-quality, documentation-and-adrs, find-skills, git-workflow-and-versioning, grill-me, grill-with-docs, hyperframes and hyperframes-registry.

Every Driftproof result in one file: JSONL or CSV, and what a row is and is not.

Want a skill measured, or think a result is wrong? It's free: how to ask.

Words with a dotted line link to the glossary, where each is explained in plain words.