Skill or sugar pill, on Fable 5.2? Twenty skills against a same-length placebo
What did Report 019 find?
- What we tested. Twenty skills on Claude Fable 5.2, three tasks each, with the skill, with no skill and with a same-length fake.
- What we found. Of the 20 skills, 10 beat the fake, 4 did not beat it, and for 6 we can't tell yet. Not beating the fake does not show a skill is useless.
- How it was graded. Another model judged the fourteen. The six run with tools on were graded on what Claude Code did, by a check written before the run.
- What it doesn't show. Beating the fake is not beating no skill. It does not show how these skills do beyond three tasks each, or how Fable 5.2 compares with other models.
Did these 20 skills beat a same-length fake?
10 beat it. 4 did not beat it (1 scored below it, 3 showed no clear difference). For 6 we can't tell yet.
On Claude Fable 5.2, three tasks per skill. Another model judged the answers of the fourteen; the six run with tools on were graded on what Claude Code did, by a check written before the run.
The skill The fake No skill Beat the fake Did not beat the fake Can't tell yet
| Skill | Average score | With the skill | With the fake | With no skill |
|---|---|---|---|---|
| Beat the fake 10 | ||||
| improve-codebase-architecture | 0.826 | 0.714 | 0.790 | |
| tdd | 1.000 | 0.917 | 0.981 | |
| hyperframes-cli | 0.963 | 0.692 | 0.738 | |
| handoff | 0.875 | 0.828 | 0.833 | |
| triage | 0.989 | 0.735 | 0.711 | |
| prototype | 0.897 | 0.418 | 0.448 | |
| grilling | 0.872 | 0.808 | 0.817 | |
| hyperframes-animation | 0.872 | 0.544 | 0.640 | |
| design-mobile-apps | 0.917 | 0.847 | 0.820 | |
| hyperframes-core | 0.917 | 0.661 | 0.658 | |
| Did not beat the fake 4 1 scored below it, 3 showed no clear difference | ||||
| reddit-automation | 0.806 | 0.939 | 0.958 | |
| grill-with-docs tools on | 1.000 | 1.000 | 0.833 | |
| agent-browser tools on | 1.000 | 1.000 | 1.000 | |
| hyperframes tools on | 1.000 | 1.000 | 1.000 | |
| Can't tell yet 6 | ||||
| setup-matt-pocock-skills | 0.917 | 0.911 | 0.907 | |
| frontend-design | 0.945 | 0.958 | 0.980 | |
| media-use | 0.981 | 0.901 | 0.942 | |
| find-skills tools on | 0.926 | 0.944 | 0.926 | |
| grill-me tools on | 0.986 | 1.000 | 1.000 | |
| hyperframes-registry tools on | 0.852 | 0.852 | 0.889 | |
Each mark is an average over all of a skill's tasks. The verdict is decided task by task, so a mark can point the other way.
Each skill
Open a skill for its tasks, its scores and its receipt.
Beat the fake
improve-codebase-architecture mattpocock/skills Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: too few answers to tell.
Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO
improve-codebase-architecture mattpocock/skills
Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below
- candidate (separated-up): with the skill 0.728571 plus or minus 0.090633; with the same-length placebo 0.4875 plus or minus 0.047871.
- adr (underpowered): with the skill 1 plus or minus 0; with the same-length placebo 0.9375 plus or minus 0.075.
- report (underpowered): with the skill 0.75 plus or minus 0; with the same-length placebo 0.716667 plus or minus 0.028868.
Against no skill: Too few answers to tell: 0.826 with the skill and 0.790 without, but the scores varied too much to call it.
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case candidate: the two arms' spreads sum to 0.141, at or above the 0.05 floor).
- candidate (underpowered): with the skill 0.728571 plus or minus 0.090633; without it 0.75 plus or minus 0.05.
- adr (underpowered): with the skill 1 plus or minus 0; without it 0.92 plus or minus 0.115109.
- report (underpowered): with the skill 0.75 plus or minus 0; without it 0.7 plus or minus 0.079057.
The placebo against no skill, shown here and not counted: PLACEBO_BELOW_BASELINE
Receipt c44af9665c99f35e673fe83c158205785c7743bc13cddba2289ab17170341c83
tdd mattpocock/skills Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: too few answers to tell.
Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO
tdd mattpocock/skills
Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below
- test (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
- coupling (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
- agreement (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.75 plus or minus 0.
Against no skill: Too few answers to tell: 1.000 with the skill and 0.981 without, but the scores varied too much to call it.
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case agreement: the two arms' spreads sum to 0.102, at or above the 0.05 floor).
- test (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
- coupling (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
- agreement (underpowered): with the skill 1 plus or minus 0; without it 0.941667 plus or minus 0.102062.
The placebo against no skill, shown here and not counted: PLACEBO_BELOW_BASELINE
Receipt 383f4db201b9416ec333a321d25f324dcb07fd22b024a2189fb80591bffad092
hyperframes-cli heygen-com/hyperframes Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: clearly helped.
Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO
hyperframes-cli heygen-com/hyperframes
Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below
- loop (separated-up): with the skill 0.89 plus or minus 0.108397; with the same-length placebo 0.625 plus or minus 0.103682.
- timeline (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.75 plus or minus 0.
- render (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.7 plus or minus 0.114018.
Against no skill: The skill clearly helped: 0.963 with it and 0.738 without, on 3 test tasks.
0.963 with the skill and 0.738 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.
- loop (separated-up): with the skill 0.89 plus or minus 0.108397; without it 0.733333 plus or minus 0.028868.
- timeline (separated-up): with the skill 1 plus or minus 0; without it 0.75 plus or minus 0.
- render (separated-up): with the skill 1 plus or minus 0; without it 0.73 plus or minus 0.057009.
The placebo against no skill, shown here and not counted: UNDERPOWERED
Receipt f68117a83f4e32cc4844353587b188f1b4cb61e6862f9a103cf0f5b25330158a
handoff mattpocock/skills Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: clearly helped.
Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO
handoff mattpocock/skills
Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below
- resume (not-separated): with the skill 0.75 plus or minus 0; with the same-length placebo 0.75 plus or minus 0.
- privacy (underpowered): with the skill 0.875 plus or minus 0.144338; with the same-length placebo 0.983333 plus or minus 0.028868.
- destination (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.75 plus or minus 0.
Against no skill: The skill clearly helped: 0.875 with it and 0.833 without, on 3 test tasks.
0.875 with the skill and 0.833 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.
- resume (not-separated): with the skill 0.75 plus or minus 0; without it 0.75 plus or minus 0.
- privacy (underpowered): with the skill 0.875 plus or minus 0.144338; without it 1 plus or minus 0.
- destination (separated-up): with the skill 1 plus or minus 0; without it 0.75 plus or minus 0.
The placebo against no skill, shown here and not counted: UNDERPOWERED
Receipt d2a21cc18ae1423f3cfb6f033cc56ec25b457421004829e3f5c67c8d6eeb33df
triage mattpocock/skills Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: clearly helped.
Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO
triage mattpocock/skills
Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below
- conflict (underpowered): with the skill 1 plus or minus 0; with the same-length placebo 0.885714 plus or minus 0.128174.
- evidence (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
- ready (separated-up): with the skill 0.966667 plus or minus 0.028868; with the same-length placebo 0.32 plus or minus 0.067082.
Against no skill: The skill clearly helped: 0.989 with it and 0.711 without, on 3 test tasks.
0.989 with the skill and 0.711 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.
- conflict (separated-up): with the skill 1 plus or minus 0; without it 0.733333 plus or minus 0.028868.
- evidence (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
- ready (separated-up): with the skill 0.966667 plus or minus 0.028868; without it 0.4 plus or minus 0.108012.
The placebo against no skill, shown here and not counted: UNDERPOWERED
Receipt 6ee5ccb7d05d791a9e90f5ece0fcd8bffd092a0f837ea76dc33287dc1cd16d01
prototype mattpocock/skills Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: clearly helped.
Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO
prototype mattpocock/skills
Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below
- logic (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.525 plus or minus 0.05.
- ui (separated-up): with the skill 0.93 plus or minus 0.057009; with the same-length placebo 0.33 plus or minus 0.175357.
- unclear (separated-up): with the skill 0.7625 plus or minus 0.131498; with the same-length placebo 0.4 plus or minus 0.094868.
Against no skill: The skill clearly helped: 0.897 with it and 0.448 without, on 3 test tasks.
0.897 with the skill and 0.448 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.
- logic (separated-up): with the skill 1 plus or minus 0; without it 0.5 plus or minus 0.
- ui (separated-up): with the skill 0.93 plus or minus 0.057009; without it 0.492857 plus or minus 0.171825.
- unclear (separated-up): with the skill 0.7625 plus or minus 0.131498; without it 0.35 plus or minus 0.
The placebo against no skill, shown here and not counted: UNDERPOWERED
Receipt e8e18fbaec9a403354b12b1761968687e8213bbf57350def1a6164eccf6226aa
grilling mattpocock/skills Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: clearly helped.
Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO
grilling mattpocock/skills
Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below
- frontier (separated-up): with the skill 0.983333 plus or minus 0.028868; with the same-length placebo 0.75 plus or minus 0.
- settled (underpowered): with the skill 0.9 plus or minus 0.106066; with the same-length placebo 0.925 plus or minus 0.093541.
- finish (underpowered): with the skill 0.733333 plus or minus 0.028868; with the same-length placebo 0.75 plus or minus 0.
Against no skill: The skill clearly helped: 0.872 with it and 0.817 without, on 3 test tasks.
0.872 with the skill and 0.817 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.
- frontier (separated-up): with the skill 0.983333 plus or minus 0.028868; without it 0.75 plus or minus 0.
- settled (underpowered): with the skill 0.9 plus or minus 0.106066; without it 1 plus or minus 0.
- finish (underpowered): with the skill 0.733333 plus or minus 0.028868; without it 0.7 plus or minus 0.05.
The placebo against no skill, shown here and not counted: UNDERPOWERED
Receipt 3400122dda9e97243aa7457a193e9514fe2a9bd8ada484ad00898d9e34f14b1a
hyperframes-animation heygen-com/hyperframes Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: clearly helped.
Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO
hyperframes-animation heygen-com/hyperframes
Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below
- timeline (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
- runtime (separated-up): with the skill 0.616667 plus or minus 0.028868; with the same-length placebo 0.033333 plus or minus 0.028868.
- template (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.6 plus or minus 0.11547.
Against no skill: The skill clearly helped: 0.872 with it and 0.640 without, on 3 test tasks.
0.872 with the skill and 0.640 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.
- timeline (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
- runtime (separated-up): with the skill 0.616667 plus or minus 0.028868; without it 0.295 plus or minus 0.146154.
- template (separated-up): with the skill 1 plus or minus 0; without it 0.625 plus or minus 0.144338.
The placebo against no skill, shown here and not counted: PLACEBO_BELOW_BASELINE
Receipt cb7fee140bf69efb5bd73549a8d01f49394e817be92b7ffe84885d6e72297cfe
design-mobile-apps designed-by-ai/skills Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: clearly helped.
Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO
design-mobile-apps designed-by-ai/skills
Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below
- response (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.95 plus or minus 0.
- shots (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.825 plus or minus 0.095743.
- ids (underpowered): with the skill 0.75 plus or minus 0; with the same-length placebo 0.766667 plus or minus 0.028868.
Against no skill: The skill clearly helped: 0.917 with it and 0.820 without, on 3 test tasks.
0.917 with the skill and 0.820 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.
- response (separated-up): with the skill 1 plus or minus 0; without it 0.891667 plus or minus 0.073598.
- shots (separated-up): with the skill 1 plus or minus 0; without it 0.78 plus or minus 0.097468.
- ids (underpowered): with the skill 0.75 plus or minus 0; without it 0.7875 plus or minus 0.047871.
The placebo against no skill, shown here and not counted: UNDERPOWERED
Receipt ee86c450b8a7c2d805f600b88fdc4e1f8ef83b6bcf9798c5a86ffbd09ce29b97
hyperframes-core heygen-com/hyperframes Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: clearly helped.
Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO
hyperframes-core heygen-com/hyperframes
Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below
- root (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
- transform (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.75 plus or minus 0.
- exit (separated-up): with the skill 0.75 plus or minus 0; with the same-length placebo 0.233333 plus or minus 0.028868.
Against no skill: The skill clearly helped: 0.917 with it and 0.658 without, on 3 test tasks.
0.917 with the skill and 0.658 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.
- root (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
- transform (separated-up): with the skill 1 plus or minus 0; without it 0.725 plus or minus 0.05.
- exit (separated-up): with the skill 0.75 plus or minus 0; without it 0.25 plus or minus 0.141421.
The placebo against no skill, shown here and not counted: UNDERPOWERED
Receipt e3ee1e3bd056d398f04c8610b9067a0fd5af11dafdd9cb04cba1d740da547193
Did not beat the fake
reddit-automation flowkit-labs/skills Did not beat the fake Scored below the fake on at least one task. Compared with no skill: clearly hurt.
Label: Delete candidate Against the same-length placebo: below the fake BELOW_PLACEBO
reddit-automation flowkit-labs/skills
Against the same-length placebo: case rule: at least one case scored below the same-length placebo
- rank (separated-down): with the skill 0.78 plus or minus 0.067082; with the same-length placebo 1 plus or minus 0.
- reply (underpowered): with the skill 0.7625 plus or minus 0.062915; with the same-length placebo 0.816667 plus or minus 0.143759.
- rules (underpowered): with the skill 0.875 plus or minus 0.144338; with the same-length placebo 1 plus or minus 0.
Against no skill: The skill clearly hurt: 0.806 with it and 0.958 without, on 3 test tasks.
0.806 with the skill and 0.958 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.
- rank (separated-down): with the skill 0.78 plus or minus 0.067082; without it 0.983333 plus or minus 0.028868.
- reply (underpowered): with the skill 0.7625 plus or minus 0.062915; without it 0.89 plus or minus 0.151658.
- rules (underpowered): with the skill 0.875 plus or minus 0.144338; without it 1 plus or minus 0.
The placebo against no skill, shown here and not counted: UNDERPOWERED
Receipt ecc1cb963a360697df5142934d4390b53ae7bcc091303a8d0c6ad56555757ebe
grill-with-docs mattpocock/skills tools on Did not beat the fake No task differed from the fake by enough to count. Compared with no skill: clearly helped.
Label: Delete candidate Against the same-length placebo: no difference NO_DIFFERENCE
grill-with-docs mattpocock/skills
Graded on the end state: each draw's score is its task's check, written before any call. The judge's grade of the reply is secondary and enters no reading.
Against the same-length placebo: case rule: no case differed from the same-length placebo at this effect floor
- gd-booking (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
- gd-cart-order (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
- gd-account-user (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
Against no skill: The skill clearly helped: 1.000 with it and 0.833 without, on 3 test tasks.
1.000 with the skill and 0.833 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.
- gd-booking (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
- gd-cart-order (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
- gd-account-user (separated-up): with the skill 1 plus or minus 0; without it 0.5 plus or minus 0.
Draws with a Skill call to grilling: 9 of 9 with the skill, 6 of 9 with no skill, 9 of 9 with the placebo.
Recorded cost USD 31.97 over 27 generations, judging included.
Receipt bb52fbbc6320aa3fc879ccaad9dcdc64f1680eb4ff4777fd4804f678efe9d2eb
agent-browser vercel-labs/agent-browser tools on Did not beat the fake No task differed from the fake by enough to count. Compared with no skill: no clear difference.
Label: Delete candidate Against the same-length placebo: no difference NO_DIFFERENCE
agent-browser vercel-labs/agent-browser
Graded on the end state: each draw's score is its task's check, written before any call. The judge's grade of the reply is secondary and enters no reading.
Against the same-length placebo: case rule: no case differed from the same-length placebo at this effect floor
- ab-ticket (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
- ab-status (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
- ab-note (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
Against no skill: No clear difference: 1.000 with the skill and 1.000 without, with enough answers to have seen a gap of 0.05.
there were enough answers to have seen a gap of 0.05, and none of the 3 test tasks showed one.
- ab-ticket (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
- ab-status (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
- ab-note (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
Recorded cost USD 79.80 over 27 generations, judging included.
Receipt 0b714814822b461a0fcc7bdb0f7e6ea45e195cc20977c10357596cfde49f3aa6
hyperframes heygen-com/hyperframes tools on Did not beat the fake No task differed from the fake by enough to count. Compared with no skill: no clear difference.
Label: Delete candidate Against the same-length placebo: no difference NO_DIFFERENCE
hyperframes heygen-com/hyperframes
Graded on the end state: each draw's score is its task's check, written before any call. The judge's grade of the reply is secondary and enters no reading.
Against the same-length placebo: case rule: no case differed from the same-length placebo at this effect floor
- hf-title (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
- hf-logo (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
- hf-caption (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
Against no skill: No clear difference: 1.000 with the skill and 1.000 without, with enough answers to have seen a gap of 0.05.
there were enough answers to have seen a gap of 0.05, and none of the 3 test tasks showed one.
- hf-title (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
- hf-logo (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
- hf-caption (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
Recorded cost USD 41.57 over 27 generations, judging included.
Receipt 2c4cb0f1a92b8c3e0c17849062ac709d62717436355b3bf25c07c81118c12f2e
Can't tell yet
setup-matt-pocock-skills mattpocock/skills Can't tell yet Not enough answers to tell it from the fake. Compared with no skill: too few answers to tell.
Label: Cannot tell Against the same-length placebo: not enough draws UNDERPOWERED
setup-matt-pocock-skills mattpocock/skills
Against the same-length placebo: case rule: Not enough draws to conclude at this effect floor
- local (underpowered): with the skill 1 plus or minus 0; with the same-length placebo 0.983333 plus or minus 0.028868.
- labels (not-separated): with the skill 0.75 plus or minus 0; with the same-length placebo 0.75 plus or minus 0.
- monorepo (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
Against no skill: Too few answers to tell: 0.917 with the skill and 0.907 without, but the scores varied too much to call it.
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case monorepo: the two arms' spreads sum to 0.095, at or above the 0.05 floor).
- local (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
- labels (underpowered): with the skill 0.75 plus or minus 0; without it 0.783333 plus or minus 0.028868.
- monorepo (underpowered): with the skill 1 plus or minus 0; without it 0.9375 plus or minus 0.094648.
The placebo against no skill, shown here and not counted: UNDERPOWERED
Receipt 2b2e2930934e39b9ce3013b2b963adb1f886f24a14e0f3678eb52bb1eb219706
frontend-design anthropics/skills Can't tell yet Not enough answers to tell it from the fake. Compared with no skill: too few answers to tell.
Label: Cannot tell Against the same-length placebo: not enough draws UNDERPOWERED
frontend-design anthropics/skills
Against the same-length placebo: case rule: Not enough draws to conclude at this effect floor
- identity (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
- brief (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
- content (underpowered): with the skill 0.835714 plus or minus 0.114434; with the same-length placebo 0.875 plus or minus 0.112916.
Against no skill: Too few answers to tell: 0.945 with the skill and 0.980 without, but the scores varied too much to call it.
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case content: the two arms' spreads sum to 0.169, at or above the 0.05 floor).
- identity (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
- brief (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
- content (underpowered): with the skill 0.835714 plus or minus 0.114434; without it 0.94 plus or minus 0.054772.
The placebo against no skill, shown here and not counted: UNDERPOWERED
Receipt 03d93027ab744923fb42141d774614a030ab85728ab26e7b650a3b2c86a0c281
media-use heygen-com/hyperframes Can't tell yet Not enough answers to tell it from the fake. Compared with no skill: too few answers to tell.
Label: Cannot tell Against the same-length placebo: not enough draws UNDERPOWERED
media-use heygen-com/hyperframes
Against the same-length placebo: case rule: Not enough draws to conclude at this effect floor
- reuse (underpowered): with the skill 0.941667 plus or minus 0.102062; with the same-length placebo 0.7375 plus or minus 0.103078.
- logo (underpowered): with the skill 1 plus or minus 0; with the same-length placebo 0.966667 plus or minus 0.08165.
- grade (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
Against no skill: Too few answers to tell: 0.981 with the skill and 0.942 without, but the scores varied too much to call it.
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case reuse: the two arms' spreads sum to 0.131, at or above the 0.05 floor).
- reuse (underpowered): with the skill 0.941667 plus or minus 0.102062; without it 0.883333 plus or minus 0.028868.
- logo (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
- grade (underpowered): with the skill 1 plus or minus 0; without it 0.941667 plus or minus 0.102062.
The placebo against no skill, shown here and not counted: PLACEBO_BELOW_BASELINE
Receipt 593f226fd71b4f3ada583bd7dbb755a6a5e2ebd44c81274b763aa5ce5485df22
find-skills vercel-labs/skills tools on Can't tell yet Not enough answers to tell it from the fake. Compared with no skill: too few answers to tell.
Label: Cannot tell Against the same-length placebo: not enough draws UNDERPOWERED
find-skills vercel-labs/skills
Graded on the end state: each draw's score is its task's check, written before any call. The judge's grade of the reply is secondary and enters no reading.
Against the same-length placebo: case rule: Not enough draws to conclude at this effect floor
- fs-changelog (underpowered): with the skill 0.777778 plus or minus 0.172133; with the same-length placebo 0.833333 plus or minus 0.19245.
- fs-pdf (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
- fs-none (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
Against no skill: Too few answers to tell: 0.926 with the skill and 0.926 without, but the scores varied too much to call it.
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case fs-changelog: the two arms' spreads sum to 0.344, at or above the 0.05 floor).
- fs-changelog (underpowered): with the skill 0.777778 plus or minus 0.172133; without it 0.777778 plus or minus 0.172133.
- fs-pdf (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
- fs-none (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
Recorded cost USD 116.14 over 34 generations, judging included.
Receipt 745ed6e2dc62f204c0ccafd7b57e545027926fdf0486e61d633e3552a67854bc
grill-me mattpocock/skills tools on Can't tell yet Not enough answers to tell it from the fake. Compared with no skill: too few answers to tell.
Label: Cannot tell Against the same-length placebo: not enough draws UNDERPOWERED
grill-me mattpocock/skills
Graded on the end state: each draw's score is its task's check, written before any call. The judge's grade of the reply is secondary and enters no reading.
Against the same-length placebo: case rule: Not enough draws to conclude at this effect floor
- gm-report-app (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
- gm-search-cache (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
- gm-split-users (underpowered): with the skill 0.958333 plus or minus 0.102062; with the same-length placebo 1 plus or minus 0.
Against no skill: Too few answers to tell: 0.986 with the skill and 1.000 without, but the scores varied too much to call it.
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case gm-split-users: the two arms' spreads sum to 0.102, at or above the 0.05 floor).
- gm-report-app (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
- gm-search-cache (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
- gm-split-users (underpowered): with the skill 0.958333 plus or minus 0.102062; without it 1 plus or minus 0.
Draws with a Skill call to grilling: 12 of 12 with the skill, 9 of 9 with no skill, 9 of 9 with the placebo.
Recorded cost USD 27.86 over 30 generations, judging included.
Receipt b18ad267e07543695d8d1ca295187f051991fd065ba80d2e7707de5d4717042d
hyperframes-registry heygen-com/hyperframes tools on Can't tell yet Not enough answers to tell it from the fake. Compared with no skill: too few answers to tell.
Label: Cannot tell Against the same-length placebo: not enough draws UNDERPOWERED
hyperframes-registry heygen-com/hyperframes
Graded on the end state: each draw's score is its task's check, written before any call. The judge's grade of the reply is secondary and enters no reading.
Against the same-length placebo: case rule: Not enough draws to conclude at this effect floor
- hr-screen (underpowered): with the skill 0.888889 plus or minus 0.272166; with the same-length placebo 1 plus or minus 0.
- hr-grain (underpowered): with the skill 1 plus or minus 0; with the same-length placebo 0.888889 plus or minus 0.272166.
- hr-chart (not-separated): with the skill 0.666667 plus or minus 0; with the same-length placebo 0.666667 plus or minus 0.
Against no skill: Too few answers to tell: 0.852 with the skill and 0.889 without, but the scores varied too much to call it.
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case hr-screen: the two arms' spreads sum to 0.272, at or above the 0.05 floor).
- hr-screen (underpowered): with the skill 0.888889 plus or minus 0.272166; without it 1 plus or minus 0.
- hr-grain (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
- hr-chart (not-separated): with the skill 0.666667 plus or minus 0; without it 0.666667 plus or minus 0.
Recorded cost USD 282.40 over 33 generations, judging included.
Receipt 78dfbfddb6ffd4d2ad88cf2fa09b12bb175cbb3d285ec09172ffb080417e4708
Keep in mind
- The six skills marked tools on ran with tools on and were graded on what Claude Code did, not on its reply. That is not the same test as the other fourteen.
- Beating the fake is not the same as beating no skill. Each skill's details show both.
- Can't tell yet means there was not enough evidence to call it. It does not mean the skill had no effect.
- This is one model, Claude Fable 5.2, and three tasks per skill. It says nothing about other models or other tasks.
Report 020: Fable 5.2 against Fable 5.1 on three skills
Three skills, each run on both models with the skill loaded. On Claude Fable 5.2, compared with Claude Fable 5.1:
- code-review-and-quality: scored higher on Fable 5.2.
- git-workflow-and-versioning: no clear difference.
- documentation-and-adrs: scored lower on Fable 5.2.
The table, how it was run and its limits
Its own section, outside the twenty-skill headline. Three of Addy Osmani's skills, one case each, run fresh on Claude Fable 5.2 and Claude Fable 5.1, with the skill and without it, judged by claude-haiku-5-5 at five samples. Each verdict reads the two with-skill arms against each other, in Report 011's labels; upward means Fable 5.2 scored above Fable 5.1.
No effort setting was passed: Fable 5.2 ran at its default, medium, and Fable 5.1 at its default, high. The table reads the two models as users get them, not at the same effort.
Report 020's judge differs from Report 011's, so Report 020's figures are not set beside Report 011's as if comparable.
| Skill | Fable 5.1, with the skill | Fable 5.2, with the skill | Verdict | Without the skill, context only (Fable 5.1; Fable 5.2) |
|---|---|---|---|---|
| code-review-and-quality | 0.63 ± 0.13, 5 draws | 0.83 ± 0.03, 3 draws | separation detected, upward | 0.21 ± 0.00, 3 draws; 0.62 ± 0.07, 5 draws |
| git-workflow-and-versioning | 0.84 ± 0.01, 3 draws | 0.82 ± 0.02, 3 draws | no separation detected | 0.84 ± 0.01, 3 draws; 0.30 ± 0.00, 3 draws |
| documentation-and-adrs | 0.75 ± 0.03, 3 draws | 0.50 ± 0.08, 4 draws | separation detected, downward | 0.58 ± 0.14, 7 draws; 0.39 ± 0.05, 3 draws |
The means without the skill are context; no verdict is read from them.
Limits
One case per skill. Nothing in the report describes how any of the three skills behaves on any other task, or how Fable 5.2 behaves in general.
Few draws. Each arm draws between 3 and 10 generations, and each case's band is the spread across its draws. With one case there is no spread across cases.
The judge is not Report 011's. The figures are comparable within Report 020 only.
The two models ran at their own default effort, medium and high.
It does not say which model is better at these skills in general, why a verdict reads as it does, or that no separation means the two models score alike. It does not compare any figure with Report 011 or with Reports 017, 018 or 019.
How we tested
The headline as fixed before the run, the full method, the judge, the price, the run record and the limits.
Placebo comparison report. Twenty skills on Fable 5.2, each against no skill and against a same-length text from a fixed corpus, judged by Haiku 5.5: fourteen graded on the reply, and six run with tools on and graded on what Claude Code left.
The headline, as fixed before the run
Of the 14 skills graded on the reply, 10 beat the fake on Fable 5.2.
Of the 6 skills graded on the end state with tools on, 0 beat the fake on Fable 5.2.
10 of 20 skills beat the fake on Fable 5.2: by the case rule, each scored above the same-length placebo on at least one of three tasks and below it on none. A skill that did not beat the fake is not shown to do no better than it.
Of the 20 skills counted: 10 beat the fake, 1 below the fake, 3 no difference, 0 inconclusive, 6 not enough draws, 0 not measured.
Of the 20: ABOVE_PLACEBO 10, BELOW_PLACEBO 1, NO_DIFFERENCE 3, INCONCLUSIVE 0, UNDERPOWERED 6, NOT_MEASURED 0. Beat the fake means ABOVE_PLACEBO only.
In plain labels: of the fourteen, Keep 10, Delete candidate 1, Cannot tell 3; of the six, Keep 0, Delete candidate 3, Cannot tell 3.
Six of these skills hand work to other skills or tools, so they ran with tools on, after the other fourteen, and each was scored on what Claude Code ended up doing.
The note fixed before the text-only run counted these six differently; Maverick's ruling of 11 Nov 2026 counts them by their tools-on reading.
The six are scored on the end state; the fourteen on the reply. The two are not the same test.
Where the fake scored below no skill, beating it does not mean beating no skill; each row shows both readings.
The judge is claude-haiku-5-5 at five samples, a different model from the target. For the six, its grade of the reply is a secondary score and enters no reading.
No effort setting was passed, so claude-fable-5-2 ran at its catalog default, medium.
The six, with tools on
What was installed, the same in every version of a skill: the called skills (grilling, domain-modeling and hyperframes-core), the command-line tools (git, npm and agent-browser), Chrome 155.0.8059.79, and local copies of the skills registry, an npm registry, the HyperFrames registry and a web app, each served on the draw's own loopback. There was no live internet: each draw's network was none or its own loopback. The router or pointer text was the only difference between the versions: the skill's text, no text, or a same-length placebo. The grade was the end state Claude Code left: files, installed skills, commands, app state and Skill calls, read by a check written before any call. The judge's grade of the reply was secondary and entered no reading.
Every version scored at or near the top on most tasks, so there was little room for a difference: 45 of 54 task means, one per task and version, were 0.9 or above.
Every version of the two pointer skills called grilling, because it was installed in all of them. Draws with a grilling call: grill-me 12 of 12 with the skill, 9 of 9 with no skill, 9 of 9 with the placebo; grill-with-docs 9 of 9 with the skill, 6 of 9 with no skill, 9 of 9 with the placebo.
Tools-on runs cost more than planned for 3 of the six. The plan projected USD 2.19 a generation and USD 0.01 a judged draw, from Report 016's 75th percentile at the Fable 5.2 estimate. Recorded cost a generation, judging included: find-skills USD 3.42, grill-me USD 0.93, grill-with-docs USD 1.18, agent-browser USD 2.96, hyperframes USD 1.54, hyperframes-registry USD 8.56. hyperframes-registry's first attempt stopped at its ceiling of USD 148.96 after USD 160.57 and was run again under a raised cap, with a ceiling of USD 662.08; both attempts' costs are counted.
The same six as text-only receipts
Published and linked as context. They enter no count and no headline line.
find-skills vercel-labs/skills (text-only, not counted) Did not beat the fake Scored below the fake on at least one task. Compared with no skill: no verdict, delegated skill not tested.
Label: Delete candidate Against the same-length placebo: below the fake BELOW_PLACEBO
find-skills vercel-labs/skills
These text tasks test the router's written response, not the delegated skill. Report 015 Amendment 1.
Graded on the reply, with tools off. The tools-on reading above is the one counted.
Against the same-length placebo: case rule: at least one case scored below the same-length placebo
- search (separated-down): with the skill 0.733333 plus or minus 0.028868; with the same-length placebo 1 plus or minus 0.
- missing (underpowered): with the skill 0.97 plus or minus 0.067082; with the same-length placebo 1 plus or minus 0.
- scope (underpowered): with the skill 0.983333 plus or minus 0.028868; with the same-length placebo 1 plus or minus 0.
Against no skill: No verdict: delegated skill not tested.
The placebo against no skill, shown here and not counted: NO_DIFFERENCE
Receipt 3c132baf43df44ed15adbed42569afd59b606bf997dc80528ef8080c94758bb5
grill-me mattpocock/skills (text-only, not counted) Can't tell yet Not enough answers to tell it from the fake. Compared with no skill: too few answers to tell.
Label: Cannot tell Against the same-length placebo: not enough draws UNDERPOWERED
grill-me mattpocock/skills
About grill-me and grill-with-docs: each skill's whole text is an instruction to call another skill. These text tasks forbid tools, so each reading reflects that one-line hand-off, not the skill it calls. Both stay in the pre-registered counts.
Graded on the reply, with tools off. The tools-on reading above is the one counted.
Against the same-length placebo: case rule: Not enough draws to conclude at this effect floor
- plan (underpowered): with the skill 0.683333 plus or minus 0.196638; with the same-length placebo 0.633333 plus or minus 0.10328.
- constraint (underpowered): with the skill 0.6 plus or minus 0.106066; with the same-length placebo 0.483333 plus or minus 0.160208.
- followup (underpowered): with the skill 0.9375 plus or minus 0.047871; with the same-length placebo 0.883333 plus or minus 0.028868.
Against no skill: Too few answers to tell: 0.740 with the skill and 0.717 without, but the scores varied too much to call it.
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case plan: the two arms' spreads sum to 0.343, at or above the 0.05 floor).
- plan (underpowered): with the skill 0.683333 plus or minus 0.196638; without it 0.685714 plus or minus 0.146385.
- constraint (underpowered): with the skill 0.6 plus or minus 0.106066; without it 0.516667 plus or minus 0.028868.
- followup (underpowered): with the skill 0.9375 plus or minus 0.047871; without it 0.95 plus or minus 0.05.
The placebo against no skill, shown here and not counted: UNDERPOWERED
Receipt f76798ed009f4f5811dd935ffd1c215ad6675f6e3336c3a79ce1c0cd2e08aa4d
grill-with-docs mattpocock/skills (text-only, not counted) Can't tell yet Not enough answers to tell it from the fake. Compared with no skill: too few answers to tell.
Label: Cannot tell Against the same-length placebo: not enough draws UNDERPOWERED
grill-with-docs mattpocock/skills
About grill-me and grill-with-docs: each skill's whole text is an instruction to call another skill. These text tasks forbid tools, so each reading reflects that one-line hand-off, not the skill it calls. Both stay in the pre-registered counts.
Graded on the reply, with tools off. The tools-on reading above is the one counted.
Against the same-length placebo: case rule: Not enough draws to conclude at this effect floor
- decision (underpowered): with the skill 0.95 plus or minus 0.057735; with the same-length placebo 0.93 plus or minus 0.097468.
- adr (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
- revision (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
Against no skill: Too few answers to tell: 0.983 with the skill and 0.976 without, but the scores varied too much to call it.
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case decision: the two arms' spreads sum to 0.148, at or above the 0.05 floor).
- decision (underpowered): with the skill 0.95 plus or minus 0.057735; without it 0.928571 plus or minus 0.090633.
- adr (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
- revision (not-separated): with the skill 1 plus or minus 0; without it 1 plus or minus 0.
The placebo against no skill, shown here and not counted: UNDERPOWERED
Receipt adc67d2d503f41826a483cd7dbce047dfead9a30ab09a34acdb36e7bdad30b2f
agent-browser vercel-labs/agent-browser (text-only, not counted) Did not beat the fake Scored below the fake on at least one task. Compared with no skill: no verdict, delegated skill not tested.
Label: Delete candidate Against the same-length placebo: below the fake BELOW_PLACEBO
agent-browser vercel-labs/agent-browser
These text tasks test the router's written response, not the delegated skill. Report 015 Amendment 1.
Graded on the reply, with tools off. The tools-on reading above is the one counted.
Against the same-length placebo: case rule: at least one case scored below the same-length placebo
- web (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
- desktop (separated-down): with the skill 0.733333 plus or minus 0.028868; with the same-length placebo 0.92 plus or minus 0.09083.
- dashboard (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.65 plus or minus 0.1.
Against no skill: No verdict: delegated skill not tested.
The placebo against no skill, shown here and not counted: UNDERPOWERED
Receipt 86257def619b1db544acda4b26b243fdb6ff419a7648936c22c4736bc54c8197
hyperframes heygen-com/hyperframes (text-only, not counted) Can't tell yet Not enough answers to tell it from the fake. Compared with no skill: no verdict, delegated skill not tested.
Label: Cannot tell Against the same-length placebo: not enough draws UNDERPOWERED
hyperframes heygen-com/hyperframes
These text tasks test the router's written response, not the delegated skill. Report 015 Amendment 1.
Graded on the reply, with tools off. The tools-on reading above is the one counted.
Against the same-length placebo: case rule: Not enough draws to conclude at this effect floor
- existing (underpowered): with the skill 0.75 plus or minus 0; with the same-length placebo 0.8 plus or minus 0.093541.
- usage (not-separated): with the skill 1 plus or minus 0; with the same-length placebo 1 plus or minus 0.
- port (underpowered): with the skill 0.875 plus or minus 0.144338; with the same-length placebo 1 plus or minus 0.
Against no skill: No verdict: delegated skill not tested.
The placebo against no skill, shown here and not counted: UNDERPOWERED
Receipt d82a78f5d986f1822f98d56ad2f55370cdc51b3aad135d0f0bda156c6b319f38
hyperframes-registry heygen-com/hyperframes (text-only, not counted) Beat the fake Scored above the fake on at least one task, and below it on none. Compared with no skill: no verdict, delegated skill not tested.
Label: Keep Against the same-length placebo: beat the fake ABOVE_PLACEBO
hyperframes-registry heygen-com/hyperframes
These text tasks test the router's written response, not the delegated skill. Report 015 Amendment 1.
Graded on the reply, with tools off. The tools-on reading above is the one counted.
Against the same-length placebo: case rule: at least one case scored above the same-length placebo, and none below
- block (separated-up): with the skill 0.85 plus or minus 0.05; with the same-length placebo 0.483333 plus or minus 0.028868.
- component (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.8375 plus or minus 0.118145.
- search (separated-up): with the skill 1 plus or minus 0; with the same-length placebo 0.516667 plus or minus 0.028868.
Against no skill: No verdict: delegated skill not tested.
The placebo against no skill, shown here and not counted: UNDERPOWERED
Receipt 64045ab20a9d60331a66f90956f3f7f8b1361fb4881bf101c7d272b8e448291e
Method
These sections are the pre-registered notes, as fixed before each run: plan (a)'s for the text-only twenty, then spec 786's for the six with tools on.
What each receipt shows
Each of the 20 frozen attempts runs one skill of Report 017's frozen roster, the same pinned skill bytes and the same committed three-task suite, on claude-fable-5-2, graded by claude-haiku-5-5, with three arms: the skill text, no skill, and a text of the skill's length in bytes from the revision 1 fixed corpus (spec 440). The runner is Driftproof 0.28.0 and the CLI is Claude Code 2.1.311, on the claude-cli subscription surface.
The report shows two readings for each receipt, each in lib/verdict.js's own words:
- the verdict:
receiptVerdict, the skill against no skill, in the receipt reader's words; - the placebo reading:
placeboLine(placeboReading(receipt)), the skill against the same-length text.
The verdict shown is the Report 019 receipt's own. The verdict reads only the skill and no-skill arms, so the placebo arm's replies move it in no way. No figure other than these readings and their counts enters the headline.
The judge
The judge is claude-haiku-5-5, the judge the judge policy fixes for the reports Driftproof publishes: "That judge is fixed at claude-haiku-5-5 and changed only rarely" (/judge-policy/). Reports 017 and 018 were graded by the same judge. No judge policy or registry entry moves for this plan.
The judge is a different model from the target. lib/receipt.js records no same-model limitation for this pair, and the plan pins that as judge_relationship. The report says nothing about judge accuracy.
The effort setting
No effort setting is passed, so the catalog default for claude-fable-5-2, medium, applies. That is the default a user of the CLI gets. The report says so beside the readings, and does not present them as the model at its highest effort.
The price
claude-fable-5-2 has no published price in this repository. Every cost figure of this report prices its calls at an estimate, USD 10 input and USD 50 output per million tokens, the most expensive known tier, and says so where it is shown. Metered spend on the subscription surface is USD 0.
Report 019 stands alone
The page and every Report 019 claim show no Report 017 or Report 018 reading, verdict, count or figure. One methods sentence names the design and what changed: Report 017's same-length placebo design, the same twenty skills, suites and placebo corpus, with claude-fable-5-2 answering, graded by the same fixed judge, on runner 0.28.0. The judge policy holds that "results scored by different graders are not compared directly"; here the grader is the same, and the target and runner are not, so the report compares nothing across reports: it does not subtract one reading from another, does not count skills that moved, and does not say which model a skill helps more.
Which receipt is read
The report reads exactly one sealed receipt per attempt id. It is the receipt the campaign's manifest chain records for that attempt, read through the campaign ledger. The first completed receipt is the only one: an attempt with a sealed receipt is never run again, and a second receipt for the same attempt id refuses the whole chain. No receipt is chosen after its results are seen.
An attempt without a receipt may be run again in a later invocation of the same frozen plan, at most three runs in all, its earlier costs counted against the USD 375 cap. Every failure stays in the ledger. A failure carries no reading, so running it again chooses nothing.
The campaign has one root directory. It keeps the plan and this note as first started and an append-only log with one line per start. A second start with no prior chain is refused, and a resume must continue the log's latest manifest. The report reads only the chain that ends at the log's latest manifest, and refuses if the root holds any other manifest under this plan. A cost correction may be added after a run and is never rewritten: each manifest records the corrections in force when it started.
Terms
"Beat the fake" means exactly one thing: the placebo reading is ABOVE_PLACEBO. In the reader's words, "case rule: at least one case scored above the same-length placebo, and none below". The case rule reads each of the three tasks by its draw bands, so this says no more than that, on at least one of three tasks, the skill's band sat above the placebo's at the effect floor, and on none below.
No other reading beats the fake. BELOW_PLACEBO, NO_DIFFERENCE, INCONCLUSIVE, UNDERPOWERED and NOT_MEASURED do not. None of them shows that a skill does no better than the fake.
A missing receipt reads NOT_MEASURED: a failed, cancelled or budget-stopped attempt is counted as not read, never as a result. A stub receipt also reads NOT_MEASURED.
Secondary reading: the placebo against no skill
Beside each skill, and never in the headline, the report shows a second reading: the same case rule over the placebo's rows against the no-skill rows of the same receipt (spec 613's tally.cjs secondaryReading). Its tokens are PLACEBO_ABOVE_BASELINE, PLACEBO_BELOW_BASELINE, NO_DIFFERENCE, INCONCLUSIVE, UNDERPOWERED and NOT_MEASURED. It is read only where the placebo reading is read, and refused wherever that is refused. It is not counted, not subtracted from the placebo reading, and not combined with it.
With tools on: what runs
Six skills from Report 019's roster: grill-me, grill-with-docs, find-skills, hyperframes-registry, agent-browser and hyperframes. Each runs on claude-fable-5-2 through Claude Code 2.1.311 in the repository lane's tools-on mode, with Report 019's pinned skill bytes. No effort setting is passed, so the catalog default for claude-fable-5-2, medium, applies.
Each attempt has three tasks and three arms: the skill as system text, no system text, and a same-length placebo from Report 019's corpus. Every arm of an attempt gets the same installed world: the called skills, the pinned command-line tools and hashed local fixtures on the draw's own loopback. Nothing reaches the internet. Each arm draws at least 3 and at most 6 times per task, by the runner's sampling rule.
With tools on: how a draw is scored
The primary score of a draw is its task's check, frozen in the plan by sha256. The check reads what Claude Code ended up doing: files, tool calls, Skill calls and the fixtures' state. For the two pointer skills it also reads the form of the reply: numbered questions, each with a recommended answer. No check credits an action from the reply's words.
The secondary score is the fixed judge's grade of the reply and its captured files: claude-haiku-5-5 at five samples, as the judge policy fixes it ("That judge is fixed at claude-haiku-5-5 and changed only rarely"). The secondary score never enters a verdict.
A draw whose end state cannot be read is not measured. It is never scored 0.
With tools on: the readings
Each skill's verdict and placebo reading are lib/verdict.js's own, read over the primary scores in its own words. Only ABOVE_PLACEBO beats the fake. An attempt without a verified receipt reads "not measured".
Limits
The placebo text is not inert: on a task that supplies the facts its answer needs, a same-length text about landscape and village life can pull an answer off them, so the fake can score below no skill. The secondary reading shows where it did.
The projection in plan.json reprices Reports 017 and 018's recorded tokens at the claude-fable-5-2 estimate for generation and at Haiku 5.5 rates for judging. Fable 5.2 may answer at another length and draw another number of times, so the projection is a planning assumption, not a bound. The cap and each attempt's ceiling are the bounds.
It does not say why a skill helps, that a skill and the fake are the same, or that the fake carries no information about a task. It says nothing about a skill that was not run with the placebo arm. It does not combine the 20 readings into one score. It does not say how Fable 5.2, Sonnet 5.5 and Haiku 5.5 compare: no reading, verdict or count of Reports 015, 017 or 018 appears on the page or in any Report 019 claim.
- Three tasks per skill, and at most six draws per arm.
- The six are scored on the end state; the fourteen on the reply. The two are not the same test.
- The tools-on attempts run from the source branch at the commit their campaign records, not from a release. The receipt's runner version is the one config.js records at that commit.
- The cost is priced at an estimate for claude-fable-5-2, USD 10 and USD 50 per million tokens.
The placebo corpus was chosen without reference to any task, so a placebo text can still bear on a task by chance.
Unkeyed seals and retained outputs permit checking internal consistency; they do not authenticate an origin.
Run record
Spend, read from the records. Report 019 text-only USD 140.06: USD 135.55 in the chain read here and USD 4.52 in stopped chains recorded in specs/783-fable-5-2-scope/chains.json. With tools on USD 790.31: USD 740.32 in the campaign manifest, the stopped attempt included, and USD 49.99 in smoke runs recorded in specs/786-tools-on-routers/freeze.json. Report 020 USD 10.03. Recorded costs are API-equivalent estimates, not a subscription invoice, with claude-fable-5-2 priced at an estimate of USD 10 input and USD 50 output per million tokens. Metered spend on the subscription surface was USD 0.
The text-only twenty: plan SHA-256 e557c3b4a5849ccfff353352a55c28742becdb5cb51be68fe9b06d0ebb7171ce, pre-registered note SHA-256 76e1913b0ae3bec0f91ef72b72685922af2906e977320d35a595b77161676a94, run at 17cdf896f7cf7fab190f663d247051321f0c6b3f, runner 0.28.0, Claude Code 2.1.311.
receipts/report-019/campaign-001/manifest.json, SHA-256d8556d35959da7e3e884571f0377d45b8e2cae7eba06a11c24ff33fbd0ff6b11receipts/report-019/campaign-002/manifest.json, SHA-2563ae34883ab2858618db65e0325cff9f87f044c60381c81446827eeb998b043cfreceipts/report-019/campaign-003/manifest.json, SHA-2565c4d439a3a1dd26354cd9c490d300f88adc862916e984f0db230b42487604bda
The six with tools on: plan SHA-256 f7968ae3753384d767b70873bc3f646857abc8afdc8c077483645d66f36808e0, pre-registered note SHA-256 25f0f51911413fc75ceca2020db64a5afc0cbaec4fef7f7e42dbdfc7f9af6f74, manifest receipts/report-019-tools-on/manifest.json, SHA-256 d06a8f03b59752815559e074f9ebd51f6e9424e7784750f2eb4a8bf6e39c0659. Each attempt ran from the source branch at the commit its start records, runner 0.28.0, Claude Code 2.1.311:
- commit
46beb63c22adae7ff5b735649695e21fd27be2d2, plan SHA-256f77e92a516922e374457431cfc46f2240288e8094998b0b7f24fbee83aa5b3a1 - commit
7be8f9bdf0a1990c2858b3e39b9322110102289e, plan SHA-256a311cf94cf1c7b3dce6a2ea9d420315d4267270086a001e5f48311f58d198145 - commit
7be8f9bdf0a1990c2858b3e39b9322110102289e, plan SHA-256a311cf94cf1c7b3dce6a2ea9d420315d4267270086a001e5f48311f58d198145 - commit
2a6f9dc5d611fc61727d2faee907c64be5b83e15, plan SHA-256c618cf2f86ed03769dcea43f15bf99e1856e673c1ddb62f91995f3f3e16bfeaa - commit
2a6f9dc5d611fc61727d2faee907c64be5b83e15, plan SHA-256c618cf2f86ed03769dcea43f15bf99e1856e673c1ddb62f91995f3f3e16bfeaa - commit
4229a9db7969601e38c5b32e253e8af132a170bf, plan SHA-256f7968ae3753384d767b70873bc3f646857abc8afdc8c077483645d66f36808e0
Report 020: plan SHA-256 40d30de73cafa2dbac9ae0303401f43d8ff50c4a55f2369ba4283f528d85e445.
receipts/report-020/campaign-001/manifest.json, SHA-25638a0cb2b90ffeac399cd5e9deb358473ee0816af217dadc74ec56cde6b3157d0, commit8550f3b03c52052e2c5a681d570a20107b58480e
The retained answers of handoff's text-only receipt show the eval account's home directory as ~/ in 5 answers. The reader restores the original from the account name and checks it against the hash the receipt seals.
Read and built by:
scripts/prepare-report-019.js, SHA-256b493a739bc93d18c71fe6def89661b28b1c7ab7658927460be4049257d9c5e51scripts/report-019-reader.js, SHA-2567764c59353736dca5f74d4d8265d2b1fb1e126e17fb0a95185e176b980a91d16scripts/answer-redaction.js, SHA-256d17de4ffeb1e120a7ad6df063d2016fd93f8d937095545abe9c5dfb9fa4f086cspecs/783-fable-5-2-scope/placebo/run-campaign.cjs, SHA-25622dcd101c6319ec07fe5091d950e63bbe7fd6e749e2b6e728d21f530fb2c3a2cspecs/783-fable-5-2-scope/boundary/run-campaign.cjs, SHA-2560fba7b1e86a5f8dc002448d40141fb26c66765d0bac463cb8f0a00380ce3d5ffspecs/613-placebo-skills-report/run-campaign.cjs, SHA-256db070608736eb9401fb8acf6a15cf800deeefbd706b18a92383242ef6609af6aspecs/613-placebo-skills-report/tally.cjs, SHA-256a7d334796f32591a9488baa2286e3ba9462305ff6a92b8b2f0c7695a96a415ddspecs/408-skill-measurements/ledger.cjs, SHA-256263a3d3423feea70aaf90074a58bcaf48d4ddb8b0eea262ec634450b17b63aaespecs/786-tools-on-routers/world.cjs, SHA-256e94047b18e0919ab60fde49503bca828af893bf33d0e36522ba7191775e9d9aalib/verdict.js, SHA-256c09edd5e1a8c62c43fa0e0d1f1a268d5d6076e474846e2cf9247a1f0e35b5b6elib/plain.js, SHA-25646f471b1487460b6101680f0b4c1191e358611020f326898e37f0e420011dbb9lib/receipt.js, SHA-256979be61ec82acca2e20c9764844188f2d1f025a34d7fbfd0365d7fed8fdef7a8scripts/offer.js, SHA-256dcc5adde7ce49b2f853708c35a7bfe671a38f38a908a01b2ecc12234c76adddb
Receipts
- improve-codebase-architecture, graded on the reply
- tdd, graded on the reply
- hyperframes-cli, graded on the reply
- setup-matt-pocock-skills, graded on the reply
- handoff, graded on the reply
- frontend-design, graded on the reply
- reddit-automation, graded on the reply
- triage, graded on the reply
- prototype, graded on the reply
- media-use, graded on the reply
- grilling, graded on the reply
- hyperframes-animation, graded on the reply
- design-mobile-apps, graded on the reply
- hyperframes-core, graded on the reply
- find-skills, with tools on
- grill-me, with tools on
- grill-with-docs, with tools on
- agent-browser, with tools on
- hyperframes, with tools on
- hyperframes-registry, with tools on
- find-skills, text-only context
- grill-me, text-only context
- grill-with-docs, text-only context
- agent-browser, text-only context
- hyperframes, text-only context
- hyperframes-registry, text-only context
- Report 020, code-review-and-quality-claude-fable-5-2-2026-11-10-ecd030219862
- Report 020, git-workflow-and-versioning-claude-fable-5-2-2026-11-10-f36e3107e67a
- Report 020, documentation-and-adrs-claude-fable-5-2-2026-11-10-dca693a087ec
- Report 020, code-review-and-quality-claude-fable-5-1-2026-11-10-79216e2ce275
- Report 020, git-workflow-and-versioning-claude-fable-5-1-2026-11-10-67e3d1e37b65
- Report 020, documentation-and-adrs-claude-fable-5-1-2026-11-10-e54efd3b4c03