Read the test. Try the question.
setup-matt-pocock-skills
The prompts and scoring guides used to test setup-matt-pocock-skills, with each report’s findings and limits.
mattpocock/skills
Configure this repo for the engineering skills: set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.
The task
local
Link to this taskWritten responses only. No tool execution, installed dependencies, media generation or cloud services are tested.
What we asked
Do not use tools, access the network, write files, run commands or install anything. Answer only in text using the supplied facts. A single-package project has no remote, already tracks requests as local markdown, and has no triage skill installed. Propose the engineering-skills setup choices.
The scoring guide
Pass threshold: 0.75.
Driftproof-authored from the pinned SKILL.md and README; no separate drafting model call.
Try it yourself
Paste the prompt into your own assistant, with or without the skill. Use the scoring guide to read the answer.
One try scored by hand is not a measurement.
Report 017: several answers per arm, each scored 5 times by a judge model. The report’s findings and caveats are below.
Report 018: several answers per arm, each scored 5 times by a judge model. The report’s findings and caveats are below.
Copying stays in your browser. Copies are never stored, counted or sent anywhere by Driftproof.
What happened on this task
Open a report to read its comparison and limits together. Beating the fake is not beating no skill.
Claude Sonnet 5.5Report 017 · Read the task result
Against the same-length fake
local (clearly higher): with the skill 0.93 plus or minus 0.057009; with the same-length placebo 0.65 plus or minus 0.1.
Against nothing
local (clearly higher): with the skill 0.93 plus or minus 0.057009; without it 0.591667 plus or minus 0.08612.
- What we tested. Report 015's twenty skills on Sonnet 5.5, each against no skill and against a same-length text from a fixed corpus, judged by Haiku 5.5.
- Beating the fake is not beating no skill. Where the fake scored below no skill, beating it does not mean beating no skill; each row shows both readings.
- What it does not show. It does not show how these skills do beyond three text tasks each, written by Driftproof for Report 015's twenty skills from the Skills.sh snapshot of 2026-10-17, on Claude Sonnet 5.5 judged by Claude Haiku 5.5.
Driftproof collected several answers per arm. A judge model scored each answer 5 times.
Claude Haiku 5.5Report 018 · Read the task result
Against the same-length fake
local (clearly higher): with the skill 0.908333 plus or minus 0.091742; with the same-length placebo 0.508333 plus or minus 0.08612.
Against nothing
local (clearly higher): with the skill 0.908333 plus or minus 0.091742; without it 0.525 plus or minus 0.125357.
- What we tested. Twenty skills on Haiku 5.5, each against no skill and against a same-length text from a fixed corpus, judged by Haiku 5.5, the same model.
- The judge is the target's model. The judge uses the same model as the target. Self-grading may favor its own model's answers.
- Beating the fake is not beating no skill. Where the fake scored below no skill, beating it does not mean beating no skill; each row shows both readings.
- What it does not show. It does not show how these skills do beyond three text tasks each, written by Driftproof for the twenty skills from the Skills.sh snapshot of 2026-10-17, on Claude Haiku 5.5 judged by Claude Haiku 5.5.
Driftproof collected several answers per arm. A judge model scored each answer 5 times.
The task
labels
Link to this taskWritten responses only. No tool execution, installed dependencies, media generation or cloud services are tested.
What we asked
Do not use tools, access the network, write files, run commands or install anything. Answer only in text using the supplied facts. The installed triage workflow uses bug:triage and bug:ready. A maintainer wants to retain those labels. Explain how to configure the canonical roles without creating duplicate labels.
The scoring guide
Pass threshold: 0.75.
Driftproof-authored from the pinned SKILL.md and README; no separate drafting model call.
Try it yourself
Paste the prompt into your own assistant, with or without the skill. Use the scoring guide to read the answer.
One try scored by hand is not a measurement.
Report 017: several answers per arm, each scored 5 times by a judge model. The report’s findings and caveats are below.
Report 018: several answers per arm, each scored 5 times by a judge model. The report’s findings and caveats are below.
Copying stays in your browser. Copies are never stored, counted or sent anywhere by Driftproof.
What happened on this task
Open a report to read its comparison and limits together. Beating the fake is not beating no skill.
Claude Sonnet 5.5Report 017 · Read the task result
Against the same-length fake
labels (too few answers to tell): with the skill 0.8 plus or minus 0.119024; with the same-length placebo 0.642857 plus or minus 0.136713.
Against nothing
labels (too few answers to tell): with the skill 0.8 plus or minus 0.119024; without it 0.671429 plus or minus 0.121988.
- What we tested. Report 015's twenty skills on Sonnet 5.5, each against no skill and against a same-length text from a fixed corpus, judged by Haiku 5.5.
- Beating the fake is not beating no skill. Where the fake scored below no skill, beating it does not mean beating no skill; each row shows both readings.
- What it does not show. It does not show how these skills do beyond three text tasks each, written by Driftproof for Report 015's twenty skills from the Skills.sh snapshot of 2026-10-17, on Claude Sonnet 5.5 judged by Claude Haiku 5.5.
Driftproof collected several answers per arm. A judge model scored each answer 5 times.
Claude Haiku 5.5Report 018 · Read the task result
Against the same-length fake
labels (clearly higher): with the skill 0.83 plus or minus 0.164317; with the same-length placebo 0.575 plus or minus 0.086603.
Against nothing
labels (too few answers to tell): with the skill 0.83 plus or minus 0.164317; without it 0.716667 plus or minus 0.028868.
- What we tested. Twenty skills on Haiku 5.5, each against no skill and against a same-length text from a fixed corpus, judged by Haiku 5.5, the same model.
- The judge is the target's model. The judge uses the same model as the target. Self-grading may favor its own model's answers.
- Beating the fake is not beating no skill. Where the fake scored below no skill, beating it does not mean beating no skill; each row shows both readings.
- What it does not show. It does not show how these skills do beyond three text tasks each, written by Driftproof for the twenty skills from the Skills.sh snapshot of 2026-10-17, on Claude Haiku 5.5 judged by Claude Haiku 5.5.
Driftproof collected several answers per arm. A judge model scored each answer 5 times.
The task
monorepo
Link to this taskWritten responses only. No tool execution, installed dependencies, media generation or cloud services are tested.
What we asked
Do not use tools, access the network, write files, run commands or install anything. Answer only in text using the supplied facts. A workspace has packages/orders and packages/catalog, each with its own glossary and ADR folder. Propose the domain-document layout decision and the confirmation needed.
The scoring guide
Pass threshold: 0.75.
Driftproof-authored from the pinned SKILL.md and README; no separate drafting model call.
Try it yourself
Paste the prompt into your own assistant, with or without the skill. Use the scoring guide to read the answer.
One try scored by hand is not a measurement.
Report 017: several answers per arm, each scored 5 times by a judge model. The report’s findings and caveats are below.
Report 018: several answers per arm, each scored 5 times by a judge model. The report’s findings and caveats are below.
Copying stays in your browser. Copies are never stored, counted or sent anywhere by Driftproof.
What happened on this task
Open a report to read its comparison and limits together. Beating the fake is not beating no skill.
Claude Sonnet 5.5Report 017 · Read the task result
Against the same-length fake
monorepo (too few answers to tell): with the skill 0.95 plus or minus 0.05; with the same-length placebo 0.835714 plus or minus 0.098802.
Against nothing
monorepo (too few answers to tell): with the skill 0.95 plus or minus 0.05; without it 0.75 plus or minus 0.192354.
- What we tested. Report 015's twenty skills on Sonnet 5.5, each against no skill and against a same-length text from a fixed corpus, judged by Haiku 5.5.
- Beating the fake is not beating no skill. Where the fake scored below no skill, beating it does not mean beating no skill; each row shows both readings.
- What it does not show. It does not show how these skills do beyond three text tasks each, written by Driftproof for Report 015's twenty skills from the Skills.sh snapshot of 2026-10-17, on Claude Sonnet 5.5 judged by Claude Haiku 5.5.
Driftproof collected several answers per arm. A judge model scored each answer 5 times.
Claude Haiku 5.5Report 018 · Read the task result
Against the same-length fake
monorepo (clearly higher): with the skill 0.833333 plus or minus 0.028868; with the same-length placebo 0.675 plus or minus 0.05.
Against nothing
monorepo (clearly higher): with the skill 0.833333 plus or minus 0.028868; without it 0.7 plus or minus 0.057735.
- What we tested. Twenty skills on Haiku 5.5, each against no skill and against a same-length text from a fixed corpus, judged by Haiku 5.5, the same model.
- The judge is the target's model. The judge uses the same model as the target. Self-grading may favor its own model's answers.
- Beating the fake is not beating no skill. Where the fake scored below no skill, beating it does not mean beating no skill; each row shows both readings.
- What it does not show. It does not show how these skills do beyond three text tasks each, written by Driftproof for the twenty skills from the Skills.sh snapshot of 2026-10-17, on Claude Haiku 5.5 judged by Claude Haiku 5.5.
Driftproof collected several answers per arm. A judge model scored each answer 5 times.