Skills.sh snapshot 2026-10-17: text tasks

What did Report 015 find?

All results 40 test records Clearly helped 24 Clearly hurt 6 Too few answers to tell 10 Skills.sh snapshot 2026-10-17: text tasks by Driftproof; Haiku 5.5 judge; eval account checked before/after runs, not during.

Sonnet 5.5 20 test records Clearly helped 12 Clearly hurt 4 Too few answers to tell 4

Opus 5.5 20 test records Clearly helped 12 Clearly hurt 2 Too few answers to tell 6

Examples: first by skill name, then model order, within each verdict state; add the first receipt for any answering model not yet shown. Selection within each group is outcome-blind, with no ranking by score or effect size.

On 18 Oct 2026, Claude Sonnet 5.5 with the agent-browser skill scored 0.954 on average, and 0.828 without it, over three test tasks: the skill clearly helped.
Three text-only tasks: Clearly helped.
Read test record
Why this result?
At least one task improved, with no task showing a clear drop.
The skill clearly helped: 0.954 with it and 0.828 without, on 3 test tasks.
dashboard: higher scores with the skill. Compare task scores
Task score bands: dashboard (separated-up): with skill 0.95 plus or minus 0.05; without skill 0.7 plus or minus 0.
A band is the average score plus or minus the recorded spread. These task bands, not the overall averages, determine the result.
0.954 with the skill and 0.828 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.
Three text-only tasks. Driftproof wrote the suite from the pinned skill description and README. These are not author-supplied tests. External tools, generated media and deployed services were not tested.
The eval account's configuration was read and found clean before and after each run, not during it.
This tests the router's written response; it does not test the delegated skill.
On 18 Oct 2026, Claude Opus 5.5 with the agent-browser skill scored 0.875 on average, and 0.876 without it, over three test tasks: there were too few answers to tell.
Three text-only tasks: Too few answers to tell.
Read test record
Why this result?
The answers varied too much to settle the result.
Too few answers to tell: 0.875 with the skill and 0.876 without, but the scores varied too much to call it.
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case web: the two arms' spreads sum to 0.094, at or above the 0.05 floor).
web, desktop, dashboard: too few answers to settle the result. Compare task scores
Task score bands: web (underpowered): with skill 0.966667 plus or minus 0.028868; without skill 0.94 plus or minus 0.065192; desktop (underpowered): with skill 0.733333 plus or minus 0.028868; without skill 0.861111 plus or minus 0.134112; dashboard (underpowered): with skill 0.925 plus or minus 0.086603; without skill 0.826667 plus or minus 0.040415.
A band is the average score plus or minus the recorded spread. These task bands, not the overall averages, determine the result.
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case web: the two arms' spreads sum to 0.094, at or above the 0.05 floor).
Three text-only tasks. Driftproof wrote the suite from the pinned skill description and README. These are not author-supplied tests. External tools, generated media and deployed services were not tested.
The eval account's configuration was read and found clean before and after each run, not during it.
This tests the router's written response; it does not test the delegated skill.
On 18 Oct 2026, Claude Sonnet 5.5 with the find-skills skill scored 0.853 on average, and 0.944 without it, over three test tasks: the skill clearly hurt.
Three text-only tasks: Clearly hurt.
Read test record
Why this result?
At least one task got worse. That takes priority over the overall average.
The skill clearly hurt: 0.853 with it and 0.944 without, on 3 test tasks.
search: lower scores with the skill. Compare task scores
Task score bands: search (separated-down): with skill 0.75 plus or minus 0; without skill 0.95 plus or minus 0.05.
A band is the average score plus or minus the recorded spread. These task bands, not the overall averages, determine the result.
0.853 with the skill and 0.944 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.
Three text-only tasks. Driftproof wrote the suite from the pinned skill description and README. These are not author-supplied tests. External tools, generated media and deployed services were not tested.
The eval account's configuration was read and found clean before and after each run, not during it.
This tests the router's written response; it does not test the delegated skill.
On 18 Oct 2026, Claude Sonnet 5.5 with the agent-browser skill was tested on three test tasks: it does not show how the skill does on other tasks, or with another grader.

Report 015: Capability-gap, claude-sonnet-5-5, claude-opus-5-5

Type Capability-gap report

What moved Skills.sh snapshot 2026-10-17: text tasks

Models claude-sonnet-5-5 claude-opus-5-5

Headline Skills.sh snapshot 2026-10-17: text tasks by Driftproof; Haiku 5.5 judge; eval account checked before/after runs, not during.

Receipts 40 linked receipts

Cite this

@techreport{driftproof-015,
  title  = {Skills.sh snapshot 2026-10-17: text tasks},
  author = {Driftproof},
  year   = {2026},
  type   = {Capability-gap report},
  note   = {claude-sonnet-5-5, claude-opus-5-5},
  url    = {https://driftproofhq.com/reports/015/}
}

Report 015. Recorded on 18 Oct 2026.

Capability-gap report. Two tiers on one provider surface; each receipt compares skill versus no skill within its answering model.

Skills.sh snapshot 2026-10-17: text tasks by Driftproof; Haiku 5.5 judge; eval account checked before/after runs, not during.

Sonnet 5.5 20 test records Clearly helped 12 Clearly hurt 4 Too few answers to tell 4

Opus 5.5 20 test records Clearly helped 12 Clearly hurt 2 Too few answers to tell 6

For the Skills.sh snapshot 2026-10-17: text tasks by Driftproof; Haiku 5.5 judge; eval account checked before/after runs, not during: Sonnet 5.5 12 helped, 4 hurt, 4 too few answers; Opus 5.5 12 helped, 2 hurt, 6 too few answers.

How we test a skill

A skill gives an AI extra instructions. We ask: do those instructions improve its written answers?

The task Same question. Same AI.
Driftproof writes tasks from the skill description.
Give the AI a task
The comparison Change the instructions
With the skill Written answer
Without the skill Written answer
Repeat both ways to see how much answers vary.
The result Judge each task
Haiku 5.5 scores the answers
A clear drop on any task takes priority. Otherwise, a better task can mean the skill helped. When answers vary too much, we cannot settle the result.

These tests cover written answers, not installation, tool use or how useful a skill is in general.

Method

The frozen Skills.sh all-time install snapshot selects the plain list, with the recorded tie rule and exclusions. Popularity is reported installs, not quality or unique people. Each pinned skill has three committed text tasks written by Driftproof from its own description and README. Suites were fixed before these runs.

Both model sets use Haiku 5.5 as judge.

The eval account's configuration was read and found clean before and after each run, not during it.

Each task has with-skill and baseline arms. The sampler starts at 3 generation draws and can extend to 10; each measured draw has 5 judge samples. It stops under the recorded spread and stability rule. Every retained judge output is parsed again before publication, and scores and aggregates must match the sealed receipt.

Each result is the reader's per-task reading, not a reading of the overall mean difference. A separated-down task takes precedence over a separated-up task. The table keeps each reader explanation and named task bands together. A point band may reflect the judge's score grid.

This table departs from the usual value-per-token layout: it reports independent receipt verdicts, with no new value ratio, cross-skill scale or model-versus-model causal claim. No deterministic post-checks were defined for these suites.

The complete comparison

20 skills, tested both ways

Both model sets use Haiku 5.5 as judge.

About these tests and their limits

Three text-only tasks. Driftproof wrote the suite from the pinned skill description and README. These are not author-supplied tests. External tools, generated media and deployed services were not tested.

The eval account's configuration was read and found clean before and after each run, not during it.

Limits

For find-skills, agent-browser, hyperframes and hyperframes-registry: This tests the router's written response; it does not test the delegated skill.

The small authored suites cover particular written tasks, not installation or general usefulness. The judge policy changed before this campaign to admit Haiku 5.5; the policy records the change and historical judges. Earlier reports keep their original judges. Unkeyed seals and retained outputs permit checking internal consistency; they do not authenticate an origin.

Run record

The campaigns ran in resumed segments. The earliest receipt date is 2026-10-18. No finish time is inferred. The surface was claude-cli; recorded costs are API-equivalent estimates, not a subscription invoice.

Campaign record

40 receipts from 40 frozen attempt identities across 5 campaigns. Every earlier failure remains in the record. Recorded spend: USD 44.303638, including the controller-observed USD 0.21 correction. USD two thousand five hundred admission budget; actual usage may overrun.

Complete attempt ledger, original manifests, plans and retry authorization. SHA-256 e1d7a154c182023ea2c7579e20c068c12be50c48cf4ef3dad61f90e6fd74643b

Campaign 1 (408): stopped on EVAL_CONFIG_CHANGED. Manifest 3c1a213c7c7f40e9abb6e4106a455f63fe8805131807090e5807d259b4b5cf57.
Campaign 2 (408-c2): stopped on EVAL_CONFIG_CHANGED. Manifest 4af9664915a3be2d567cddf191fb893a0db587eb177ec8cd040ee5c3bbb2d46b.
Campaign 3 (408-c3): stopped on EVAL_CONFIG_CHANGED. Manifest 4f899e0df590a66451613d1d64b756ac2481903ab1efdc906a4d1f99f1de0d19.
Campaign 4 (408-c4): completed with terminal ACTUAL_COUNTS failure. Manifest 33e4d1cbd6585f68081bf24aad056d3cee2c4361576a5f80ffa700c9d378f707.
Campaign 5 (408-c5): operator-authorized retry. Manifest e87e021c8d278d23f795e634bc08c7dddcf3170c2efe2abef7f0ea75cbb3e423.

2026-10-18; Claude Sonnet 5.5; Claude Opus 5.5

Evidence

Popularity snapshot · Pinned subjects and suites · All manifests, failures, final states and original plans. Every table cell links its receipt, suite and retained answers.