Driftproof

Report 016 · 7 rules files · 2 models · 2,646 coding runs · outcomes chosen 27 Oct, before any score was read

Popular rules files vs. a placebo: most made Claude write more

What did Report 016 find?

  • What we tested. These small tasks test injected instructions on two Claude models, not harder work or native project loading.
  • What we found. On Haiku, six made it write more than the fakes did; Sonnet was mixed. None made code clearly longer.
  • What it does not show. We counted what changed, not whether the extra writing helped.

We gave Claude seven popular rules files and same-length fakes. On Haiku, six made it write more than the fakes did; Sonnet was mixed. None made code clearly longer.

Driftproof · 2026-10-28

Rules-file behaviour report

These small tasks test injected instructions on two Claude models, not harder work or native project loading.

See what changed ↓See the receipts ↓
The fileIts fake352 bytes
## Approach
- Read existing files before writing. Don't re-read unless changed.
- Thorough in reasoning, concise in output.
- Skip files over 100KB unless required.
- No sycophantic openers or closing fluff.
- No emojis or em-dashes.
- Do not guess APIs, versions, flags, commit SHAs, or package names. Verify by reading code or docs before asserting.

became

The morning light settled slowly over the hills beyond the old market town. The ferry crosses the sound twice a day when the weather is calm enough. On clear nights the stars over the moor are bright enough to cast faint shadows. The museum displays a loom that weavers in the district used for linen and wool.

Fishing boats leave before dawn and retu
The shortest of the files, drona23/claude-token-efficient (352 bytes), becoming its fake: same length to the byte, about hills, a ferry and fishing boats, with nothing about code. See your file's fake.

A rules file is a note an AI coding assistant reads before it starts work.

We picked the seven most-starred files the rule picked: four CLAUDE.md, and the only three AGENTS.md files in the search that met the rule.

  • 222,760andrej-karpathy-skillsCLAUDE.md
  • 174,414ponytailAGENTS.md
  • 23,078vibe-coding-cnCLAUDE.md
  • 21,200learn-harness-engineering CLAUDECLAUDE.md
  • 21,200learn-harness-engineering AGENTSAGENTS.md
  • 6,110claude-token-efficientCLAUDE.md
  • 2,306fable-methodAGENTS.md

Repository stars on 2026-10-23. The Karpathy-named file was not written by Andrej Karpathy. This was a capped search, not a census of GitHub.

Then we ran Claude 2,646 times.

Claude Haiku 5.5 and Sonnet 5.5 each did 20 small coding tasks three ways: with nothing, with the file, with its fake. Hidden tests checked the code. 2,646 real Claude Code runs in all.

NothingNo added file
## Approach
- Read existing files before writing. Don't re-read unless changed.
- Thorough in reasoning, concise in output.
- Skip files over 100KB unless required.
- No sycophantic openers or closing fluff.
- No emojis or em-dashes.
- Do not guess APIs, versions, flags, commit SHAs, or package names. Verify by reading code or docs before asserting.
The fileInjected instructions

The morning light settled slowly over the hills beyond the old market town. The ferry crosses the sound twice a day when the weather is calm enough. On clear nights the stars over the moor are bright enough to cast faint shadows. The museum displays a loom that weavers in the district used for linen and wool. Fishing boats leave before dawn and retu

Its fakeIrrelevant prose, equal bytes

First surprise: the code was already right.

Both models passed nearly every hidden test, with or without a file. In 246 of 266 task-and-job pairs, every measured run in every arm was perfect. A file can’t fix code that isn’t broken.

246of 266 tested task-and-job pairs
Solid squares: perfect in every measured run in every arm. Hollow squares: something slipped. All task-level pass rates and lost draws remain in appendix A.

3 exceptions, each one task with few runs: ponytail helped Sonnet on utc-day-python; two Haiku cells separated only because the fake’s runs all failed. Appendix A retains the reader’s refusals and exact cases.

On Haiku, 6 of 7 files made Claude write more.

Between 13% and 40% more than with the fake file. Only ponytail, which asks for the least new code, didn’t clearly change it.

The fake / The file. Mean output tokens per task.

andrej-karpathy-skills+13%
vibe-coding-cn+29%
learn-harness-engineering CLAUDE+20%
claude-token-efficient+20%
ponytailCan’t tell (-5%)
learn-harness-engineering AGENTS+24%
fable-method+40%
09,000
“Write” means recorded output tokens across the session, including code saved to files. Each bar compares the file with its same-length fake, with a 95% interval. The recorded usage fields do not establish whether output tokens include thinking tokens.

On Sonnet, it went both ways.

3 files made Sonnet write more, 2 made it write less, and 2 showed no clear change.

The fake / The file. Mean output tokens per task.

andrej-karpathy-skills-6%
vibe-coding-cn+32%
learn-harness-engineering CLAUDECan’t tell (+7%)
claude-token-efficient+11%
ponytail-24%
learn-harness-engineering AGENTSCan’t tell (+2%)
fable-method+14%
03,000
“Write” means recorded output tokens across the session, including code saved to files. Each bar compares the file with its same-length fake, with a 95% interval. The recorded usage fields do not establish whether output tokens include thinking tokens.

The two files that ask for less code got it.

No file made the code clearly longer. The two that cut it on both models are the two that ask for it: ponytail and the Karpathy-named file. A pattern, not proof.

andrej-karpathy-skills
Haiku
22.8 → 20.5 lines
ponytail
Haiku
22.8 → 17.1 lines
andrej-karpathy-skills
Sonnet
24.4 → 20.4 lines
ponytail
Sonnet
22.2 → 14.8 lines

Mean lines in the solution file, fake vs file. Shapes are drawn; counts are measured. The Karpathy-named file on Sonnet is shorter than with the fake, but not clearly shorter than with no file.

Same file, different model, opposite result.

The Karpathy-named file made Haiku write more and Sonnet less. A rules file doesn’t have one effect. It has one per model, so test it on yours.

+13%Haiku669.6 output tokens; interval 315.3 to 1,046.4
-6%Sonnet-134.8 output tokens; interval -254.0 to -27.0

Named after Andrej Karpathy; he didn’t write it. Sonnet’s drop is clear against the fake, not against no file.

Two templates sent Claude hunting for files that weren’t there.

They tell Claude to start each session by reading a progress log and running a setup script. Our test project had neither. Claude went looking in 252 of 263 runs.

Recorded setup lookups, counted once per run
pwd && ls -la /workspace && ls -la /workspace/src 2>/dev/null; cat /workspace/claude-progress.md 2>/dev/null; cat /workspace/feature_list.json 2>/dev/null; cat /workspace/init.sh 2>/dev/null
252 of 263went looking
No lookup was observed with the other files. 2 file-arm runs have incomplete tool evidence. We can’t say how much of the extra writing this explains. Copying a template copies its assumptions.

What this doesn’t show

  • Harder work. These 20 tasks were small, and the code was already right.
  • Your setup. Each file was added to Claude’s instructions, not loaded from a project folder.
  • Other models. Only Claude Haiku 5.5 and Sonnet 5.5. No Opus.
  • Whether it was worth it. We counted what changed, not whether the extra writing helped.

What each file asks for, and what changed

Each file’s own words beside what changed against its fake. “Can’t tell” means the 95% interval includes zero, not “no effect”.

What each file asks for, and what changedAgainst its same-length fake. “Can’t tell” means the interval includes zero.
FileIts own wordsHaiku outputHaiku codeSonnet outputSonnet code
andrej-karpathy-skillsCLAUDE.md“Minimum code that solves the problem. Nothing speculative.”13% morefewer6% fewerfewer
vibe-coding-cnCLAUDE.md启用「ultrathink」深度思考
“Enable ‘ultrathink’ deep thinking” (machine-translated)
29% moreCan't tell32% moreCan't tell
learn-harness-engineering CLAUDECLAUDE.md“Prioritize reliable completion, continuity across sessions, and explicit verification over speed.”20% moreCan't tellCan't tellfewer
claude-token-efficientCLAUDE.md“Thorough in reasoning, concise in output.”20% moreCan't tell11% moreCan't tell
ponytailAGENTS.md“You solve the whole problem with the least new code.”Can't tellfewer24% fewerfewer
learn-harness-engineering AGENTSAGENTS.md“The goal is not to maximize raw code output.”24% moreCan't tellCan't tellfewer
fable-methodAGENTS.md“the quality lives in the structure, the evidence, and the honesty”40% moreCan't tell14% moreCan't tell
Output means recorded output tokens; code means nonblank, noncomment solution lines. Two-sided paired task bootstrap intervals. The same analysis builds this page and its share images.

Both templates. The progress log and setup script they expect were absent.

Karpathy-named file. Its Haiku correctness receipt is incomplete; the behaviour analysis uses only eligible measured tasks.

vibe-coding-cn. The quote above keeps the original Chinese and a literal machine translation.

Tests and asking. No clear change in assertion-bearing test files was detected. Asking is unavailable where no solution exists because the evidence bundle has no hash-verified final reply. Missing solutions are reported separately.

Want your rules file tested?

Open an issue or a discussion with a public CLAUDE.md or AGENTS.md. We pick which files to run and can’t run them all. Every file we run gets published with its receipt.

Check it yourself

Each of the 14 jobs has a JSON receipt sealed by its own SHA-256. The files are pinned by commit and hash, and each fake can be rebuilt from its seed. Seals check consistency; they do not authenticate the origin.

Download the offline analysis script · Evidence manifest · Hash-bound workspace snapshots · Download the table image

node scripts/analyze-report-016.js --out analysis.json

Run from the Driftproof 0.24.0 source checkout. This reads the published receipts and snapshots without model calls. These runs used unreleased source. Appendix E lists each runner commit. The sandboxed hidden-test setup needs an operator setup.

How we ran it

File, no-file and placebo arms share the frozen tasks, starter workspace, runtime image and sampling rules within each job. Instructions were injected into Claude Code. The placebo matches bytes, not input tokens or meaning. TypeScript is judge-only and excluded from the correctness ceiling; eligible TypeScript draws remain in behaviour summaries.

Behaviour uses equal task weights after averaging eligible draws within each arm. A task must have observations in all arms for that metric. Missing values stay missing. Paired task differences use 4000 bootstrap samples with fixed seed 501. Percentage intervals divide by the comparator’s fixed task mean. Zero-touching intervals mean no clear change. These are exploratory comparisons without multiple-comparison adjustment.

Lines exclude blank lines and language comments in the required solution file. Tests require a new or changed test file containing an assertion, compared by hash with the starter. Asking requires a question in the final reply and no solution; absence alone is not asking.

Outcomes note: outcomes chosen 27 Oct, before any score was read

SHA-256 9ea1330142b80a69650c38e541fff304633baefcc071a1fb9638de0ba226e2d5

Selection used the fixed popularity rule and a capped API search. Only three AGENTS.md files met it. Maverick approved these seven on 23 Oct. The surviving notes reconstruct the query and disclose incomplete or truncated tree reads; this is not an exhaustive GitHub ranking.

Selection notes and search limits

Prior work

Others have tested context files. Gloaguen et al., Evaluating AGENTS.md found that they do not generally improve task success and raise inference cost by over 20%. Khatri’s context-injection study found no measurable correctness change across 288 runs on 17 real tasks. This report adds a same-length placebo, popular third-party files people copy, and behaviour outcomes chosen before scoring.

Both abstracts were checked against their source for this build. Author responses can be added as dated amendments after outreach.

Appendix A: correctness, every task and every reader result
FileModelTaskNothing: pass rate / measured / lostFile: pass rate / measured / lostFake: pass rate / measured / lostPerfect in all measured arms
claude-1Haikustable-dedup-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Haikuparse-integer-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Haikumerge-intervals-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Haikuttl-cache-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Haikubounded-map-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Haikuretry-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Haikutopological-sort-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Haikusql-parameters-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Haikucsv-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Haikupagination-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Haikudeep-merge-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Haikumoney-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Haikuutc-day-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Haikuutf8-stream-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Haikuversion-order-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Haikuevent-emitter-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Haikulatest-row-sql100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Haikubinary-search-js80.0% / 10 / 050.0% / 4 / 050.0% / 6 / 0no
claude-1Haikusafe-path-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Haikustable-dedup-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Haikuparse-integer-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Haikumerge-intervals-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Haikuttl-cache-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Haikubounded-map-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Haikuretry-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Haikutopological-sort-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Haikusql-parameters-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Haikucsv-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Haikupagination-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Haikudeep-merge-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Haikumoney-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Haikuutc-day-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Haikuutf8-stream-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Haikuversion-order-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Haikuevent-emitter-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Haikulatest-row-sql100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Haikubinary-search-js0.0% / 3 / 050.0% / 4 / 050.0% / 4 / 0no
claude-2Haikusafe-path-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Haikustable-dedup-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Haikuparse-integer-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Haikumerge-intervals-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Haikuttl-cache-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Haikubounded-map-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Haikuretry-js100.0% / 3 / 095.8% / 6 / 0100.0% / 3 / 0no
claude-3Haikutopological-sort-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Haikusql-parameters-python100.0% / 3 / 090.0% / 10 / 0100.0% / 3 / 0no
claude-3Haikucsv-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Haikupagination-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Haikudeep-merge-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Haikumoney-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Haikuutc-day-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Haikuutf8-stream-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Haikuversion-order-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Haikuevent-emitter-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Haikulatest-row-sql100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Haikubinary-search-js70.0% / 10 / 090.0% / 10 / 00.0% / 3 / 0no
claude-3Haikusafe-path-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Haikustable-dedup-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Haikuparse-integer-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Haikumerge-intervals-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Haikuttl-cache-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Haikubounded-map-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Haikuretry-js95.8% / 6 / 087.5% / 4 / 087.5% / 4 / 0no
claude-4Haikutopological-sort-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Haikusql-parameters-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Haikucsv-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Haikupagination-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Haikudeep-merge-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Haikumoney-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Haikuutc-day-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Haikuutf8-stream-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Haikuversion-order-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Haikuevent-emitter-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Haikulatest-row-sql100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Haikubinary-search-js50.0% / 4 / 050.0% / 4 / 00.0% / 3 / 0no
claude-4Haikusafe-path-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Haikustable-dedup-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Haikuparse-integer-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Haikumerge-intervals-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Haikuttl-cache-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Haikubounded-map-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Haikuretry-js95.8% / 6 / 0100.0% / 3 / 087.5% / 6 / 0no
agents-1Haikutopological-sort-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Haikusql-parameters-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Haikucsv-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Haikupagination-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Haikudeep-merge-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Haikumoney-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Haikuutc-day-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Haikuutf8-stream-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Haikuversion-order-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Haikuevent-emitter-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Haikulatest-row-sql100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Haikubinary-search-js100.0% / 3 / 0100.0% / 3 / 00.0% / 3 / 0no
agents-1Haikusafe-path-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Haikustable-dedup-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Haikuparse-integer-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Haikumerge-intervals-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Haikuttl-cache-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Haikubounded-map-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Haikuretry-js100.0% / 3 / 0100.0% / 3 / 091.7% / 6 / 0no
agents-2Haikutopological-sort-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Haikusql-parameters-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Haikucsv-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Haikupagination-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Haikudeep-merge-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Haikumoney-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Haikuutc-day-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Haikuutf8-stream-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Haikuversion-order-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Haikuevent-emitter-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Haikulatest-row-sql100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Haikubinary-search-js55.6% / 9 / 0100.0% / 3 / 0100.0% / 3 / 0no
agents-2Haikusafe-path-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Haikustable-dedup-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Haikuparse-integer-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Haikumerge-intervals-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Haikuttl-cache-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Haikubounded-map-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Haikuretry-js100.0% / 3 / 0100.0% / 3 / 091.7% / 6 / 0no
agents-3Haikutopological-sort-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Haikusql-parameters-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Haikucsv-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Haikupagination-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Haikudeep-merge-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Haikumoney-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Haikuutc-day-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Haikuutf8-stream-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Haikuversion-order-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Haikuevent-emitter-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Haikulatest-row-sql100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Haikubinary-search-js50.0% / 4 / 0100.0% / 3 / 050.0% / 4 / 0no
agents-3Haikusafe-path-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Sonnetstable-dedup-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Sonnetparse-integer-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Sonnetmerge-intervals-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Sonnetttl-cache-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Sonnetbounded-map-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Sonnetretry-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Sonnettopological-sort-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Sonnetsql-parameters-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Sonnetcsv-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Sonnetpagination-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Sonnetdeep-merge-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Sonnetmoney-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Sonnetutc-day-python75.0% / 3 / 075.0% / 3 / 075.0% / 3 / 0no
claude-1Sonnetutf8-stream-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Sonnetversion-order-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Sonnetevent-emitter-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Sonnetlatest-row-sql100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Sonnetbinary-search-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-1Sonnetsafe-path-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Sonnetstable-dedup-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Sonnetparse-integer-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Sonnetmerge-intervals-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Sonnetttl-cache-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Sonnetbounded-map-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Sonnetretry-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Sonnettopological-sort-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Sonnetsql-parameters-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Sonnetcsv-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Sonnetpagination-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Sonnetdeep-merge-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Sonnetmoney-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Sonnetutc-day-python75.0% / 3 / 087.5% / 4 / 087.5% / 4 / 0no
claude-2Sonnetutf8-stream-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Sonnetversion-order-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Sonnetevent-emitter-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Sonnetlatest-row-sql100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Sonnetbinary-search-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-2Sonnetsafe-path-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Sonnetstable-dedup-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Sonnetparse-integer-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Sonnetmerge-intervals-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Sonnetttl-cache-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Sonnetbounded-map-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Sonnetretry-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Sonnettopological-sort-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Sonnetsql-parameters-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Sonnetcsv-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Sonnetpagination-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Sonnetdeep-merge-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Sonnetmoney-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Sonnetutc-day-python75.0% / 3 / 075.0% / 3 / 075.0% / 3 / 0no
claude-3Sonnetutf8-stream-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Sonnetversion-order-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Sonnetevent-emitter-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Sonnetlatest-row-sql100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Sonnetbinary-search-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-3Sonnetsafe-path-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Sonnetstable-dedup-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Sonnetparse-integer-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Sonnetmerge-intervals-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Sonnetttl-cache-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Sonnetbounded-map-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Sonnetretry-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Sonnettopological-sort-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Sonnetsql-parameters-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Sonnetcsv-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Sonnetpagination-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Sonnetdeep-merge-js100.0% / 3 / 0100.0% / 3 / 1100.0% / 3 / 0yes
claude-4Sonnetmoney-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Sonnetutc-day-python79.2% / 6 / 075.0% / 3 / 075.0% / 3 / 0no
claude-4Sonnetutf8-stream-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Sonnetversion-order-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Sonnetevent-emitter-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Sonnetlatest-row-sql100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Sonnetbinary-search-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
claude-4Sonnetsafe-path-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Sonnetstable-dedup-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Sonnetparse-integer-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Sonnetmerge-intervals-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Sonnetttl-cache-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Sonnetbounded-map-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Sonnetretry-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Sonnettopological-sort-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Sonnetsql-parameters-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Sonnetcsv-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Sonnetpagination-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Sonnetdeep-merge-js100.0% / 3 / 1100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Sonnetmoney-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Sonnetutc-day-python75.0% / 3 / 095.8% / 6 / 075.0% / 3 / 0no
agents-1Sonnetutf8-stream-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Sonnetversion-order-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Sonnetevent-emitter-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Sonnetlatest-row-sql100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Sonnetbinary-search-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-1Sonnetsafe-path-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Sonnetstable-dedup-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Sonnetparse-integer-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Sonnetmerge-intervals-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Sonnetttl-cache-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Sonnetbounded-map-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Sonnetretry-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Sonnettopological-sort-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Sonnetsql-parameters-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Sonnetcsv-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Sonnetpagination-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Sonnetdeep-merge-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Sonnetmoney-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Sonnetutc-day-python75.0% / 3 / 075.0% / 3 / 075.0% / 3 / 0no
agents-2Sonnetutf8-stream-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Sonnetversion-order-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Sonnetevent-emitter-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Sonnetlatest-row-sql100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Sonnetbinary-search-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-2Sonnetsafe-path-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Sonnetstable-dedup-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Sonnetparse-integer-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Sonnetmerge-intervals-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Sonnetttl-cache-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Sonnetbounded-map-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Sonnetretry-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Sonnettopological-sort-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Sonnetsql-parameters-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Sonnetcsv-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Sonnetpagination-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Sonnetdeep-merge-js100.0% / 3 / 0100.0% / 3 / 1100.0% / 3 / 0yes
agents-3Sonnetmoney-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Sonnetutc-day-python75.0% / 3 / 079.2% / 6 / 075.0% / 3 / 0no
agents-3Sonnetutf8-stream-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Sonnetversion-order-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Sonnetevent-emitter-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Sonnetlatest-row-sql100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Sonnetbinary-search-js100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
agents-3Sonnetsafe-path-python100.0% / 3 / 0100.0% / 3 / 0100.0% / 3 / 0yes
FileModelAgainst nothingAgainst fakeSeparated vs nothingSeparated vs fake
claude-1HaikuNOT_MEASUREDNOT_MEASUREDnonenone
claude-2HaikuUNDERPOWEREDUNDERPOWEREDnonenone
claude-3HaikuUNDERPOWEREDABOVE_PLACEBOnonebinary-search-js: separated-up
claude-4HaikuUNDERPOWEREDUNDERPOWEREDnonenone
agents-1HaikuUNDERPOWEREDABOVE_PLACEBOnonebinary-search-js: separated-up
agents-2HaikuUNDERPOWEREDUNDERPOWEREDnonenone
agents-3HaikuUNDERPOWEREDUNDERPOWEREDnonenone
claude-1SonnetUNDERPOWEREDUNDERPOWEREDnonenone
claude-2SonnetUNDERPOWEREDUNDERPOWEREDnonenone
claude-3SonnetUNDERPOWEREDUNDERPOWEREDnonenone
claude-4SonnetUNDERPOWEREDUNDERPOWEREDnonenone
agents-1SonnetPASSEDABOVE_PLACEBOutc-day-python: separated-uputc-day-python: separated-up
agents-2SonnetUNDERPOWEREDUNDERPOWEREDnonenone
agents-3SonnetUNDERPOWEREDUNDERPOWEREDnonenone

The full library readings, including underpowered cases, lost-draw refusals and draw requirements, are in the analysis output. “Not measured” for the incomplete Haiku Karpathy-named job is retained.

Appendix B: behaviour and paired intervals

File-to-fake output comparisons: 6 haiku increases · 1 haiku unclear · 3 sonnet increases · 2 sonnet decreases · 2 sonnet unclear. These are behaviour changes, not correctness verdicts.

FileModelMetricTasksNothingFileFakeFile minus fake (95% interval)File minus nothing (95% interval)
claude-1Haikuout195,471.8565,955.5665,285.956669.610 (315.289, 1,046.439)483.710 (215.823, 769.596)
claude-1Haikuloc1923.71920.54422.754-2.211 (-4.439, -0.404)-3.175 (-6.211, -0.912)
claude-1Haikutests190.9370.9210.9210.000 (0.000, 0.000)-0.016 (-0.047, 0.000)
claude-1Haikumissing_solution190.0110.0260.0260.000 (0.000, 0.000)0.016 (0.000, 0.047)
claude-1Haikutool_calls196.1006.5486.2460.303 (-0.232, 0.934)0.448 (-0.086, 0.965)
claude-2Haikuout205,334.7006,808.9085,290.4041,518.504 (964.617, 2,154.100)1,474.208 (982.392, 1,988.542)
claude-2Haikuloc1922.82521.31622.456-1.140 (-3.947, 1.018)-1.509 (-4.877, 1.158)
claude-2Haikutests200.9000.9250.9250.000 (0.000, 0.000)0.025 (0.000, 0.075)
claude-2Haikumissing_solution200.0500.0250.0250.000 (0.000, 0.000)-0.025 (-0.075, 0.000)
claude-2Haikutool_calls206.1926.6086.5830.025 (-0.367, 0.458)0.417 (-0.133, 1.008)
claude-3Haikuout205,556.9036,738.2745,593.1231,145.151 (694.150, 1,578.681)1,181.371 (683.700, 1,765.696)
claude-3Haikuloc1923.07022.85123.758-0.907 (-3.035, 0.449)-0.219 (-2.596, 1.544)
claude-3Haikutests200.9350.9570.9000.057 (-0.010, 0.168)0.022 (-0.010, 0.065)
claude-3Haikumissing_solution200.0150.0100.050-0.040 (-0.135, 0.015)-0.005 (-0.030, 0.015)
claude-3Haikutool_calls206.2777.7056.2331.472 (0.822, 2.317)1.428 (0.883, 2.015)
claude-4Haikuout205,674.0086,371.6485,295.0711,076.578 (411.875, 1,860.942)697.640 (127.507, 1,286.940)
claude-4Haikuloc1922.98223.47622.8640.612 (-1.982, 2.672)0.494 (-1.246, 2.488)
claude-4Haikutests200.9250.9250.9000.025 (0.000, 0.075)0.000 (0.000, 0.000)
claude-4Haikumissing_solution200.0250.0250.050-0.025 (-0.075, 0.000)0.000 (0.000, 0.000)
claude-4Haikutool_calls206.5427.2986.3500.948 (0.132, 1.853)0.757 (0.175, 1.382)
agents-1Haikuout205,651.5295,061.5675,330.992-269.425 (-645.842, 143.967)-589.963 (-1,088.800, -94.529)
agents-1Haikuloc1922.18917.07022.772-5.702 (-9.614, -3.105)-5.118 (-8.526, -3.035)
agents-1Haikutests200.9500.9500.9000.050 (0.000, 0.150)0.000 (0.000, 0.000)
agents-1Haikumissing_solution200.0000.0000.050-0.050 (-0.150, 0.000)0.000 (0.000, 0.000)
agents-1Haikutool_calls206.2256.1676.0170.150 (-0.433, 1.000)-0.058 (-0.383, 0.300)
agents-2Haikuout205,486.6716,672.9835,396.9751,276.008 (727.675, 1,848.233)1,186.313 (705.733, 1,757.396)
agents-2Haikuloc2021.89122.35021.4750.875 (-0.858, 3.308)0.459 (-1.741, 3.117)
agents-2Haikutests200.9280.9500.950-0.000 (-0.050, 0.050)0.022 (-0.033, 0.089)
agents-2Haikumissing_solution200.0220.0000.0000.000 (0.000, 0.000)-0.022 (-0.067, 0.000)
agents-2Haikutool_calls206.3476.9176.7330.183 (-0.350, 0.667)0.569 (-0.017, 1.192)
agents-3Haikuout205,597.3297,756.3335,526.2082,230.125 (1,597.025, 2,960.075)2,159.004 (1,555.450, 2,836.612)
agents-3Haikuloc2021.79221.85021.888-0.038 (-1.108, 1.229)0.058 (-1.017, 1.417)
agents-3Haikutests200.9250.9500.9250.025 (0.000, 0.075)0.025 (0.000, 0.075)
agents-3Haikumissing_solution200.0250.0000.025-0.025 (-0.075, 0.000)-0.025 (-0.075, 0.000)
agents-3Haikutool_calls206.4468.0506.3921.658 (1.038, 2.308)1.604 (0.900, 2.304)
claude-1Sonnetout202,085.0502,034.3002,169.117-134.817 (-254.000, -27.000)-50.750 (-163.733, 61.517)
claude-1Sonnetloc2021.31720.43324.367-3.933 (-7.600, -1.567)-0.883 (-2.200, 0.750)
claude-1Sonnettests200.9500.9500.9500.000 (0.000, 0.000)0.000 (0.000, 0.000)
claude-1Sonnetmissing_solution200.0000.0000.0000.000 (0.000, 0.000)0.000 (0.000, 0.000)
claude-1Sonnettool_calls203.2502.9003.583-0.683 (-1.183, -0.217)-0.350 (-0.700, 0.000)
claude-2Sonnetout202,094.2832,768.8252,096.696672.129 (482.550, 872.029)674.542 (505.608, 861.633)
claude-2Sonnetloc2022.30021.05421.167-0.113 (-1.904, 2.017)-1.246 (-3.679, 1.254)
claude-2Sonnettests200.9500.9500.9500.000 (0.000, 0.000)0.000 (0.000, 0.000)
claude-2Sonnetmissing_solution200.0000.0000.0000.000 (0.000, 0.000)0.000 (0.000, 0.000)
claude-2Sonnettool_calls203.2834.1713.2500.921 (0.429, 1.404)0.888 (0.433, 1.358)
claude-3Sonnetout202,081.0332,330.5502,178.483152.067 (-43.400, 343.950)249.517 (120.783, 389.333)
claude-3Sonnetloc2021.45021.11722.783-1.667 (-3.067, -0.517)-0.333 (-1.267, 0.683)
claude-3Sonnettests200.9500.9830.9500.033 (0.000, 0.100)0.033 (0.000, 0.100)
claude-3Sonnetmissing_solution200.0000.0000.0000.000 (0.000, 0.000)0.000 (0.000, 0.000)
claude-3Sonnettool_calls203.0674.1333.3830.750 (0.250, 1.267)1.067 (0.583, 1.617)
claude-4Sonnetout202,185.3502,382.2002,151.167231.033 (78.850, 433.883)196.850 (11.133, 412.867)
claude-4Sonnetloc2022.04224.11721.6832.433 (-0.283, 6.083)2.075 (-0.167, 5.450)
claude-4Sonnettests200.9500.9500.9500.000 (0.000, 0.000)0.000 (0.000, 0.000)
claude-4Sonnetmissing_solution200.0000.0000.0000.000 (0.000, 0.000)0.000 (0.000, 0.000)
claude-4Sonnettool_calls203.3923.9173.4500.467 (-0.017, 0.950)0.525 (-0.083, 1.150)
agents-1Sonnetout202,126.1171,648.4212,160.475-512.054 (-662.750, -378.433)-477.696 (-595.996, -358.029)
agents-1Sonnetloc2022.15014.85022.158-7.308 (-10.375, -4.742)-7.300 (-10.100, -4.850)
agents-1Sonnettests200.9500.9500.9500.000 (0.000, 0.000)0.000 (0.000, 0.000)
agents-1Sonnetmissing_solution200.0000.0000.0000.000 (0.000, 0.000)0.000 (0.000, 0.000)
agents-1Sonnettool_calls203.3503.0213.604-0.583 (-0.950, -0.217)-0.329 (-0.721, 0.050)
agents-2Sonnetout202,130.0372,221.4502,179.65041.800 (-91.833, 175.800)91.413 (-68.933, 227.567)
agents-2Sonnetloc2022.72520.73323.533-2.800 (-6.217, -0.283)-1.992 (-5.717, 0.892)
agents-2Sonnettests200.9500.9830.9500.033 (0.000, 0.100)0.033 (0.000, 0.100)
agents-2Sonnetmissing_solution200.0000.0000.0000.000 (0.000, 0.000)0.000 (0.000, 0.000)
agents-2Sonnettool_calls203.5044.0673.3500.717 (0.283, 1.200)0.562 (0.125, 1.042)
agents-3Sonnetout202,133.7672,407.5332,110.408297.125 (168.083, 433.475)273.767 (111.450, 432.067)
agents-3Sonnetloc2022.13322.64221.4921.150 (-0.533, 3.600)0.508 (-0.825, 2.025)
agents-3Sonnettests200.9500.9500.9500.000 (0.000, 0.000)0.000 (0.000, 0.000)
agents-3Sonnetmissing_solution200.0000.0000.0000.000 (0.000, 0.000)0.000 (0.000, 0.000)
agents-3Sonnettool_calls203.2003.8173.4790.338 (0.025, 0.683)0.617 (0.233, 0.967)

Tests and missing solutions are task means of binary draw outcomes. The published analysis includes the exact task IDs, eligible and missing draw counts, excluded tasks, and unrounded intervals for each metric.

Appendix C: losses and setup searches

Of 2,646 drawn runs, 58 had a primary or secondary loss: 3 primary execution failures, 32 judge-only primary losses, and 23 secondary-only losses. A draw with both losses is counted once. These losses do not establish their cause.

FileModelExecutionJudge-only primarySecondary-onlyAll affectedBaseline setup lookups / all / unknownFile setup lookups / all / unknownPlacebo setup lookups / all / unknown
claude-1Haiku04260 / 67 / 00 / 63 / 00 / 63 / 0
claude-2Haiku02020 / 63 / 00 / 62 / 00 / 62 / 0
claude-3Haiku03470 / 70 / 075 / 79 / 00 / 62 / 0
claude-4Haiku00110 / 64 / 00 / 64 / 00 / 61 / 0
agents-1Haiku03140 / 66 / 00 / 61 / 00 / 63 / 0
agents-2Haiku04260 / 68 / 055 / 62 / 00 / 64 / 0
agents-3Haiku02240 / 63 / 00 / 60 / 00 / 66 / 0
claude-1Sonnet04370 / 62 / 00 / 60 / 00 / 62 / 0
claude-2Sonnet01010 / 60 / 00 / 62 / 00 / 62 / 0
claude-3Sonnet01120 / 60 / 061 / 61 / 00 / 60 / 0
claude-4Sonnet14160 / 67 / 00 / 61 / 10 / 60 / 0
agents-1Sonnet11570 / 61 / 10 / 65 / 00 / 61 / 0
agents-2Sonnet02020 / 61 / 061 / 61 / 00 / 61 / 0
agents-3Sonnet11130 / 60 / 00 / 64 / 10 / 62 / 0

Setup searches count actual Bash, Read, Glob or Grep inputs for the named progress, feature, setup and handoff files or git log, including git with a working-directory option. Descriptions and tool-result text do not count. Lost runs with complete tool evidence remain in the denominator; incomplete tool evidence is unknown.

Receipts

Appendix D: receipts and pinned subjects
FilePinned sourceStarsBytesSHA-256
claude-1multica-ai/andrej-karpathy-skills/CLAUDE.md222,7602357694a2d721e41c385f3db492838c23299826df5ba9809e3b0721aac70021e196a
claude-22025Emma/vibe-coding-cn/i18n/zh/prompts/system_prompts/CLAUDE.md/10/CLAUDE.md23,0782121712dc0451a147a87a93e43de16535c0d956f66e838014067307b10d2619507422
claude-3walkinglabs/learn-harness-engineering/docs/en/resources/templates/CLAUDE.md21,2001408d1730a3ba184aa9f0b12c25aebdba89712dac5009bc0b1b44b4d355dff40355a
claude-4drona23/claude-token-efficient/CLAUDE.md6,1103526656125f2b4f777d9cb4149b2acc3b0281aab4e8d8ebdd4ddcdb1ae9274e8253
agents-1DietrichGebert/ponytail/AGENTS.md174,4142721b98b15e1d683ee205e124b52b8d330128593f4a87b545cf8185e2712cd1bb25f
agents-2walkinglabs/learn-harness-engineering/docs/en/resources/templates/AGENTS.md21,200194942d2ed896cc6db9e459d9f59e2e39af2ceba664a40f092544f1b34678a90aa58
agents-3Sahir619/fable-method/AGENTS.md2,306164868237d3aea88700d5ec3051adc2a0641a9fe68ddca9e96dbef6c1fc6473c86b3c
Appendix E: execution conditions and usage

Execution settings come from the sealed receipts. Runner commits come from the checkpoint evidence they seal. A common model name alone is not a comparison contract. The manifest records the expected settings, task rows and capture context. Each original receipt seals its own values. No cross-model causal comparison is claimed.

FileModelRunner sourceHarnessCall limitsJudgeSuite / freezeRuntime / starter / environment
claude-1HaikuDriftproof source at commit c74110f89a62f0b9192b9c38c177859c4365e475, released in 0.24.0{"name":"claude-code","version":"2.1.287"}{"max_output_tokens":16384,"max_turns":32,"timeout_ms":300000}claude-sonnet-5-5d60f94bb204643e335239404dae412d482588514d658380734dcaf027793b67f / 3ee521dbd58c525e4f4cb3aa0ecf05a6f21c21884a28afd6456032198b101698650ea8175b6eaaf14056b150162e9b6698253d14ed86b1c3aa9e4618cbf73b5d / 47ea2419cc901bac7e89253ca7735db211aff5248d748cb6892fc88f6c53856c / a8d2649ffddaffda72d559f070b4d21d4d25781ffd1c4f5415e1780f85c9984e
claude-2HaikuDriftproof source at commit ff3b5671926842cf5bd0c087291df91b1843f759, released in 0.24.0{"name":"claude-code","version":"2.1.287"}{"max_output_tokens":16384,"max_turns":32,"timeout_ms":300000}claude-sonnet-5-5d60f94bb204643e335239404dae412d482588514d658380734dcaf027793b67f / 3ee521dbd58c525e4f4cb3aa0ecf05a6f21c21884a28afd6456032198b101698650ea8175b6eaaf14056b150162e9b6698253d14ed86b1c3aa9e4618cbf73b5d / 47ea2419cc901bac7e89253ca7735db211aff5248d748cb6892fc88f6c53856c / a8d2649ffddaffda72d559f070b4d21d4d25781ffd1c4f5415e1780f85c9984e
claude-3HaikuDriftproof source at commit ff3b5671926842cf5bd0c087291df91b1843f759, released in 0.24.0{"name":"claude-code","version":"2.1.287"}{"max_output_tokens":16384,"max_turns":32,"timeout_ms":300000}claude-sonnet-5-5d60f94bb204643e335239404dae412d482588514d658380734dcaf027793b67f / 3ee521dbd58c525e4f4cb3aa0ecf05a6f21c21884a28afd6456032198b101698650ea8175b6eaaf14056b150162e9b6698253d14ed86b1c3aa9e4618cbf73b5d / 47ea2419cc901bac7e89253ca7735db211aff5248d748cb6892fc88f6c53856c / a8d2649ffddaffda72d559f070b4d21d4d25781ffd1c4f5415e1780f85c9984e
claude-4HaikuDriftproof source at commit ff3b5671926842cf5bd0c087291df91b1843f759, released in 0.24.0{"name":"claude-code","version":"2.1.287"}{"max_output_tokens":16384,"max_turns":32,"timeout_ms":300000}claude-sonnet-5-5d60f94bb204643e335239404dae412d482588514d658380734dcaf027793b67f / 3ee521dbd58c525e4f4cb3aa0ecf05a6f21c21884a28afd6456032198b101698650ea8175b6eaaf14056b150162e9b6698253d14ed86b1c3aa9e4618cbf73b5d / 47ea2419cc901bac7e89253ca7735db211aff5248d748cb6892fc88f6c53856c / a8d2649ffddaffda72d559f070b4d21d4d25781ffd1c4f5415e1780f85c9984e
agents-1HaikuDriftproof source at commit ff3b5671926842cf5bd0c087291df91b1843f759, released in 0.24.0{"name":"claude-code","version":"2.1.287"}{"max_output_tokens":16384,"max_turns":32,"timeout_ms":300000}claude-sonnet-5-5d60f94bb204643e335239404dae412d482588514d658380734dcaf027793b67f / 3ee521dbd58c525e4f4cb3aa0ecf05a6f21c21884a28afd6456032198b101698650ea8175b6eaaf14056b150162e9b6698253d14ed86b1c3aa9e4618cbf73b5d / 47ea2419cc901bac7e89253ca7735db211aff5248d748cb6892fc88f6c53856c / a8d2649ffddaffda72d559f070b4d21d4d25781ffd1c4f5415e1780f85c9984e
agents-2HaikuDriftproof source at commit ff3b5671926842cf5bd0c087291df91b1843f759, released in 0.24.0{"name":"claude-code","version":"2.1.287"}{"max_output_tokens":16384,"max_turns":32,"timeout_ms":300000}claude-sonnet-5-5d60f94bb204643e335239404dae412d482588514d658380734dcaf027793b67f / 3ee521dbd58c525e4f4cb3aa0ecf05a6f21c21884a28afd6456032198b101698650ea8175b6eaaf14056b150162e9b6698253d14ed86b1c3aa9e4618cbf73b5d / 47ea2419cc901bac7e89253ca7735db211aff5248d748cb6892fc88f6c53856c / a8d2649ffddaffda72d559f070b4d21d4d25781ffd1c4f5415e1780f85c9984e
agents-3HaikuDriftproof source at commit ff3b5671926842cf5bd0c087291df91b1843f759, released in 0.24.0{"name":"claude-code","version":"2.1.287"}{"max_output_tokens":16384,"max_turns":32,"timeout_ms":300000}claude-sonnet-5-5d60f94bb204643e335239404dae412d482588514d658380734dcaf027793b67f / 3ee521dbd58c525e4f4cb3aa0ecf05a6f21c21884a28afd6456032198b101698650ea8175b6eaaf14056b150162e9b6698253d14ed86b1c3aa9e4618cbf73b5d / 47ea2419cc901bac7e89253ca7735db211aff5248d748cb6892fc88f6c53856c / a8d2649ffddaffda72d559f070b4d21d4d25781ffd1c4f5415e1780f85c9984e
claude-1SonnetDriftproof source at commit ff3b5671926842cf5bd0c087291df91b1843f759, released in 0.24.0{"name":"claude-code","version":"2.1.287"}{"max_output_tokens":16384,"max_turns":32,"timeout_ms":300000}claude-sonnet-5-5d60f94bb204643e335239404dae412d482588514d658380734dcaf027793b67f / 3ee521dbd58c525e4f4cb3aa0ecf05a6f21c21884a28afd6456032198b101698650ea8175b6eaaf14056b150162e9b6698253d14ed86b1c3aa9e4618cbf73b5d / 47ea2419cc901bac7e89253ca7735db211aff5248d748cb6892fc88f6c53856c / a8d2649ffddaffda72d559f070b4d21d4d25781ffd1c4f5415e1780f85c9984e
claude-2SonnetDriftproof source at commit ff3b5671926842cf5bd0c087291df91b1843f759, released in 0.24.0{"name":"claude-code","version":"2.1.287"}{"max_output_tokens":16384,"max_turns":32,"timeout_ms":300000}claude-sonnet-5-5d60f94bb204643e335239404dae412d482588514d658380734dcaf027793b67f / 3ee521dbd58c525e4f4cb3aa0ecf05a6f21c21884a28afd6456032198b101698650ea8175b6eaaf14056b150162e9b6698253d14ed86b1c3aa9e4618cbf73b5d / 47ea2419cc901bac7e89253ca7735db211aff5248d748cb6892fc88f6c53856c / a8d2649ffddaffda72d559f070b4d21d4d25781ffd1c4f5415e1780f85c9984e
claude-3SonnetDriftproof source at commit ff3b5671926842cf5bd0c087291df91b1843f759, released in 0.24.0{"name":"claude-code","version":"2.1.287"}{"max_output_tokens":16384,"max_turns":32,"timeout_ms":300000}claude-sonnet-5-5d60f94bb204643e335239404dae412d482588514d658380734dcaf027793b67f / 3ee521dbd58c525e4f4cb3aa0ecf05a6f21c21884a28afd6456032198b101698650ea8175b6eaaf14056b150162e9b6698253d14ed86b1c3aa9e4618cbf73b5d / 47ea2419cc901bac7e89253ca7735db211aff5248d748cb6892fc88f6c53856c / a8d2649ffddaffda72d559f070b4d21d4d25781ffd1c4f5415e1780f85c9984e
claude-4SonnetDriftproof source at commit ff3b5671926842cf5bd0c087291df91b1843f759, released in 0.24.0{"name":"claude-code","version":"2.1.287"}{"max_output_tokens":16384,"max_turns":32,"timeout_ms":300000}claude-sonnet-5-5d60f94bb204643e335239404dae412d482588514d658380734dcaf027793b67f / 3ee521dbd58c525e4f4cb3aa0ecf05a6f21c21884a28afd6456032198b101698650ea8175b6eaaf14056b150162e9b6698253d14ed86b1c3aa9e4618cbf73b5d / 47ea2419cc901bac7e89253ca7735db211aff5248d748cb6892fc88f6c53856c / a8d2649ffddaffda72d559f070b4d21d4d25781ffd1c4f5415e1780f85c9984e
agents-1SonnetDriftproof source at commit ff3b5671926842cf5bd0c087291df91b1843f759, released in 0.24.0{"name":"claude-code","version":"2.1.287"}{"max_output_tokens":16384,"max_turns":32,"timeout_ms":300000}claude-sonnet-5-5d60f94bb204643e335239404dae412d482588514d658380734dcaf027793b67f / 3ee521dbd58c525e4f4cb3aa0ecf05a6f21c21884a28afd6456032198b101698650ea8175b6eaaf14056b150162e9b6698253d14ed86b1c3aa9e4618cbf73b5d / 47ea2419cc901bac7e89253ca7735db211aff5248d748cb6892fc88f6c53856c / a8d2649ffddaffda72d559f070b4d21d4d25781ffd1c4f5415e1780f85c9984e
agents-2SonnetDriftproof source at commit ff3b5671926842cf5bd0c087291df91b1843f759, released in 0.24.0{"name":"claude-code","version":"2.1.287"}{"max_output_tokens":16384,"max_turns":32,"timeout_ms":300000}claude-sonnet-5-5d60f94bb204643e335239404dae412d482588514d658380734dcaf027793b67f / 3ee521dbd58c525e4f4cb3aa0ecf05a6f21c21884a28afd6456032198b101698650ea8175b6eaaf14056b150162e9b6698253d14ed86b1c3aa9e4618cbf73b5d / 47ea2419cc901bac7e89253ca7735db211aff5248d748cb6892fc88f6c53856c / a8d2649ffddaffda72d559f070b4d21d4d25781ffd1c4f5415e1780f85c9984e
agents-3SonnetDriftproof source at commit ff3b5671926842cf5bd0c087291df91b1843f759, released in 0.24.0{"name":"claude-code","version":"2.1.287"}{"max_output_tokens":16384,"max_turns":32,"timeout_ms":300000}claude-sonnet-5-5d60f94bb204643e335239404dae412d482588514d658380734dcaf027793b67f / 3ee521dbd58c525e4f4cb3aa0ecf05a6f21c21884a28afd6456032198b101698650ea8175b6eaaf14056b150162e9b6698253d14ed86b1c3aa9e4618cbf73b5d / 47ea2419cc901bac7e89253ca7735db211aff5248d748cb6892fc88f6c53856c / a8d2649ffddaffda72d559f070b4d21d4d25781ffd1c4f5415e1780f85c9984e

The recorded usage fields do not establish whether output tokens include thinking tokens. Usage fields: cached_tokens, input_tokens, output_tokens, wall_ms.

Amendments

, amendment 1 (issue 577): added a visible receipts link to the opening and a Receipts heading so both links reach appendix D. Findings and receipt counts are unchanged.

, amendment two (issue six hundred fourteen): added a link from the file and fake panel to the local preview. Findings and receipts are unchanged.

Skills measured: 2025Emma/vibe-coding-cn: CLAUDE.md; DietrichGebert/ponytail: AGENTS.md; Sahir619/fable-method: AGENTS.md; drona23/claude-token-efficient: CLAUDE.md; multica-ai/andrej-karpathy-skills: CLAUDE.md; walkinglabs/learn-harness-engineering: AGENTS.md and CLAUDE.md.

Every Driftproof result in one file: JSONL or CSV, and what a row is and is not.

Want a skill measured, or think a result is wrong? It's free: how to ask.

Words with a dotted line link to the glossary, where each is explained in plain words.