What outside audits found, and what we fixed.
Three external audits examined Driftproof 0.10.0 and 0.10.1 in September 2026. This page records what each found, what is fixed and by which spec, and what is still open.
We publish this for the same reason we publish refusals. A measuring tool that keeps its own defects quiet is asking to be trusted on the one point it cannot show. Every figure below is the auditors' own, taken from their reports.
The audits
The 0.10.0 plugin audit, 13 September 2026
It installed the published Claude Code plugin, validated its manifest and commands, and ran the init, run and badge backends with the model stubbed out.
It found that each command file told Claude to fill in the plugin's location itself, where Claude Code only substitutes ${CLAUDE_PLUGIN_ROOT}, so a command ran from whatever path the model wrote. It also found that no command set disable-model-invocation, so Claude could start /driftproof:run on its own judgement and spend the user's subscription. Both were fixed in 0.10.1 (spec 028, amendment A-028-37), and our gate now checks each of the two properties separately (A-028-39). The same audit raised the --trusted-skill flag and Windows, which are still open.
The 0.10.1 end-to-end retest, 14 September 2026
It installed 0.10.1 from the marketplace and ran all three commands in a fresh Claude Code session on a subscription, with no API key. The live receipt was TESTED, its schema valid and its hash verified, and every generation and judge hash matched the output the auditor captured.
It found that the badge refused an edited generation text but accepted an edited judge output, and that the badge said judge text was not retained when the transcripts beside the receipt held it. On the auditor's macOS machine the shipped gate passed 592 of 611, with 19 failed and 1 not applicable. All nineteen failures are in the publishing helper, which needs GNU realpath -m, and the run published nothing.
The 0.10.1 reliability audit, received 14 September
It ran 210 subscription CLI calls on three small exact-answer cases it wrote itself, then injected faults at the provider boundary and ran 200,000 synthetic run pairs through the production statistics. On the live cases the planted regression and the planted improvement were each reported on all three, and the unchanged comparison reported no separation on any of them. The auditor is clear that three authored cases cannot establish a long-run error rate.
A judge reply whose score was not a number was measured anyway: null, false, an empty string and an empty array became 0, and true became 1. With the generated answers unchanged, switching the judge's scores from 1 to null produced a regression report. This was fault injection at the provider boundary, not a reply a live model sent.
The badge did not verify retained judge output, and it said judge text was not kept.
Disagreement between judge samples can vanish from a band. In a synthetic control where every draw got conflicting grades, the spread within the judge's samples was 0.707 and 0.566 in two runs while the spread across draws was zero, and Driftproof reported an improvement on bands of zero width.
Failure handling mostly held: timeouts and empty or truncated generations produced incomplete receipts and no verdict. But a receipt with no reported model identity records attested as false and can still be marked TESTED, and an incomplete receipt can carry TESTED as well.
In a separate challenge of 12 fixed responses each graded twice, the Haiku judge scored 22 of 24 (91.7%), below the auditor's 95% target. It twice accepted a quoted answer where the rubric asked for the bare string. Opus 5 scored 24 of 24.
The statistical tests set targets before any result: at most 5% false changes, and at least 90% detection of a planted regression. A synthetic perfect judge isolates the rule, so these are results under stated assumptions, not general error rates.
| Score distribution | Shift | Measured | Wilson interval | Target met |
|---|---|---|---|---|
| clipped normal, sd 0.05 | 0.00 | false change 6.43% | 6.10% to 6.78% | no |
| clipped normal, sd 0.05 | 0.30 | detection 100.00% | 99.98% to 100.00% | yes |
| clipped normal, sd 0.1 | 0.00 | false change 6.16% | 5.83% to 6.50% | no |
| clipped normal, sd 0.1 | 0.30 | detection 94.21% | 93.88% to 94.53% | yes |
| clipped normal, sd 0.2 | 0.00 | false change 3.26% | 3.02% to 3.51% | yes |
| clipped normal, sd 0.2 | 0.30 | detection 38.39% | 37.72% to 39.07% | no |
| binary | 0.00 | false change 8.85% | 8.46% to 9.25% | no |
| binary | 0.30 | detection 16.11% | 15.61% to 16.63% | no |
| rare outlier | 0.00 | false change 0.27% | 0.20% to 0.35% | yes |
| rare outlier | 0.30 | detection 82.62% | 82.09% to 83.14% | no |
What is fixed in 0.10.2
Specs 031 and 032 are merged and ship in 0.10.2.
- Spec 031: the badge rechecks retained judge text against the digests the receipt carries, and refuses the receipt when any retained judge output no longer matches. Before, it checked generation text only.
- Spec 031: the badge's statement says what it checked. It no longer says judge text is not retained when the transcripts beside the receipt hold it.
- Spec 031: the caveat that calls a comparison not like for like prints only when the two receipts' bands really do come from different sources. It had printed whenever either receipt's band carried a source label, which every v0.4 and v0.5 band does, so it printed on pairs that were like for like.
- Spec 031: when our plugin gate declines to judge a case, it records a refusal rather than a failure. The two used to write the same result, so a gate that would not judge looked like a product that had failed.
- Spec 031: the band wording. A band is the mean plus or minus one standard deviation, a descriptive spread and not a confidence interval. The site, the differ's output and twelve published records described overlapping bands as though they showed that nothing had changed, and separated bands as though they proved a change. The wording is corrected on the site and in the differ. The twelve records keep their original text and carry a versioned amendment that corrects it, as every report amendment does.
- Spec 032: a judge reply whose score is not a number is refused the way an unparseable reply is, and no score is recorded from it. It had been measured as 0 or 1.
- Spec 032: the badge reads transcripts only from inside the transcripts directory. It refuses an index entry, or an index file, that resolves outside it, where a crafted index could have made it read files elsewhere.
What is still open
- False changes. In the five no-change conditions of the ten synthetic conditions, a change was reported from 0.27% to 8.85% of the time, and three of the five missed the 5% target. The highest rate was on the binary suite.
- Missed regressions. A planted regression was detected 16.11% of the time on the binary suite, and 38.39% of the time on the clipped normal suite with sd 0.2, against a 90% target.
- The single-receipt badge uses only the effect floor, with no requirement that bands separate. Under the synthetic binary null it showed passing 34.56% of the time and regressed 34.785% of the time, though the underlying means were equal. A green badge is not a calibrated claim.
- A receipt can be TESTED with attested set to false, meaning no model identity was reported, and an incomplete receipt can carry TESTED too. Read a receipt's status and its verdict, not its level alone.
- The
--trusted-skillflag checks only that the skill resolves inside the current git repository. That is not authorship or review, and the name claims more than the check. - Disagreement between judge samples can still disappear from a band, as the reliability audit's synthetic control showed.
- The Haiku judge's 22 of 24 on the judge challenge is below the auditor's target. The sample is small and clustered, and the judge has not changed.
- Windows is untested. From Node's source, the plugin starts
npxwithout a shell, which on a standard Windows install would not find it, but no Windows run has been made and that failure has never been observed. A CI matrix across Linux, macOS and Windows is planned. - How few samples can support a verdict. The underpowered-verdict rule is decided, and it is the next spec.