Driftproof

The judge is deliberately fixed and cheap.

Driftproof grades every generation with an LLM judge. That judge is fixed at claude-haiku-4-5 and changed only rarely. This page says why, and states the limitations plainly — a judge you can't inspect is not evidence.

Why a fixed, cheap judge

Driftproof measures how a skill's effect moves as the model under it changes. If the judge also moved from report to report, a change in the verdict could come from the skill, the target model, or the grader — three variables, one number. Pinning the judge removes one of them: the judge is held constant so the target model is the only thing that varies across a drift comparison.

Cheap is a feature, not a compromise. A frontier judge would cost more than the runs it grades and would tempt per-report upgrades that quietly break comparability. Haiku is fast, inexpensive, and — paired with anchored rubrics and repeated sampling — stable enough that a verdict rests on band separation, not on the judge's brilliance.

Determinism where the surface allows it

On the api surface the judge is pinned to temperature 0, and the receipt records sampling: "api-temperature-0". On the claude-cli surface sampling parameters are surface-controlled and cannot be set; the receipt records sampling: "surface-controlled" and a null temperature — honestly, rather than pretending to a determinism we don't have. Either way the judge is sampled several times per case, and every verdict requires the two confidence bands to separate. Residual judge variance is why we band, not a bug in the banding.

Upgrades are rare, announced, and reset comparability

The judge model changes only when there is a compelling reason (the pinned model retires, or a materially better cheap judge ships). When it does:

Known limitations (stated plainly)

The judge is one held-constant instrument. Hold it constant, sample it, band it, and state where it is weak — that is the whole policy.