The judge is deliberately fixed and cheap.
Driftproof grades every generation with an LLM judge. That judge is fixed at claude-haiku-4-5 and changed only rarely. This page says why, and states the limitations plainly — a judge you can't inspect is not evidence.
Why a fixed, cheap judge
Driftproof measures how a skill's effect moves as the model under it changes. If the judge also moved from report to report, a change in the verdict could come from the skill, the target model, or the grader — three variables, one number. Pinning the judge removes one of them: the judge is held constant so the target model is the only thing that varies across a drift comparison.
Cheap is a feature, not a compromise. A frontier judge would cost more than the runs it grades and would tempt per-report upgrades that quietly break comparability. Haiku is fast, inexpensive, and — paired with anchored rubrics and repeated sampling — stable enough that a verdict rests on band separation, not on the judge's brilliance.
Determinism where the surface allows it
On the api surface the judge is pinned to temperature 0, and the receipt records sampling: "api-temperature-0". On the claude-cli surface sampling parameters are surface-controlled and cannot be set; the receipt records sampling: "surface-controlled" and a null temperature — honestly, rather than pretending to a determinism we don't have. Either way the judge is sampled several times per case, and every verdict requires the two confidence bands to separate. Residual judge variance is why we band, not a bug in the banding.
Upgrades are rare, announced, and reset comparability
The judge model changes only when there is a compelling reason (the pinned model retires, or a materially better cheap judge ships). When it does:
- the change is announced in the report that first uses the new judge;
- it resets cross-report comparability — verdicts graded by different judges are not directly comparable, and reports say so;
- the eligibility is encoded in the model registry: only a model with
judge_eligible: trueinconfig/models.jsonmay serve as the judge. Today that is the Haiku judge alone.
Known limitations (stated plainly)
- Style affinity. An LLM judge can favor prose that resembles its own. A Haiku judge grading a Haiku-family generation may be marginally kinder to it than to another family's output. Anchored rubrics (a fully-correct answer scores ~0.80, not 1.0) and with/without-skill differencing blunt this, but it is not zero.
- Ceiling on hard suites. A cheap judge cannot reliably grade a task it could not itself do well. On genuinely hard rubrics the judge's own competence is the ceiling on grading accuracy; scores compress toward the middle and bands widen, which correctly makes verdicts harder to claim rather than falsely confident.
- Quantization. The judge emits scores on a coarse ~0.05–0.1 grid, so a confident grade often collapses to a zero-width "point band". This is exactly why the drift rule adds a 0.05 minimum-effect floor on top of band separation — see any report's methodology.
- Parse failures fail closed. An unparseable judge reply scores 0, never a silent pass. The transcript hashes in receipt spec v0.3 let a reader check the raw judge outputs a verdict was built from.
The judge is one held-constant instrument. Hold it constant, sample it, band it, and state where it is weak — that is the whole policy.