One grader, fixed, and cheap on purpose
Another model, the grader, scores every answer. For our reports it is one fixed model, and a cheap one, by design; a report that uses another grader says so.
Why one grader
We measure how a skill's effect moves when the model under it changes. If the grader changed too, a moved result could come from the skill, the model or the grader. Holding the grader fixed leaves the model as the one thing that changes.
Cheap is part of the point. A costly grader would cost more than the runs it scores, and would tempt upgrades that leave old and new reports impossible to compare.
When the grader changes
Rarely, and only for a good reason, such as the model being retired. The first report that uses a new grader says so, and results scored by different graders are not compared directly.
What the grader cannot do
It may favour its own family
A grader can be a little kinder to writing like its own. Comparing each model with and without the skill reduces this; it does not remove it.
It cannot mark what it cannot do
On hard tasks the grader's own ability is the ceiling. Scores bunch in the middle and spreads widen, which makes a change harder to call, not easier.
It scores in coarse steps
Its scores fall on a coarse grid of about 0.05 to 0.1. That is why a change has to be at least 0.05, about one step, before it is called.
A broken reply is not a score
A reply that cannot be read as a score between zero and one is recorded as not measured. It is never counted as a result.
The precise policy
The judge policy in full, with its terms
The judge is deliberately fixed and cheap.
Driftproof grades every generation with an LLM judge. That judge is fixed at claude-haiku-4-5 and changed only rarely. This page says why, and states the limitations plainly - a judge you can't inspect is not evidence.
The fixed Haiku judge applies to the reports Driftproof publishes; a receipt you produce records whichever judge its surface and configuration ran, in run.judge. In the installed tool the judge follows your own configuration rather than this pin, and the receipt records which judge ran.
Why a fixed, cheap judge
Driftproof measures how a skill's effect moves as the model under it changes. If the judge also moved from report to report, a change in the verdict could come from the skill, the target model, or the grader - three variables, one number. Pinning the judge removes one of them: the judge is held constant so the target model is the only thing that varies across a drift comparison.
Cheap is a feature, not a compromise. A frontier judge would cost more than the runs it grades and would tempt per-report upgrades that quietly break comparability. Haiku is fast, inexpensive, and - paired with anchored rubrics and repeated sampling - stable enough that a verdict rests on band separation, not on the judge's brilliance.
Determinism where the surface allows it
On the api surface the judge is pinned to temperature 0, and the receipt records sampling: "api-temperature-0". On the claude-cli surface sampling parameters are surface-controlled and cannot be set; the receipt records sampling: "surface-controlled" and a null temperature - honestly, rather than pretending to a determinism we don't have. Either way the judge is sampled several times per case, and every verdict requires the two bands, each a descriptive spread of one sample standard deviation, to separate. Residual judge variance is why we band, not a bug in the banding.
Upgrades are rare, announced, and reset comparability
The judge model changes only when there is a compelling reason (the pinned model retires, or a materially better cheap judge ships). When it does:
- the change is announced in the report that first uses the new judge;
- it resets cross-report comparability - verdicts graded by different judges are not directly comparable, and reports say so;
- the eligibility is encoded in the model registry: only a model with
judge_eligible: trueinconfig/models.jsonmay serve as the judge. Today that is the Haiku judge alone.
Known limitations (stated plainly)
- Style affinity. An LLM judge can favor prose that resembles its own. A Haiku judge grading a Haiku-family generation may be marginally kinder to it than to another family's output. Anchored rubrics (a fully-correct answer scores ~0.80, not 1.0) and with/without-skill differencing blunt this, but it is not zero.
- Ceiling on hard suites. A cheap judge cannot reliably grade a task it could not itself do well. On genuinely hard rubrics the judge's own competence is the ceiling on grading accuracy; scores compress toward the middle and bands widen, which correctly makes verdicts harder to claim rather than falsely confident.
- Quantization. The judge emits scores on a coarse ~0.05–0.1 grid, so a confident grade often collapses to a zero-width "point band". This is exactly why the drift rule adds a 0.05 minimum-effect floor on top of band separation - see any report's methodology.
- Parse failures fail closed. A judge reply that is empty, cut off at its output cap, unparseable, or carries a score that is not a number between 0 and 1 is refused and recorded as unmeasured, never a score and never a silent pass. The transcript hashes in receipt spec v0.3 let a reader check the raw judge outputs a verdict was built from.
The judge is one held-constant instrument. Hold it constant, sample it, band it, and state where it is weak - that is the whole policy.