A paper posted on 3 September reports two preregistered studies of black-box LLM judges that never got as far as their research questions. The reliability checks the authors had fixed in advance failed, and the paper is the report of that failure across 52,988 audited request attempts.
The two gates that failed
Repeat rankings taken within the same window agreed at Spearman 0.400 against a required 0.90. Byte-identical replays run the next day agreed at 0.78 against a required 0.99. These are not post-hoc thresholds chosen to make a point — they were preregistered before the data was collected, which is what gives the negative result its force.
It is not one provider's problem
On provider switching, "four providers share the floor, medians 0.74 to 0.88." Simply waiting did not help — 0.805 versus 0.800, replicated over five further days. Self-hosting on batch-invariant kernels, the known mitigation for inference nondeterminism, "helped only while the server was quiet", which points at the mechanism: batching behaviour under load in the serving stack.
The cheap check the authors recommend
The paper notes that a pilot at roughly 2% of the study's call volume would have exposed both unreachable gates in advance. It ships a three-level snapshot-identity ladder, eight design rules and a reporting checklist as the practical output.
What the received framing gets wrong
"LLM judges are unreliable" is too broad a reading. The finding concerns externally measured behaviour on shared serving infrastructure: a model name behind a shared endpoint is not a frozen instrument, and that is a property of the serving stack rather than of the model's judgement. Two further caveats sit in the paper's favour and against the sensational version. The 0.90 and 0.99 gates are unusually strict, and a study choosing looser ones would have passed — the value is that they were fixed in advance, not that 0.400 is a constant. And part of the failure is that the candidate gaps being compared were seven orders of magnitude below the instrument's own noise floor, which is to say the differences were too small to measure at all.