LLM-as-judge

Plain English. Using one model to grade another's output. Human evaluation does not scale to millions of answers, so a judge model scores them instead — in benchmarks, in reinforcement-learning reward loops, in product quality checks, in model ensembles.

Why it moves money. A large share of the quality claims investors consume were graded by a model, which makes judge reliability a hidden dependency under the whole measurement stack. The biases are now measured, not hypothesised: models rank their own vendor's outputs about half a position too high in controlled tests, and research on agents optimising against black-box judges shows the judge itself becomes an attack surface. Where a judge is the reward signal, its blind spots are trained into the product.

What to watch. Whether an eval discloses its judge's model family (grading your own vendor's homework is the base case, not the exception); reported agreement rates with human graders; and evidence the graded model is optimising the judge rather than the task.

From the signals. Self-preference measured: vendors rank their own models half a position too high. Inverse rubric optimisation: measuring agents is becoming the binding constraint. OpenRouter's Fusion ensemble is only as good as its judge model.

Further reading. Judging LLM-as-a-Judge (Zheng et al., 2023) — the paper that named the method and its biases.