LLM-as-a-Judge uses a model to label, score or compare AI outputs against explicit criteria. It can make large evaluation suites practical, but it does not create ground truth. A judge can share the same blind spots as the system it evaluates, prefer a writing style, miss unsupported facts or change its verdict when candidate order changes.

Use model judges as one layer within AI Testing & Evaluation, alongside deterministic graders and calibrated human review.

Choose the right judging format

Classification

Ask for a bounded label such as pass, fail or needs_review. This works when the rubric describes observable conditions and includes examples.

Criteria scoring

Score separate dimensions—such as groundedness, completeness and clarity—rather than one vague overall number. Keep hard failures outside the average.

Pairwise comparison

Show two candidate outputs for the same task and ask which better satisfies the rubric. Pairwise judgments are often easier to define than absolute scores, but order and presentation can bias the result.

Reference-based grading

Give the judge approved facts, source excerpts or expected behavior. This is stronger than asking it to rely on its own memory, especially for factual or domain-specific tasks.

Write a judge rubric

A useful rubric specifies:

• the product task and intended user;
• evidence the judge may use;
• one criterion per decision;
• passing and failing examples;
• treatment of uncertainty and missing context;
• output labels with machine-readable definitions;
• cases that must be escalated to a human.

Avoid “rate quality from 1 to 5.” The judge cannot apply a standard that the team has not defined.

{
  "label": "unsupported_claim",
  "definition": "The answer states a factual claim not supported by the supplied sources.",
  "allowed": ["paraphrase supported by source", "explicit uncertainty"],
  "fail": ["new number without source", "confident claim contradicting source"]
}

Prefer objective graders first

Do not use an LLM judge for behavior code can establish directly:

• valid JSON or schema conformance;
• required fields or citations;
• permitted tool names and argument ranges;
• unit-test or compiler results;
• database state after an agent action;
• latency, token use and cost.

The Structured Outputs guide and JSON parsing foundation cover deterministic validation. Reserve judge calls for meaning that cannot be reduced to a reliable rule.

Control common judge biases

Position bias

For pairwise evaluation, swap candidate order on a sample—or on every important case—and measure whether the verdict changes. An unstable pair should be marked as disagreement, not forced into a winner.

Verbosity and style bias

Tell the judge not to reward length, formatting or confident tone unless the product rubric explicitly values them. Include concise high-quality examples during calibration.

Self-preference and shared blind spots

A different judge family is not automatically unbiased, and the same family is not automatically invalid. Measure performance on reviewed cases rather than relying on a brand rule. Use external evidence for factuality.

Prompt sensitivity and variance

Pin the judge configuration where possible, record it with every run and repeat ambiguous cases. A new judge model or rubric version creates a new measurement system and must be recalibrated.

Calibrate against human review

Create a calibration set that represents ordinary, borderline and severe failures. Have qualified reviewers label it independently, resolve or document disagreement, then compare the judge against those reviewed labels.

Track more than overall agreement:

• false passes on severe errors;
• false failures on acceptable output;
• performance by language, intent and risk slice;
• pairwise order reversals;
• cases where human reviewers disagree.

Choose acceptance thresholds from product risk. A judge that is useful for triaging marketing copy may be unacceptable as the only safety gate for an agent with production permissions.

Combine automation and humans

A scalable workflow is:

1. deterministic graders reject contract and safety violations;
2. model judges score the remaining subjective criteria;
3. humans review high-risk cases, disagreements and a random sample;
4. confirmed mistakes become calibration or regression cases;
5. judge and rubric versions are stored with results.

Use the judge to prioritize review, not to erase accountability. Human overrides should retain a reason so the team can improve the rubric or identify a genuinely ambiguous requirement.

Use judges in regression testing

Run baseline and candidate outputs through the same judge configuration. Compare distributions and named slices rather than one average. Preserve the raw output and judge label so reviewers can inspect regressions.

Avoid letting an experimental prompt author change the rubric merely to make the candidate pass. Dataset, system and grader changes need separate version histories. See LLM regression testing and building LLM evaluation datasets.

Cost, latency and privacy

Judge calls consume time and tokens. Use deterministic filters first, select an appropriate judge for the rubric and run the broadest suites at protected merge or scheduled intervals rather than on every keystroke.

The judge receives model outputs and often the original input or reference context. Treat that payload as application data: minimize it, protect credentials and confirm provider storage and regional controls. Do not send sensitive production traces merely because evaluation is “internal.”

When not to rely on a model judge

Do not make it the sole authority for:

• exact factual correctness without supplied evidence;
• executable code correctness;
• legal, medical or financial decisions;
• regulatory compliance;
• authorization and permission boundaries;
• destructive or irreversible agent actions.

For agent evaluation, judge the narrative output separately from tool execution, approval and final state. The latter belong in deterministic trace checks and human-controlled gates. See Coding Agents and AI Operations.

LLM-as-a-Judge is successful when it turns a well-defined rubric into repeatable triage and measurement. It fails when a model score is treated as objective truth without evidence, calibration or an escalation path.