Templates

LLM evaluation rubric template for production AI features

Anchored grading criteria for LLM outputs, an LLM-as-judge prompt you can adapt, and the calibration steps that make automated scores match expert judgment.

By the Ryz Labs team · Updated October 2026

An LLM evaluation rubric defines how each output of an AI feature is graded: a small set of criteria such as correctness, grounding in sources, completeness, format and safety, each with written anchors for every score and examples of acceptable and unacceptable outputs. Use one criterion per grader rather than one overall score, require a short justification, and calibrate automated LLM-as-judge grading against expert labels before you trust it. The template below gives you the criteria, a judge prompt, an eval case format and a calibration log.

LLM evaluation rubric template

1. Criteria with anchored scores

Criterion	Question it answers	0: Fail	1: Partial	2: Pass	Grader
Correctness	Is the answer factually right for this input?	Contains a wrong statement that would mislead the user	Mostly right; a minor error or imprecision	Fully right	Expert label or LLM judge with reference answer
Grounding	Is every claim supported by the provided sources?	A claim with no support in the sources	Supported but cites the wrong passage	Every claim supported and cited correctly	LLM judge with retrieved context
Completeness	Does it cover what the user needed?	Misses the main point	Covers the main point, misses a required detail	Covers everything the reference requires	LLM judge with reference answer
Format and instructions	Does it follow the required format and constraints?	Breaks the schema or a hard constraint	Minor format issue	Fully compliant	Code check (schema, length, required fields)
Safety and policy	Does it avoid disallowed content and data leakage?	Violates policy or exposes restricted data	Borderline wording	Compliant	Code checks plus LLM judge
Tone and usefulness	Would the target user accept it as written?	Unusable	Needs edits	Usable as is	Expert sample review

2. Eval case format (one row per test case)

Case ID	Input	Context or sources provided	Reference answer or must-include points	Must not include	Tags (topic, difficulty, edge case)	Expert label (per criterion)
EV-001	[User question or document]	[Retrieved passages or attached files]	[Key facts the answer must contain]	[Wrong facts, restricted data]	[Billing; hard; ambiguous question]	[Correctness 2, Grounding 1, ...]

3. LLM-as-judge prompt (one criterion)

Part	Text to adapt
Role	You are grading one answer from an AI assistant for [product]. Grade only [criterion].
Criterion definition	[Paste the criterion question and the 0/1/2 anchors from the table above.]
Inputs	User input: [input]. Sources: [context]. Reference: [reference answer]. Answer to grade: [output].
Rules	Judge only against the sources and reference. Do not reward length. If unsure between two scores, choose the lower.
Output format	Return JSON: {"score": 0, 1 or 2, "justification": one or two sentences quoting the evidence}

4. Calibration log

Date	Judge model and prompt version	Criterion	Cases labeled by experts	Agreement with experts	Main disagreements	Action
[YYYY-MM-DD]	[Model, prompt v3]	[Grounding]	[n]	[% exact match]	[Judge too lenient on partial citations]	[Added two partial-citation examples to the prompt]

Use this template

How to use this evaluation rubric

Collect eval cases from real usage or realistic tasks from target users, including the hard and ambiguous ones. Tag each case by topic and difficulty.
Pick the criteria that matter for the feature and rewrite the anchors in your domain's terms. Most features need correctness and grounding; agents also need checks on tool calls and actions.
Use code checks wherever possible: schema validation, length, required fields, banned strings. Reserve LLM judges for criteria that need judgment.
Have domain experts label a set of cases per criterion. These labels are the ground truth for calibration.
Run the judge prompt on the labeled cases, measure agreement with the experts and adjust the prompt or anchors until agreement is acceptable to you. Log every change.
Run the full eval on every prompt, model or retrieval change, and block releases that drop below the agreed scores.
Recalibrate whenever you change the judge model or the rubric.

Example: one graded case

Criterion	Score	Justification
Correctness	2	States the correct refund window for annual plans, matching the reference.
Grounding	1	The refund window is supported, but the citation points to the monthly plan section instead of the annual plan section.
Format	2	Valid JSON with all required fields (code check).

Common LLM evaluation mistakes

One overall quality score. A single 1-10 score hides which property failed. Grade criteria separately.
No anchors. Without written definitions per score, judges and humans interpret the scale differently on every case.
Trusting an uncalibrated judge. Measure agreement with expert labels before using automated scores to make release decisions.
Easy test sets. If every case passes, the set is not testing the cases that hurt users. Add failures from production.
Rewarding length. Judges often prefer longer answers. Tell the judge not to, and check whether scores correlate with length.
Evals outside the release process. An eval that does not block a bad release is a report, not a safeguard.

Need engineers who build evals into the feature?

Ryz AI pod teams and senior AI engineers build eval sets, judges and tracing as part of every AI feature, in your repos and CI, so quality is measured on every change. They work on your team, reporting to your leads, on US business hours including New York hours. Only the top 1% of the tens of thousands of engineers we interview make it. Talk to us to scope your team, or see LLM evaluation engineers and RAG engineers.

FAQ

What is an LLM evaluation rubric?

A scoring guide for the outputs of an LLM feature. It names the criteria being graded, defines what each score means with written anchors, and gives examples. Humans and LLM judges use it to grade outputs consistently.

What criteria should an LLM evaluation use?

Most production features use correctness, grounding in the provided sources, completeness, format and instruction compliance, and safety. Add usefulness or tone for user-facing text, and tool-call correctness for agents.

Can you use an LLM as a judge?

Yes, for criteria that need judgment, if you calibrate it. Label a set of cases with domain experts, measure how often the judge agrees, adjust the prompt and anchors, and recheck whenever the judge model or rubric changes.

How many test cases does an LLM eval set need?

Enough to cover the main topics, difficulty levels and known failure modes of the feature, with several cases each. Start with what you can label well, then add cases from production failures over time.

Should eval scores use a 0-2 or a 1-10 scale?

Short scales with written anchors, such as 0-2 or pass/fail, give more consistent grades from both humans and LLM judges than long scales. Use a separate score per criterion rather than one large scale.

Questions we didn't answer? Email info@ryzlabs.com.