How to Evaluate LLM Applications: Evals, Metrics and Test Sets That Catch Regressions

Every team building with LLMs hits the same wall. You tweak a prompt to fix one bad answer, and three other answers quietly get worse. You switch to a newer model, and nobody can say with confidence whether it's better. The fix is the same thing that made traditional software reliable: tests. In the LLM world they're called evals.

This guide shows how to build an evaluation practice that's cheap to start, grows with your product, and actually catches regressions.

#Key takeaways

• Evals are automated tests for non-deterministic outputs. Without them, every change is a guess.
• Start with a golden dataset of 50–200 real inputs. It's the highest-leverage asset you'll build.
• Prefer code-based checks (exact match, schema validation, regex) wherever possible; use LLM-as-a-judge for subjective qualities, and calibrate it against humans.
• For RAG, evaluate retrieval and generation separately.
• Run evals in CI on every prompt or model change, and keep monitoring online after release.

#Why LLM evaluation is different

Traditional tests assert output == expected. LLM outputs vary in wording even when they're correct, and "correct" is often a spectrum: accurate but too long, helpful but slightly off-tone. So LLM evals use a mix of techniques:

#Step 1: Build a golden dataset

Your eval set is a list of test cases: inputs plus what "good" looks like.

json{"id": "refund-017",
 "input": "I was charged twice for March, can I get one refunded?",
 "expected_intent": "billing_refund",
 "must_include": ["duplicate charge", "refund"],
 "must_not_include": ["legal advice"],
 "tags": ["billing", "happy-path"]}

Where to get cases:

• Real user inputs from logs (anonymized). This beats anything you invent.
• Known failures. Every bug report becomes a test case, just like regression tests.
• Edge cases: empty inputs, very long inputs, other languages, adversarial or out-of-scope requests.
• Synthetic cases generated by an LLM to fill coverage gaps, reviewed by a human.

Tag cases by category so you can see where quality changes, not just the average score. Fifty well-chosen cases beat 5,000 random ones.

#Step 2: Choose metrics per task

Pick metrics that reflect what users actually need:

• Classification / routing: accuracy, precision and recall per class, confusion matrix. (Refresher: classification metrics explained.)
• Extraction: field-level exact match, and schema-validity rate.
• Summarization: coverage of key points (rubric), factual consistency with the source, length compliance.
• Chat / assistance: task success, helpfulness rubric, policy compliance, refusal appropriateness.
• Operational: latency (p50/p95), cost per request, error and timeout rates.

Keep operational metrics in the same report. A prompt that improves quality by 2% but doubles latency might not be a win.

#Step 3: Use code-based checks first

Code checks are fast, cheap, deterministic and trustworthy. Use them wherever the task allows:

pythondef check(case, output):
    results = {}
    results["valid_json"] = is_valid_json(output)
    data = json.loads(output) if results["valid_json"] else {}
    results["intent_correct"] = data.get("intent") == case["expected_intent"]
    text = data.get("reply", "").lower()
    results["includes_required"] = all(p in text for p in case["must_include"])
    results["avoids_forbidden"] = not any(p in text for p in case["must_not_include"])
    return results

#Step 4: LLM-as-a-judge, done carefully

For qualities code can't check, use a strong model as a grader with a specific rubric:

textYou are grading a customer-support reply.

Criteria (score each 1–5):
1. Correctness: consistent with the policy excerpt provided.
2. Completeness: addresses every question the customer asked.
3. Tone: polite, concise, no blame.

Return JSON: {"correctness": n, "completeness": n, "tone": n, "reasoning": "..."}

Make judges trustworthy:

• Ask for reasoning before the score, and prefer small scales (pass/fail or 1–5) over 1–100.
• Grade one criterion at a time when precision matters.
• Calibrate: have humans grade 50–100 samples and measure agreement with the judge. Adjust the rubric until agreement is high.
• Watch for known biases: judges can favor longer answers, their own model family's outputs, or whichever answer appears first in pairwise comparisons. Randomize order and control for length.

#Evaluating RAG systems

RAG has two failure points, so measure both (background: RAG explained):

Retrieval

• Context recall: did retrieval return the chunks that contain the answer?
• Context precision: how much of the retrieved context was relevant?

Generation

• Faithfulness / groundedness: is every claim supported by the retrieved context?
• Answer relevance: does the answer address the question?
• Citation accuracy: do cited sources actually support the sentences they're attached to?

If faithfulness is high but answers are wrong, fix retrieval. If retrieval is good but answers are wrong, fix the prompt or the model. This split saves days of guessing. It's also the most direct way to measure progress on reducing hallucinations.

#Step 5: Run evals in CI

Treat eval runs like a test suite:

textPR changes prompt/model/config
   → CI runs eval suite (parallel, cached where inputs unchanged)
   → Report: overall + per-tag scores, latency, cost, diffs vs main
   → Fail if a critical metric drops below threshold

Practical tips:

• Pin model versions in configs so results are reproducible.
• Run with temperature 0 where possible, or run multiple samples and average them for stochastic tasks.
• Show diffs: which specific cases flipped from pass to fail. That's where the insight is.
• Keep a fast "smoke" subset for every commit and the full suite for merges.

#Step 6: Monitor online

Offline evals can't anticipate everything users do. In production, track:

• Explicit feedback: thumbs up/down, "regenerate" clicks, edits to AI drafts.
• Implicit signals: conversation abandonment, escalation to a human, task completion.
• Automated sampling: run your LLM judge on a random sample of live traffic daily.
• Drift: shifts in input topics, lengths or languages that your golden set doesn't cover.

Feed interesting production failures back into the golden dataset. That loop is what makes quality compound over time.

#A minimal starter plan

If you have nothing today, do this in your first week:

1. Collect 50 real inputs and write expected outcomes for each.
2. Write code checks for anything structured.
3. Add one LLM-judge rubric for the main subjective quality, and hand-check 20 judgments.
4. Put it in a script that outputs a table and a score, and run it before every change.
5. Wire it into CI when it stabilizes.

#FAQ

#How many test cases do I need?

Start with 50–200 high-quality, diverse cases. That's enough to catch most regressions. Grow the set as you find failures in production. Coverage of important categories matters more than raw size.

#Can I trust an LLM to grade another LLM?

With a clear rubric and calibration against human labels, LLM judges can agree with humans well enough for regression testing. Without calibration, they can be confidently wrong. Always validate on a human-labeled sample.

#What's the difference between evals and benchmarks?

Public benchmarks measure general model capabilities on standardized tasks. Evals measure how well your application performs on your users' tasks. Benchmarks help shortlist models; only your evals tell you which one to ship.

#How do I evaluate an AI agent?

Evaluate the final outcome (task success, correctness), the trajectory (right tools, reasonable number of steps, no unsafe actions), and efficiency (cost, latency). Replay recorded scenarios in a sandbox with mocked tools. See making AI agents reliable.