LLM Evals Production Checklist: 25 Steps Before You Ship

LLM evals are the difference between AI features that work in demo and AI features that work in production. Run through this 25-point checklist before shipping any LLM-powered system to users.

Phase 1: Test Set Creation (6 points)

#	Checklist Item	Pass/Fail
1	50+ test cases covering core use cases
2	20+ edge cases (empty input, long input, ambiguous queries)
3	10+ adversarial cases (prompt injection, off-topic, harmful requests)
4	Expected outputs defined (exact match, contains, or rubric-scored)
5	Test cases sourced from real user queries (not just synthetic)
6	Test set versioned in git alongside code

Minimum viable: 50 core + 20 edge + 10 adversarial = 80 test cases.

Phase 2: Metrics Definition (5 points)

#	Checklist Item	Pass/Fail
7	Primary metric defined (accuracy, relevance, or task completion)
8	Secondary metrics tracked (latency, cost, token usage)
9	Pass thresholds set (e.g., > 85% accuracy on core cases)
10	Failure categories tagged (hallucination, wrong format, incomplete, off-topic)
11	Baseline score recorded before optimization

Phase 3: Automated Eval Pipeline (6 points)

#	Checklist Item	Pass/Fail
12	Eval script runs against test set programmatically
13	LLM-as-judge configured for open-ended outputs
14	Eval runs in CI on every prompt or model change
15	Regression detected: score drop > 3% blocks deployment
16	Eval results logged with timestamp, model version, and prompt version
17	Comparison view: current vs. previous eval run

Tools: Braintrust, LangSmith, Promptfoo, or custom scripts. Pick one and use it consistently.

Phase 4: Pre-Production Validation (4 points)

#	Checklist Item	Pass/Fail
18	Full eval suite passes at or above thresholds
19	Latency tested at expected load (P50, P95, P99)
20	Cost per query calculated and within budget
21	Human review of 20 random outputs confirms eval scores

Phase 5: Production Monitoring (4 points)

#	Checklist Item	Pass/Fail
22	Production logging captures inputs, outputs, latency, and cost
23	Sample-based eval running daily on live traffic (1-5%)
24	Alert configured for accuracy drop or error rate spike
25	Rollback procedure documented and tested (revert prompt/model)

Eval Metrics Explained

For Q&A / RAG Systems

Metric	How to Measure	Target
Retrieval accuracy	Correct doc in top-3	> 85%
Answer accuracy	Matches expected or passes rubric	> 80%
Hallucination rate	Claims not in retrieved context	< 5%
Citation accuracy	Sources match claims	> 90%

For Classification / Extraction

Metric	How to Measure	Target
Exact match accuracy	Output matches expected label	> 90%
F1 score	Precision/recall balance	> 0.85
Format compliance	Valid JSON/structure	> 95%

For Generation (Content, Email, Copy)

Metric	How to Measure	Target
Rubric score	LLM-as-judge on 1-5 scale	> 4.0 average
Brand compliance	Matches style guide checklist	> 90%
Factual accuracy	Claims verified against sources	> 95%

Running Evals in CI

Example workflow:

On pull request:
  → Run eval suite against changed prompts
  → Compare scores to main branch baseline
  → Block merge if primary metric drops > 3%
  → Post eval report as PR comment

On deploy:
  → Run full eval suite (including adversarial)
  → Deploy only if all thresholds pass
  → Log eval results for audit trail

Common Eval Mistakes

1. Testing only happy paths: Edge cases cause 80% of production failures
2. No adversarial tests: Prompt injection attacks are real
3. Eval set never updated: Stale tests miss new failure modes
4. Manual-only evals: If evals require a human every time, they will not run consistently
5. No production monitoring: Pre-launch evals do not catch drift from model updates or data changes

Pair evals with model routing for cost control and ops monitoring.

Building a custom app? See custom AI app vs. wrapper for architecture decisions.

Need eval infrastructure built for your LLM features? TopAhead’s AI Software service delivers test suites, CI integration, and production monitoring.

FAQ

How many test cases do we really need? Start with 50. Expand to 100-200 as you discover failure modes in production. Add a new test case for every bug reported.

Should we use LLM-as-judge or human eval? LLM-as-judge for speed and scale (80%+ correlation with human judgment). Human eval for calibration monthly and for high-stakes features.

How often should evals run? CI on every change. Daily sample on production traffic. Full suite weekly.

What if eval scores are good but users complain? Your test set does not cover real usage. Add failing user queries as test cases immediately.

Do evals work for image/audio models? Yes with modality-specific metrics (CLIP score, transcription accuracy, etc.). Same framework applies.