AI Ops Monitoring Playbook: Keep Production LLMs Healthy

Production LLM systems fail silently: accuracy drifts, costs spike, and latency degrades before users complain. An AI ops monitoring playbook tracks four pillars (latency, cost, quality, errors) with automated alerts and defined incident response so problems get fixed in hours, not weeks.

The Four Monitoring Pillars

1. Latency

Metric	Alert Threshold	Dashboard
P50 response time	> 2 seconds	Real-time
P95 response time	> 5 seconds	Real-time
P99 response time	> 10 seconds	Real-time
Time to first token (streaming)	> 1 second	Real-time
Queue wait time	> 3 seconds	Real-time

Latency spikes often indicate model provider issues, context length problems, or infrastructure bottlenecks.

2. Cost

Metric	Alert Threshold	Dashboard
Cost per query	> 2x 7-day average	Daily
Daily total spend	> 80% of budget	Real-time
Token usage per query	> 2x average	Daily
Cost by model tier	Unusual tier shift	Weekly
Cache hit rate	< 5% (if caching enabled)	Daily

Track cost per query alongside model routing decisions to verify routing saves money without hurting quality.

3. Quality

Metric	Alert Threshold	Dashboard
Eval score (sample-based)	Drop > 5% from baseline	Daily
User thumbs down rate	> 10% of sessions	Daily
Escalation to human rate	> 40% (support bots)	Daily
Hallucination flag rate	> 5% of responses	Daily
Empty/error response rate	> 2%	Real-time

Quality monitoring requires sampling. Evaluating every response is too expensive. Sample 1-5% of production traffic daily.

4. Errors and Reliability

Metric	Alert Threshold	Dashboard
API error rate	> 1%	Real-time
Timeout rate	> 0.5%	Real-time
Rate limit hits	> 0 (any)	Real-time
Guardrail rejection rate	> 15%	Daily
Uptime	< 99.5%	Monthly

Monitoring Architecture

Production LLM calls
  → Structured logging (input hash, output, latency, tokens, model, cost)
  → Log aggregator (Datadog, Grafana, custom)
  → Dashboards (four pillars)
  → Alert rules
  → On-call notification (PagerDuty, Slack)
  → Incident response runbook

What to Log (Never Log Raw PII)

{
  "timestamp": "2025-09-01T14:30:00Z",
  "request_id": "abc-123",
  "user_id_hash": "sha256...",
  "query_length": 45,
  "model": "gpt-4o-mini",
  "tier": "simple",
  "input_tokens": 1200,
  "output_tokens": 350,
  "latency_ms": 1800,
  "cost_usd": 0.003,
  "retrieval_score": 0.87,
  "guardrail_passed": true,
  "user_feedback": null
}

Hash or truncate user inputs. Store full inputs only with explicit consent and retention policies.

Alert Configuration

Critical (Page Immediately)

• API error rate > 5% for 5 minutes
• Daily budget exceeded
• Uptime drop below 99% in rolling hour
• Security guardrail breach (prompt injection success)

Warning (Slack Notification)

• P95 latency > 5 seconds for 15 minutes
• Eval score drop > 3%
• Cost per query > 1.5x average
• Escalation rate spike > 20% above baseline

Informational (Daily Digest)

• Daily cost summary
• Top 10 failed queries
• Model tier distribution
• Cache performance

Drift Detection

LLM quality drifts from:

• Model provider updates: OpenAI/Anthropic silently update models
• Data changes: Knowledge base content added, removed, or modified
• Usage pattern shifts: New user segments with different query types
• Prompt degradation: Unauthorized prompt changes in production

Detection methods:

1. Daily sample eval: Run 50-100 production queries through eval suite
2. Embedding drift: Compare query embedding distribution week-over-week
3. Output length monitoring: Sudden changes in average response length
4. User feedback trends: Declining satisfaction over 7-day rolling window

When drift detected: investigate, re-run full eval suite, rollback if needed.

Incident Response Runbook

Severity 1: System Down or Major Quality Failure

1. Acknowledge alert (5 min)
2. Check provider status page
3. Rollback to last known good prompt/model version
4. Notify stakeholders
5. Root cause analysis within 24 hours
6. Add failing case to eval suite

Severity 2: Degraded Performance

1. Acknowledge alert (30 min)
2. Identify affected query types
3. Adjust routing or increase model tier temporarily
4. Schedule fix within 48 hours

Severity 3: Cost Anomaly

1. Review within 4 hours
2. Check for traffic spike, loop bug, or routing failure
3. Apply budget guardrails if needed
4. Optimize within 1 week

Cross-reference with LLM evals checklist for quality recovery procedures and prompt optimization for drift fixes.

Need AI ops monitoring built for your production systems? TopAhead’s Managed AI Ops service sets up dashboards, alerts, and incident runbooks.

FAQ

Which monitoring tools work for LLM systems? LangSmith, Braintrust, Helicone, and Datadog (with custom metrics) are popular. Custom Grafana dashboards work for teams with existing observability stacks.

How much does monitoring add to ops cost? $100-$500/month for tools plus 2-5 hours/week for review. Far cheaper than undetected quality degradation.

Should we monitor every query or sample? Log every query (metadata only). Eval quality on 1-5% sample daily. Full eval suite weekly.

How do we monitor RAG-specific issues? Track retrieval scores, chunk relevance, and citation accuracy separately from generation quality.

What is the minimum viable monitoring setup? Daily cost tracking, error rate alerts, and weekly manual review of 20 random conversations. Expand from there.