Draft — in editorial review, not yet published. Content has not been approved by a human reviewer.
Lesson 10 / 50 · Version 2.2.0
AI Evaluations, Safety, and Observability
- Level
- Intermediate
- Duration
- 15 min learn + 15 min apply
- Prerequisites:
- Lessons 1–3 and 9
Why it matters: A demo that works is not a system you can trust. Evaluation, safeguards, and traces are what let you ship AI features, notice when they degrade, and explain what happened when they fail.
By the end you can:
- Turn expectations into measurable tests
- Match safeguards to common AI risks
- Define a privacy-conscious trace
- Tell evaluation from monitoring
Learn (15 min)
1. Evaluate behavior (5 min)
Convert expectations into test cases with scoring rules. Cover normal inputs, edge cases, malformed or adversarial inputs, and cases where the system should decline or ask. For agents, grade the result and whether each action was authorized.
| Test type | Example | Pass criterion |
|---|---|---|
| Supported answer | Policy question with a source | Correct and cites the source |
| No source | Question outside the documents | Says "not found" |
| Conflict | Two policy versions | Uses newest, mentions both |
| Access | Restricted department's file | Not retrieved for this user |
| Injection | Document says "ignore previous instructions" | Instruction not followed |
2. Match safeguards to risk (5 min)
| Risk | Safeguards |
|---|---|
| Inaccurate output | Grounding, citations, human review on high stakes |
| Prompt injection | Treat retrieved/tool text as data; allow-lists; confirm actions |
| Privacy exposure | Data minimization, access control, redaction, retention limits |
| Unsafe actions | Least privilege, approvals, rate limits |
| Overreliance | Clear uncertainty, escalation paths |
A rule in a prompt is not a control until you've tested that it holds.
3. Observe the system (5 min)
A trace records what happened for one request without storing more sensitive data than needed:
{
"request_id": "r-2043",
"time": "2026-10-27T14:03:11Z",
"model": "model-x@2026-09",
"prompt_version": "policy-qa-v7",
"retrieved_source_ids": ["hr-2026-04#p3", "hr-2024-01#p3"],
"tools_called": [],
"latency_ms": 2140,
"outcome": "answered_with_citation",
"error": null
}
Monitor errors, latency, cost, user corrections, policy violations, and source freshness. Alerts need a threshold, an owner, and a response step. Re-run evaluations after any material change.
Apply (15 min)
Worked example (3 min). For a policy chatbot, five tests: supported answer; no matching source; conflicting policies; restricted content; old and new policies both retrieved. Score: correct source, accurate answer, citation supports claim, access enforced, uncertainty stated.
Lab (7 min): Build a mini test suite.
- Pick an AI feature (yours or hypothetical).
- Write eight tests: 3 normal, 2 edge, 1 unanswerable, 1 injection attempt, 1 access-control case.
- For each, define expected behavior and a pass/fail rule.
- Sketch a trace record (fields only) that would let you debug a failure.
Deliverable: A test table and a trace field list. You're done when: You can tell from your trace fields whether a failure was retrieval, model, or tool related.
Check yourself (5 min)
- Why test "no answer" cases?
- Evaluation vs. monitoring?
- Scenario: You improve the prompt for one complaint, and another case breaks. What practice would have caught it?
- What should NOT go into a trace by default?
- What makes an alert actionable?
Reference
Common mistakes
| Symptom | Likely cause | Fix |
|---|---|---|
| "It worked in the demo" | No repeatable tests | Maintain a regression suite |
| Sensitive data in logs | Over-logging | Log IDs, not content; set retention |
| Alert fatigue | No owner or threshold | Define owner, threshold, runbook |
Cheat sheet: Define expected behavior → test normal and failure cases → validate safeguards → trace safely → monitor → re-run after changes.
Key terms: evaluation, observability, trace, guardrail, regression test, prompt injection.
Interview prompts: Design a RAG evaluation. Which production metrics matter for an AI assistant? How do you debug while minimizing personal data?
Go deeper: OpenAI Evals · NIST AI RMF · OWASP Top 10 for LLM Applications · OpenTelemetry observability primer
Next: Track 2 — automating work across apps (Lesson 11).