Model Selection and Benchmarking

Why it matters: Leaderboards rank models on someone else's tasks. A small test set built from your own work is the fastest way to avoid paying for the wrong model, or trusting a cheap one that fails where it counts.

By the end you can:

• Define selection criteria and mark which are mandatory
• Build a small evaluation set with common, edge, and "should ask for help" cases
• Score candidates consistently and record what you tested
• Weigh quality against cost, speed, privacy, and risk

Learn (15 min)

1. Start with the task and the cost of errors (5 min)

Write one sentence: "The system must ___ for ___." Then ask what a wrong answer costs. A mislabeled social caption is cheap; a wrong invoice total is not. Split criteria into must-haves (e.g., data stays in region, valid JSON) and tradeoffs (cost, speed, tone).

2. Build and run a fair test (5 min)

Use 20–50 real or anonymized examples: mostly typical, some edge cases, a few where the right behavior is to decline or ask. Run every candidate on the same inputs with equivalent settings. Record model ID, prompt version, settings, and date.

AI graders can help triage but may share the system's blind spots; sample their grades with a human on anything consequential.

3. Include operational fit (5 min)

Quality is one column. Also measure end-to-end latency (including retrieval and retries), cost at your real volume, rate limits, data-handling terms, licensing, and availability. Re-run the test whenever the model, prompt, data, or policy changes.

Apply (15 min)

Worked example (3 min). Invoice extraction, 100 invoices/day. Rubric: field accuracy (vendor, date, number, total) 40% · handles missing/conflicting fields by flagging 20% · valid structured output 15% · poor-scan performance 10% · cost per 1,000 invoices 10% · privacy terms pass/fail. Result: Model A is 2% more accurate but 6× the cost; the team keeps B and routes low-confidence invoices to a person.

Lab (7 min): A ten-case bake-off.

1. Choose a task you actually do (classify inquiries, summarize notes, extract fields).
2. Write 10 test cases: 6 typical, 2 edge, 2 "should ask or refuse."
3. Run two models with the same prompt. Score each 0/1 against your expected result.
4. Compute accuracy, note the worst failure, estimate cost per 1,000 runs.

Deliverable: A comparison table and a one-paragraph recommendation. You're done when: Your recommendation names the failure that would matter most in production.

Check yourself (5 min)

1. Why include "should ask for help" cases?
2. Two models get different prompts in your test. What is wrong?
3. Scenario: A leaderboard leader scores 3% higher than your current model on your test but costs 8× more and has unclear data terms. How do you decide?
4. What should you record so a result can be reproduced?
5. When do you re-run the evaluation?

Reference

Common mistakes

Cheat sheet: Task → criteria → representative cases → same inputs → score → measure ops → document → re-test on change.

Key terms: benchmark, evaluation set, rubric, regression, latency, model-graded evaluation.

Interview prompts: How would you pick a model for support triage? How do you test refusal behavior? How do you detect a regression after a vendor update?

Go deeper: OpenAI Evals · Stanford HELM · NIST AI Risk Management Framework

Next: Lesson 4 — try a model on your own hardware.