RAG Architecture Fundamentals

Why it matters: RAG is how most business AI assistants answer from private, current documents. It reduces invented answers only when retrieval, permissions, and evaluation are done well.

By the end you can:

• Explain the RAG pipeline end to end
• Choose sensible chunking and metadata
• Diagnose retrieval failures separately from generation failures
• Design a small RAG evaluation set

Learn (15 min)

1. Why retrieval (5 min)

A model's parameters are not a reliable, current database of your documents. Retrieval-augmented generation (RAG) searches an approved collection at question time, gives the best passages to the model, and asks it to answer from them with citations. It does not guarantee correctness: retrieval can miss, sources can be stale, and the model can misread evidence.

2. The pipeline (5 min)

INDEXING (offline)                      QUERYING (live)
approved docs → extract text →          user question → access check →
split into chunks → add metadata →      retrieve top-k chunks → (rerank) →
embed / index → store                   build prompt → generate → cite sources

• Chunking: commonly a few hundred tokens with some overlap; split on headings when possible. Too small loses context; too large dilutes relevance.
• Metadata: source, section, effective date, department, permissions.
• Retrieval: embeddings (meaning-based), keyword search, or both (hybrid).
• Permissions: filter by the user's access before retrieval, not after generation.

3. Ground and evaluate (5 min)

Instruct: answer only from the supplied passages; say "not found" otherwise. Evaluate two things separately:

Include questions with no answer, conflicting documents, and outdated versions.

Apply (15 min)

Worked example (3 min). HR policy assistant. Ingest approved PDFs with an effective_date field; filter by employee department; retrieve top 5; prompt: "Use only the passages. If two passages conflict, prefer the later effective date and mention both." Test: "How many vacation days do new hires get?" Retrieval returns the 2024 and 2026 policies; the answer uses 2026 and notes the change.

Lab (7 min): Paper RAG.

1. Take 5 short documents (FAQ entries, policies). Split into chunks of ~3–5 sentences; label each chunk with source and date.
2. Write 8 questions: 5 answerable, 1 unanswerable, 1 conflicting, 1 outdated.
3. For each, manually list which chunks should be retrieved.
4. Run a free AI chat with the chunks pasted in and the grounded prompt; mark answers correct/incorrect and cite-supported/unsupported.

Deliverable: An evaluation table: question | expected chunk | answer | grounded? | notes. You're done when: You can say whether each failure was a retrieval or generation failure.

Check yourself (5 min)

1. What does RAG add at answer time?
2. Why filter by permissions before retrieval?
3. Scenario: The answer is wrong, but the correct passage was in the retrieved set. Where is the failure?
4. Why include unanswerable questions in tests?
5. Does using a vector database guarantee accuracy?

Reference

Common mistakes

Cheat sheet: Approved sources → chunk + metadata → index → permission filter → retrieve → ground → cite → evaluate retrieval and generation separately.

Key terms: retrieval, embedding, chunk, reranking, hybrid search, grounding.

Interview prompts: Trace a RAG query. How do you handle conflicting sources? How do you debug poor retrieval?

Go deeper: RAG paper (Lewis et al.) · OpenAI Retrieval guide · NIST AI RMF

Next: Lesson 8 — connect models to tools and data with MCP.