What an LLM Is and How It Works

Why it matters: Every AI tool you will use is built on the same mechanism, and its biggest weakness, confident wrong answers, comes directly from how it works. Understanding the mechanism tells you when to trust output and when to check it.

By the end you can:

• Explain tokens and next-token prediction in plain language
• Tell training from inference, and say what a chat does and does not change
• Explain why fluent output can still be wrong
• Decide when an answer needs outside verification

Learn (15 min)

1. Text becomes tokens (5 min)

A tokenizer splits text into tokens: whole words, word pieces, punctuation, or spaces. The model never sees letters the way you do; it sees token IDs.

Exact splits differ by model. Token count, not word count, drives cost, speed, and the context window (how much text the model can consider at once). A rough English rule of thumb is that 100 tokens is about 75 words, but code, names, and non-English text usually use more tokens.

2. The model predicts a continuation (5 min)

A Transformer reads all tokens in its context, uses attention to weigh which earlier tokens matter, and outputs a probability for every possible next token. One token is chosen, appended, and the loop repeats.

Prompt:  "The capital of New York State is"
Step 1:  Albany 71% | New 9% | Manhattan 6% | ...   → picks "Albany"
Step 2:  (context now includes "Albany") "." 62% | "," 20% | ...

Settings such as temperature change how adventurous the choice is. Low temperature favors the most likely token (steadier); high temperature allows less likely choices (more varied). Probability measures what is likely to be written, not what has been checked. That gap is where hallucinations live.

3. Training vs. inference, and what apps add (5 min)

• Training adjusts the model's parameters using large amounts of text and objectives. Post-training (instruction tuning, preference tuning) shapes how it responds.
• Inference is using the finished model to generate output. An ordinary chat does not retrain the model.
• Apps add things around the model: your conversation history, uploaded files, search, retrieval, and tools. When a chatbot "knows" today's news, an app layer supplied it.

Apply (15 min)

Worked example (3 min). Question: "What is our refund window?" asked of an assistant with no policy file connected.

• What the model has: your prompt and general patterns about refund policies.
• What is missing: your current policy.
• Likely failure: a plausible, specific, invented window like "30 days."
• Safe fix: retrieve the approved policy, put it in the context, and require the answer to quote it or say "not found."

Lab (7 min): Watch tokens and confidence in action.

1. Open any free tokenizer tool (search for "tokenizer playground" for your model family).
2. Paste three strings: a normal sentence, your business name, and a date plus phone number. Record token counts next to word counts in a small table.
3. In any chat assistant, ask: "What is the exact refund policy of [a small local business near you]?" Note whether it invents details or asks for a source.
4. Re-ask with a pasted policy paragraph and the instruction "Answer only from the text below. If it isn't there, say so."

Deliverable: A 3-row token table and two sentences comparing the invented vs. grounded answers. You're done when: You can point to the exact difference in the prompt that changed the behavior.

Check yourself (5 min)

1. What is a token, and why does it matter for cost?
2. A model gives a precise statistic with a made-up source. What mechanism explains this?
3. Scenario: A colleague says "I corrected the AI yesterday, so it won't make that mistake today." What's wrong with that belief?
4. True or false: more context always improves answers.
5. Which needs a live, authoritative source: (a) explaining photosynthesis, (b) today's mortgage rate, (c) the meaning of "token"?

Reference

Common mistakes

Cheat sheet: Text → tokens → context → next-token probabilities → one token → repeat. Training builds the model; inference uses it; apps add context and tools. Verify anything current, consequential, or numeric.

Key terms: token, tokenizer, Transformer, attention, context window, temperature, inference, hallucination.

Interview prompts: Explain next-token prediction to a non-technical manager. Why is fluency not evidence? How does an app give a model current information? What can go wrong with very long contexts?

Go deeper: Attention Is All You Need · Hugging Face LLM Course · Transformers generation guide

Next: Lesson 2 — turn this understanding into reliable prompts.