Running Local LLMs: 24–32 GB

Why it matters: More memory opens bigger models and longer contexts, but also invites bad conclusions like "more RAM = better answers." Learning to test matched workloads keeps hardware spending honest.

By the end you can:

• Say what extra memory changes and what it does not
• Compare quantization levels on the same task
• Report a speed claim with the conditions that make it reproducible
• Plan a simple multi-user check

Learn (15 min)

1. Memory is capacity, not quality (5 min)

Extra memory lets you load larger models, use higher-precision quantization, or hold longer contexts. Answer quality comes from the model, its training, quantization, and your prompt. Speed depends on memory bandwidth, runtime, and how much work lands on GPU vs. CPU.

(Approximate; always verify with the exact file and your runtime.)

2. Pick candidate builds deliberately (5 min)

Choose by task, license, runtime support, file size, and evidence from your own evaluation (Lesson 3). Quantization lowers numeric precision to save memory; quality loss varies by model and task, so compare two quantization levels of the same model before trusting a smaller one. Record the exact tag or revision, since "latest" changes.

3. Measure properly (5 min)

Use a fixed test set. Record time to first token, total time, tokens/second, peak memory, and failures at normal and long context. A speed number without hardware, model, quantization, runtime, context length, and method is not a claim you can compare. For shared use, add concurrency: two simultaneous users can more than halve each user's speed.

Apply (15 min)

Worked example (3 min). A 32 GB desktop summarizes project documents. Compare a ~8B model at 8-bit against a ~14B model at 4-bit on 10 identical questions. Score factual coverage, omissions, unsupported claims, speed, and memory. A human reviewer blind-ranks outputs. Result: the 14B at 4-bit wins on coverage but is 40% slower; the team chooses by whether the task is interactive or batch.

Lab (7 min): Quantization comparison.

1. Pick one model family and download two quantization levels (e.g., 4-bit and 8-bit).
2. Run the same five prompts on both; time each.
3. Score answers for correctness and omissions (0–2 each).
4. Re-run one prompt with a long pasted document on each.

Deliverable: A table: quantization | file size | peak memory | tokens/sec | quality score | long-context result. You're done when: Your table lets a reader choose a build for a stated use case.

Check yourself (5 min)

1. Does 32 GB mean a more accurate model than 16 GB? Explain.
2. What makes "42 tokens/second" a weak claim?
3. Scenario: Your test shows 4-bit and 8-bit answers are indistinguishable on your task. Which do you use and why?
4. Why test long contexts?
5. What changes when three colleagues use the same machine?

Reference

Common mistakes

Cheat sheet: Memory = capacity. Compare matched workloads. Record model, quantization, context, runtime, hardware. Test concurrency before sharing.

Key terms: memory bandwidth, offload, throughput, concurrency, model revision.

Interview prompts: How would you benchmark two quantizations? What would make you distrust a published speed number?

Go deeper: Ollama FAQ · LM Studio docs · Hugging Face quantization overview

Next: Lesson 6 — high-memory systems and shared services.