Running Local LLMs: 8–16 GB

Why it matters: A local model can handle private drafts, offline work, and high-volume simple tasks without per-token fees. But on a laptop with 8–16 GB, whether it works depends on arithmetic you can do in your head.

By the end you can:

• Estimate whether a model's weights will fit
• Account for context (KV cache) and runtime overhead
• Run a controlled trial and record the setup
• Explain why "local" does not automatically mean "private"

Learn (15 min)

1. What "local" gives and doesn't (5 min)

Local inference runs the model on hardware you control. Benefits: fewer vendor dependencies, offline use, predictable cost. Limits: memory, speed, heat, setup effort, and smaller models. It is not automatically private: check extensions, telemetry, logs, and any connected tools.

2. The sizing math (5 min)

Weights ≈ parameters × bits per weight ÷ 8.

Real files run somewhat larger (quantization metadata, embeddings). Then add:

• KV cache (grows with context). Illustrative for an 8B-class model with grouped-query attention: about 128 KB per token at 16-bit, so an 8,000-token context is roughly 1 GB.
• Runtime buffers and your operating system and other apps.

Rule of thumb for this tier: leave several GB free. A 16 GB machine comfortably targets ~7–8B models at 4–5 bit with moderate context, and an 8 GB machine targets ~3–4B models. Treat these as trial ranges, not guarantees.

3. Run and record a controlled test (5 min)

With a runtime such as Ollama or LM Studio, record: model name and quantization, runtime version, OS, CPU/GPU, RAM, context length, and whether it ran on CPU, GPU, or both.

ollama list                 # models you have and their sizes
ollama run <model> --verbose  # prints timing stats after each reply
ollama ps                   # shows memory use and the CPU/GPU split of loaded models

A model that loads can still be unusably slow or fail on long inputs. Test with realistic prompts and known answers.

Apply (15 min)

Worked example (3 min). A 16 GB laptop should summarize meeting notes. Plan: pick a ~7–8B model at 4-bit (~4–5 GB file), set context to 4,000 tokens, confirm the app makes no network calls during inference, test on three non-sensitive notes with known key points, and score coverage and omissions. Result: acceptable summaries in ~20 seconds; the model misses action items in long notes, so the team limits input length.

Lab (7 min): Fit and speed test.

1. Choose a small model (3B–8B) from a runtime's library; note its file size.
2. Run it on five identical prompts (short and long). Capture load time, response time, and ollama ps memory.
3. Re-run with a longer pasted document. Note any slowdown or failure.
4. Write a setup record using the fields above.

Deliverable: A one-page record: setup, results table, and a verdict ("fits for X, not for Y"). You're done when: Someone else could reproduce your test from your record.

Check yourself (5 min)

1. Estimate the 4-bit weight size of a 14B model.
2. Does 16 GB RAM guarantee a 12 GB model file runs well? Why?
3. Scenario: A tool says it's "100% local" but works only when online. What do you check?
4. What does ollama ps help you see?
5. Name three details to record for a fair comparison.

Reference

Common mistakes

Cheat sheet: weights ≈ params × bits ÷ 8, then add cache, buffers, OS; leave headroom; test on real prompts; record the setup.

Key terms: local inference, quantization, KV cache, VRAM, unified memory.

Interview prompts: Estimate local fit for a given model. Compare local vs. hosted for sensitive notes. How would you reproduce a colleague's benchmark?

Go deeper: Ollama FAQ · Ollama model library · LM Studio docs · Hugging Face quantization overview

Next: Lesson 5 — what changes with 24–32 GB.