Draft — in editorial review, not yet published. Content has not been approved by a human reviewer.

Lesson 6 / 50 · Version 2.2.0

Running Local LLMs: 64–128 GB

Level: Intermediate
Duration: 15 min learn + 15 min apply
Prerequisites:: Lessons 3–5

Why it matters: Big-memory machines tempt teams to host a shared "private ChatGPT." Whether that works depends on architecture, concurrency, and security controls, not just whether a model loads.

By the end you can:

Compare GPU VRAM, multi-GPU, unified-memory, and CPU setups
Separate "the model fits" from "the service is ready"
Plan a realistic benchmark with concurrency
List the controls a shared service needs

Learn (15 min)

1. Define the workload first (5 min)

High memory can mean larger models, higher precision, very long context, or many concurrent users, and these needs differ.

Workload	What matters most
One researcher, one large model	Memory capacity, quality
Team assistant for 4–10 people	Concurrency, latency, isolation
Batch processing overnight	Throughput, not latency
Long-document analysis	Context memory and prompt-processing speed

2. Memory architectures (5 min)

Dedicated GPU(s): very fast VRAM, but capacity is the installed total. Multi-GPU needs runtime support, interconnect bandwidth, power, and cooling.
Unified memory: CPU and GPU share one pool. Large capacity, but usable share and bandwidth vary by hardware and runtime.
CPU + system RAM: works for large models but is typically much slower.
Do not add RAM and VRAM as if they were one pool; spillover usually works but slows generation sharply.

3. Measure the whole service (5 min)

Benchmark realistic prompts at realistic concurrency. Record prompt-processing speed (matters for long inputs), generation speed, peak memory, errors, and thermals. For a shared service also test queues, cancellation, authentication, per-user isolation, log retention, and what happens on a model update.

Apply (15 min)

Worked example (3 min). Four researchers want a shared assistant. Requirement sheet: simultaneous users (4), document size (≤50 pages), response time (<30 s for a summary), privacy rule (no prompts retained beyond 7 days), license check, authentication through company SSO, per-user chat isolation, backup of configs, monthly update window. Result: a single large model fits, but at four concurrent long-document requests latency triples. Team adds a queue and a smaller model for quick questions.

Lab (7 min): Write a service-readiness checklist.

Pick a hypothetical shared deployment (team, documents, privacy needs).
List ten checks across: capacity, concurrency, latency target, authentication, authorization, isolation, logging/retention, updates, backup, cost.
For each, write how you would test it (not just that it matters).

Deliverable: A 10-row table: check | why it matters | how to test | pass criterion. You're done when: Each row has a pass criterion a colleague could verify.

Check yourself (5 min)

Why isn't a memory tier a guarantee that a model works well?
What is the risk of a model spilling from VRAM into system RAM?
Scenario: A demo works for one user. What must you test before offering it to 20 employees?
Name three controls a shared service needs.
What must appear in a performance claim?

Reference

Common mistakes

Symptom	Likely cause	Fix
Demo works, rollout stalls	Never tested concurrency	Load-test with realistic users
Slow after a minor change	Spill to system RAM	Check `ps`-style memory/placement output
Data leaks between users	Shared sessions	Per-user isolation and access checks

Cheat sheet: Workload → architecture → runtime support → realistic benchmark → security controls → monitoring.

Key terms: model parallelism, interconnect, peak memory, queueing, session isolation.

Interview prompts: Compare GPU and unified-memory setups for a team assistant. How would you design logging that protects privacy?

Go deeper: Ollama FAQ · Hugging Face Accelerate big-model inference · NVIDIA CUDA Programming Guide

Next: Lesson 7 — give models your own documents with RAG.