← All drillsRunning and Hosting Models
What you'll be able to do- ✓Measure how batch size changes a model's speed and memory on a CPU, with real timings from your own machine
- ✓Compute how much memory a model's weights need from its parameter count and precision, and the speed ceiling memory bandwidth sets
- ✓Shrink a model with int8 quantisation and measure what it saves and what it costs in quality
- ✓Compare four ways to serve one model (in-process, behind an API, a local runner, a hosted service) and measure what batching and a cache change
- ✓Work out the break-even point between renting hardware and paying per token from your own throughput and quotes
Decide where a model should run and what it will cost, the question every team faces once a model leaves the notebook, using numbers you measured yourself instead of rules of thumb.
⌁ Every company that puts a language model into a product makes this choice between running it themselves, on rented or owned hardware, and paying a hosted service per token, and it is revisited as traffic grows.
Start this internshipCreate an account to unlock the 4 sections, the workbench, and askThili.
BeginSections
1. Why this course: from what you teach to where a model runs
🔒 locked2. What a Model Needs: Memory, Batching and Bandwidth
🔒 locked3. Four Ways to Serve One Model
🔒 locked4. Rent or Pay per Token?
🔒 lockedDig deeper
📄Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023, SOSP, arXiv:2309.06180)
paper🔗Ollama API reference (the local HTTP API shown in Four Ways to Serve One Model)
docs🔗Ollama OpenAI-compatible API (the chat-completions request shape)
docs🔗llama.cpp server (serving a GGUF model locally)
docs🔗EleutherAI Pythia-70m model card (the small model measured in the lessons)
article