A 32 GB machine with no GPU runs a useful model. Not the useful model — the other one. Here is the arithmetic I actually use, after running this on a Xeon E3-1275 v5 with DDR3 and a fan that never spins up.
What fits, in one table
- 7–8B at Q4_K_M — around 5 GB resident, ~18 tok/s. Reads logs, classifies failures, drafts playbooks. This is the workhorse.
- 14B at Q4_K_M — around 9 GB, ~7 tok/s. Noticeably better at YAML and at not inventing flags. Still comfortable.
- 32B at Q4_K_M — around 19 GB, ~2 tok/s. Fine for a nightly batch, unusable interactively.
- Anything at Q8 — double the memory for a quality gain you will not measure on a shell command.
The rule that matters
Leave RAM for the page cache. A model that evicts the filesystem cache to stay resident makes the whole host slower, and you will blame the storage.
ollama run qwen3:8b
# watch RSS while it answers, not while it loads
watch -n1 'ps -o rss=,cmd= -C ollama | sort -rn | head -3'Quantisation is not a religion
Q4_K_M is the default because it is the point where the quality curve flattens and the memory curve does not. If a task fails at Q4 and succeeds at Q6, the task is out of scope — a bigger quantisation is a way of not fixing the prompt.
If the answer has to be right, the model is the second opinion. The machine is the first.
Measure throughput once, on your own hardware, with the prompt you actually send. Every published tok/s number was measured on somebody else’s box, with a shorter prompt, in a cooler room.