Running models on your own machine

Getting startedCopy Anchor Link

Install Ollama or LM Studio, then pull a model with one command. Choose a size your memory can hold. As a rule, a 4-bit model needs about half a gigabyte per billion parameters plus some overhead. So 8 GB of memory fits roughly a 7 to 8 billion parameter model. Unified memory on Apple Silicon lets the GPU use system memory, which is why laptops handle larger models than a typical PC graphics card.

What to use them forCopy Anchor Link

Classification, extraction, summaries and drafting where the data should not leave your machine. Local models are also a cheap test bed for prompts and tool use. Expect weaker results than the frontier on hard reasoning and coding. Most local runners expose an API compatible with the common cloud APIs, so your code can switch with a changed URL.

A worked example: what fitsCopy Anchor Link

A 7 to 8 billion parameter model at 4-bit is about 4.5 GB, comfortable on a laptop with 16 GB of memory. A 14B model is about 8 GB, tighter. A 70B model at 4-bit is about 40 GB, needing a workstation. Start with the small one and see if it is good enough for your task.

A memory budget in practiceCopy Anchor Link

The rule of half a gigabyte per billion parameters ignores everything else on the machine. On a 16 GB laptop, the system and open apps may take 4 to 5 GB (invented, check your own). That leaves about 11 GB. A 14B model at 4-bit needs about 7 GB, and the context cache adds one or two more, so it fits. A 32B model needs 32 × 0.5 = 16 GB and does not.

What goes wrong: it slows to a crawlCopy Anchor Link

If the model plus its cache is larger than free memory, the system pushes part of it to disk. The symptom is words arriving every few seconds and a laptop that gets hot and stuck. To confirm, open the memory monitor while it generates and look for heavy swap use or memory pressure in the red. Choose a smaller model or a more compressed version. Another symptom is a model that ignores the start of a long document. Many runners start with a short context window by default, so early text is silently dropped. Paste a document with a fact in the first line, ask about it, and raise the context setting if the answer misses.

Try it yourself: is it good enoughCopy Anchor Link

Write 20 real prompts from your task and run them on the local model and a cloud model. Mark each answer acceptable or not. If the local model passes 18 of 20 and the cloud one 19, the local model is good enough. Also note speed. Around 5 to 8 tokens a second is close to reading speed, so slower than that feels sluggish.

When the cloud winsCopy Anchor Link

One laptop serves one request at a time well. If a hundred people need answers together, or the task needs hard multi-step reasoning, a hosted model fits better.

Common mistakeCopy Anchor Link

Expecting a laptop model to match the largest cloud models.