How to run LLMs locally and on edge devices
Tools like Ollama and llama.cpp download a quantized open-weight model and run it on your own machine, with an API your code can call. Small models (1 to 8 billion parameters at 4-bit) run on most modern laptops, and larger ones need a GPU or an Apple Silicon machine with enough memory. Run locally when data can't leave the machine, when you need to work offline, or when you want zero per-token cost for development and narrow tasks, and match the task to the model, because local models are smaller than frontier APIs.
Why run locally
- Privacy: prompts and documents never leave your machine or your network.
- Offline and air-gapped environments, like client servers with no internet access.
- No per-token cost for development, testing and high-volume narrow tasks.
- Control over the exact model version.
The trade-off is capability: a model that fits on a laptop is smaller and weaker than a frontier API model. Classification, extraction, summarisation, drafting and RAG over a narrow domain work well. Open-ended reasoning, complex coding and long multi-step agent tasks work less well.
Model sizes
The "B" in a model's name is billions of parameters, the numbers the model learned in training. An 8B model has about 8 billion of them. At 4-bit quantization, a model needs roughly 0.5 to 0.7 GB of memory per billion parameters, plus memory for the context.
| Model size | Memory at 4-bit (approx.) | Runs on |
|---|---|---|
| 1B to 3B | 1 to 2.5 GB | Most laptops, phones |
| 7B to 8B | 4 to 6 GB | Laptops with 16 GB RAM |
| 13B to 14B | 8 to 10 GB | Laptops with 32 GB, single consumer GPU |
| 30B to 70B | 18 to 45 GB | Workstations, large Apple Silicon machines, server GPUs |
See quantization for why 4-bit is the common default.
Ollama
Ollama is the simplest way to start: an engine that downloads and runs models with one command, and serves them on a local API.
ollama pull llama3.2:3b
ollama run llama3.2:3b "Summarise this paragraph: ..."
ollama list
Your code calls the local API:
import json
from urllib import request
payload = {"model": "llama3.2:3b", "prompt": "Classify this email as billing, technical or other: ...", "stream": False}
req = request.Request("http://localhost:11434/api/generate", data=json.dumps(payload).encode(), headers={"Content-Type": "application/json"})
with request.urlopen(req) as resp:
print(json.loads(resp.read())["response"])
Ollama also offers an OpenAI-compatible endpoint, so many SDKs and frameworks can switch to a local model by changing the base URL.
llama.cpp and others
- llama.cpp: the C/C++ engine behind many local tools, runs GGUF models on CPUs and GPUs, with fine control over quantization and threads.
- LM Studio: a desktop app for downloading and chatting with local models.
- Open WebUI: a ChatGPT-style web interface for local models, usually run with Docker.
- MLX: Apple's framework for running and fine-tuning models efficiently on Apple Silicon.
- For serving many users on your own GPU server, use an inference server like vLLM instead. See inference serving.
Edge devices
Small models (around 1B to 3B parameters, quantized) can run on phones and edge hardware, through frameworks like llama.cpp, MLC LLM, or the platform's own on-device AI APIs. Expect short contexts, slower generation and narrower abilities, and design the feature around that.
Checking a local model is good enough
Run one feature of your app on a local model, and compare its pass rate with your API model on the same eval set. Measure tokens per second on your target hardware, not on your development machine.
From the floor
In the session on running LLMs locally, we installed Ollama, pulled DeepSeek R1 8B (a 4.9 GB download), added Open WebUI with Docker for a ChatGPT-style interface, and then ran an open-source browser agent on top of the local model. The reasoning model I tried there took around 13 minutes to think through one browser task on my machine, which is the honest trade-off of running a reasoning model locally on a laptop: private and free, and much slower than a hosted API.
Neural Vault, one of the AI Engineer HQ artifacts, applied the same idea to a knowledge engine for company data: local LLMs, fully offline, with no data leaving the organisation's network.