A private AI that never phones home is now a five-minute install on almost any phone or PC. This hands-on guide walks through running a local LLM on Android, iPhone, and Windows: how to read your RAM and VRAM limits, which app and model to pick, the exact steps to install and prove it works offline, and the heat, battery, and accuracy trade-offs nobody warns you about.
A few years ago, running a large language model on your own hardware was a weekend project for people with a spare graphics card and a tolerance for Python stack traces. In 2026 it is something you can do on the phone in your pocket during a coffee break. The model that answers you runs entirely on your own chip. Your words never leave the device, the whole thing works in airplane mode, and after the initial download it costs nothing to run, forever.
That last sentence is worth sitting with, because it inverts the deal every cloud chatbot offers you. A cloud model is astonishingly capable and, in exchange, it sees everything you type, needs a live connection, and meters your usage. A local model gives up some of that raw capability and hands you privacy, offline access, and zero marginal cost in return. For a large and growing set of everyday tasks, that is a trade worth making.
This is a practical guide to making it, on three platforms: Android, iPhone, and Windows. Each section stands on its own, so you can skip to the device in your hand. But read the next two minutes first, because one idea and one warning apply everywhere.
The single idea to carry through all three platforms is this: the model has to fit in memory, and memory is the thing you have least of. On a phone that means RAM; on a PC it means your graphics card's VRAM. Everything else, the app you choose, the exact model, the acceleration chip, is secondary to whether the weights fit in the memory available. Get that one number right and the rest is easy.
The one warning, stated plainly because it is the most common disappointment: small local models are confident liars. They are genuinely good at chatting, summarizing, drafting, translating, and brainstorming. They are genuinely unreliable at facts, math, code, long documents, and anything that happened after their training cutoff. Testers of even the newest small models report answers that "sound perfectly coherent but the information is completely wrong." Treat on-device output as a fast first draft written by a bright intern who never says "I don't know," and you will be delighted rather than misled.
Android is the friendliest on-ramp. The best beginner app is free, needs no account, and pulls models straight from the internet inside the app, so you never touch a file manager.
The tempting thing is to look at your phone's chipset and its "AI TOPS" number. Ignore it. The gatekeeper is RAM, because the model's weights have to sit in memory alongside the operating system, which itself reserves roughly 2 to 4 GB before you load anything. What is left over is your real budget.
You do not need a Snapdragon or a headline NPU. Most Android apps run on the CPU or GPU, so general performance and RAM dominate. There is a nuance worth knowing: a few apps (MLC Chat, and newer builds of PocketPal and ChatterUI) can tap the Snapdragon Hexagon NPU or Adreno GPU for a real 2–4× speedup. But on Pixel hardware the Tensor NPU is reserved for Google's own apps and is not exposed to third-party inference apps, so there the "RAM is what matters" rule holds completely.
For almost everyone, the answer is PocketPal AI from the Google Play Store. It is free, open-source, carries no ads or telemetry, needs no account, and has an in-app browser that downloads models directly from Hugging Face. If you want to know your other options, the decision is short:
Open PocketPal, tap into its Hugging Face search, and download one model. This is the only step that needs a connection. The format you want is GGUF at Q4_K_M quantization, which is the standard sweet spot of size, speed, and quality (more on what those words mean later). Match your pick to the RAM tier you found in Step 1.
| Model | Approx. size | Best for |
|---|---|---|
| Gemma 4 E2B | ~3 GB | Best all-round default on an 8 GB phone; fast, multimodal, light on battery |
| Phi-4 Mini (3.8B) | ~2.7 GB | The strongest reasoning in the sub-4B class |
| Qwen3 1.7B | ~1.1 GB | Best multilingual option; fits 6 GB phones |
| SmolLM2 1.7B | ~1.1 GB | Fastest tokens/sec; older or constrained devices |
| Gemma 3 1B | ~720 MB | The lightweight pick for 4 GB phones |
When in doubt on a modern 8 GB phone, download Gemma 4 E2B and move on.
Tap the downloaded model to load it into memory and start a conversation. The first response takes a few seconds while the model warms up; subsequent ones are quicker. Now do the satisfying part: turn on airplane mode and ask it something else. It still answers. That is your proof that nothing is going to a server, because there is no longer a server to reach.
Keep it running well Heat, not RAM, is the real ceiling on sustained use. Long generations are one of the heaviest loads you can put on a phone and will trigger thermal throttling that slows or halts inference. Keep sessions short and bursty, take the case off during heavy use, and expect noticeable battery drain. On Samsung One UI, OnePlus, and Xiaomi, whitelist the app in battery optimization (Samsung calls it "Sleeping apps") so the system does not kill it mid-answer. Do not run two inference apps at once.
iOS is nearly as easy as Android, and Apple Silicon is efficient enough that token speeds are good. One clarification first, because it confuses newcomers: Apple Intelligence, the AI built into iOS, is a separate, managed system. You cannot swap its model or run it as a general chatbot. The apps below are different: they let you download and run whatever open model you like, fully under your control.
| App | Price | Why choose it |
|---|---|---|
| PocketPal AI | Free | The same excellent beginner app as on Android. Start here. |
| Private LLM | ~$10 once | Best paid option: Siri and Shortcuts integration, 140+ models, runs on iPhone, iPad, and Mac. |
| LLM Farm | Free | The most configurable; good if you like tuning inference settings. |
| Off Grid | Free | Simple, open-source, no-internet-after-download chat. |
The standout trick belongs to Private LLM: because it plugs into Shortcuts, you can wire your on-device model to a voice command and get something the built-in assistant cannot give you, a private answer generated with no connection at all.
Grab one model over Wi-Fi. The same GGUF Q4_K_M advice applies. Sensible iPhone picks:
| Model | Fits | Notes |
|---|---|---|
| Phi-4 Mini (3.8B) | 8 GB iPhone | Great default on a 15 Pro / 16 |
| Llama 3.2 3B (~2.2 GB) | 8 GB iPhone | The most-tested 3B workhorse |
| Qwen3 4B (~2.5 GB) | 8 GB iPhone | Strong multilingual performance |
| Gemma 3 1B / Llama 3.2 1B | 6 GB iPhone 15 or older | Lightweight and fast |
Select the model to load it, send a prompt, then flip on airplane mode and prompt again to confirm it is running entirely on-device. If an app crashes as the model loads, that is almost always a sign the model is too big for your RAM tier; drop down a size and try again.
iPhone reality check The same heat and battery caveats apply as on Android, and the numbers are concrete: roughly 200 inferences per full charge is typical on a recent iPhone. This is a burst tool for private, focused tasks, not an always-on companion. Keep iOS updated, since newer versions handle memory pressure more gracefully.
A Windows PC is where local models stop feeling like a novelty and start feeling like a tool. With a mid-range graphics card you can run 8B to 14B models that are meaningfully more capable than anything on a phone, and you get something phones cannot easily offer: a local API server you can point your own apps, scripts, and coding assistants at, keeping the whole pipeline private.
On a PC the memory that matters is your GPU's VRAM. The rule is blunt: pick a model that fits inside it. You can run models on the CPU alone, but it is dramatically slower, and there is a nasty performance cliff the moment a model spills out of VRAM into ordinary system RAM.
That cliff deserves its own picture, because it is the difference between an assistant that feels instant and one that feels broken.
Two tools cover almost everyone, and they split cleanly by temperament.
Download the installer from lmstudio.ai and run it; it installs like any normal Windows app and auto-detects your GPU. Open the discover tab, download a model that fits your VRAM (for an 8 GB card, start with Llama 3.1 8B or Qwen3 8B at Q4_K_M), switch to the chat tab, pick your model, and type. To expose the model to your own tools, open the Local Server tab and click Start; it serves an OpenAI-compatible API at http://localhost:1234.
Download from ollama.com, run the installer, then open PowerShell and pull-and-run a model in a single command:
ollama run llama3.1
The first run downloads the weights; after that it launches instantly. Swap llama3.1 for qwen3, phi4, or gemma3. Chat right there in the terminal, and type /bye to exit. Ollama quietly runs an API server at http://localhost:11434 the whole time, so scripts and apps can call your local model with the same interface they would use for a cloud one.
| Model | VRAM | Why |
|---|---|---|
| Llama 3.1 8B | 8 GB | The reliable all-rounder |
| Qwen3 8B / 14B | 8 / 12 GB | Best multilingual and coding; 14B if you have the headroom |
| Phi-4 | 8 GB | Strong reasoning for its size |
| Mistral 7B | 8 GB | Lightweight and fast |
Start small, confirm the speed is comfortable, then step up a size only if your VRAM has room. Disconnect from the internet and prompt once more to confirm the whole thing runs offline.
Windows tips The local API server is the real unlock: point a coding assistant, a note app, or your own script at localhost and everything stays on your machine. Update your GPU drivers for the best performance, and stick with Q4_K_M unless you have VRAM to spare, since higher-precision quants (Q6, Q8) improve quality but eat memory fast and can push you off the cliff above.
You will keep seeing cryptic strings like GGUF and Q4_K_M. They are simpler than they look. A model's "weights" are just a huge pile of numbers. Quantization shrinks those numbers from high precision to lower precision so the model takes less memory, in exchange for a small, usually unnoticeable, drop in quality. Q4_K_M means a particular, well-tuned 4-bit scheme, and it is the pragmatic default almost everyone should use. The format is just the file container the weights live in.
It is worth being honest and specific about the capability line, because the gap between what a small local model does well and what it does badly is wide and sharp. Knowing which side of it you are on is the whole art of using one happily.
The failure mode to internalize is that these models fail confidently. They will not hedge or admit uncertainty; they will hand you a fluent, well-structured, completely wrong answer with the same tone they use for a right one. This is not a bug you can prompt your way out of, it is a property of small, quantized models. So use them for the tasks in the left column, where fluency is the point, and reach for a cloud model or a primary source whenever correctness actually matters.
| Platform | Install | Download | Then |
|---|---|---|---|
| Android | PocketPal AI | Gemma 4 E2B (8 GB phone) | Load, chat, verify in airplane mode |
| iPhone | PocketPal AI, or Private LLM for Siri | Phi-4 Mini or Llama 3.2 3B | Load, chat, verify in airplane mode |
| Windows | LM Studio (easy) or Ollama (developer) | Llama 3.1 8B or Qwen3 | Chat, or expose the local API on localhost |
You now have a private AI that lives entirely on your own hardware, answers without a connection, and never sends a word of your conversation to anyone. Start small, respect the one number that matters, RAM on a phone and VRAM on a PC, and let the model do the everyday drafting and thinking work it is genuinely good at. The novelty of watching a coherent answer appear in airplane mode wears off surprisingly slowly.
A note on sources: the Android guidance draws on a 2026 field review of on-device Android apps and models; the iPhone and Windows figures come from current hands-on guides and hardware write-ups from DEV Community, PromptQuorum, Private LLM, and SitePoint. Token-per-second and RAM figures are indicative and shift with thermal state, model, and quantization, so treat them as ranges, not promises.