What you'll learn

By the end of this you'll have a clear picture of when running an LLM locally actually makes sense versus when you're just adding infrastructure complexity without a real benefit. You'll know the hardware requirements for different local model sizes, how to get started with Ollama in under 10 minutes, and the specific use cases where local genuinely wins.

The honest version first: local LLMs are exciting and the tooling has gotten genuinely good. But most of the "why you should run local" content glosses over the hardware requirements and the quality gap between local models and frontier cloud models. This post doesn't.

Who this is for

  • Developers curious about local LLMs who want an honest picture of what it takes
  • Engineers working on apps that process sensitive data and need to understand the privacy trade-offs
  • Anyone hitting significant cloud API costs who wants to know if local is a realistic alternative

You can skip this if you've already run Ollama with a 70B model on proper hardware and have a clear opinion. Jump to the Patterns section if you just want the decision framework.

What are local LLMs and cloud LLMs?

A local LLM is a model you run on your own machine (or your own server). The model weights live on your hardware, inference runs on your CPU or GPU, and no data leaves your network. Popular tools: Ollama (the simplest path), LM Studio (GUI-focused), llama.cpp (raw inference), and vLLM (for production self-hosting).

A cloud LLM is a model hosted by a provider — OpenAI, Anthropic, Google, Mistral, Groq — that you call via API. The inference happens on their infrastructure, your data travels to their servers, and you pay per token (or per API call).

Plain English: local means the model runs on your computer. Cloud means you send a request to someone else's computer and get an answer back.

Simple idea: local gives you privacy and eliminates per-request costs. Cloud gives you much better models with zero infrastructure maintenance. The question is which trade-off matters more for your situation.

Prerequisites

  • Basic familiarity with running terminal commands — Ollama is a CLI tool
  • An understanding of what LLMs are used for in apps — if you're new, read the LLM APIs comparison first
  • Honest assessment of your hardware: Apple Silicon Mac, discrete GPU, or CPU-only matters a lot here

Setup from zero

Step 1 — Install Ollama and pull a model

On macOS or Linux:

curl -fsSL https://ollama.ai/install.sh | sh

On Windows, download the installer from ollama.ai. Then pull a model:

# Small but capable — good for testing on CPU or 8GB RAM
ollama pull llama3.2:3b

# Mid-range — needs at least 8GB VRAM or 16GB unified memory
ollama pull llama3.1:8b

# Serious — needs 24GB+ VRAM or Apple Silicon Mac with 32GB+ RAM
ollama pull llama3.1:70b

The model size you can run usably depends almost entirely on your hardware. The 3B model runs on anything. The 70B model needs real hardware to be usable at a reasonable speed.

Step 2 — Make a test call

Ollama exposes an OpenAI-compatible API on port 11434:

import OpenAI from "openai";

const localClient = new OpenAI({
  baseURL: "http://localhost:11434/v1",
  apiKey: "ollama", // required by the SDK but not checked by Ollama
});

const response = await localClient.chat.completions.create({
  model: "llama3.2:3b",
  messages: [{ role: "user", content: "What is 47 times 83?" }],
});

console.log(response.choices[0].message.content);

And the actual math answer: 3901. Worth checking — smaller models make arithmetic errors more often than you'd expect.

Step 3 — Benchmark against the cloud on your actual task

Don't form opinions on toy prompts. Run the actual task your app would do — summarization, code generation, extraction, classification — on both a local model (sized for your hardware) and a cloud model (GPT-4o or Claude Sonnet). Note both the quality and the latency. The comparison on real tasks tells you everything the synthetic benchmarks don't.

async function benchmarkBoth(prompt: string) {
  const start = Date.now();

  const localRes = await localClient.chat.completions.create({
    model: "llama3.1:8b",
    messages: [{ role: "user", content: prompt }],
  });

  const localMs = Date.now() - start;

  const cloudStart = Date.now();
  const cloudRes = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: prompt }],
  });
  const cloudMs = Date.now() - cloudStart;

  return {
    local: { text: localRes.choices[0].message.content, ms: localMs },
    cloud: { text: cloudRes.choices[0].message.content, ms: cloudMs },
  };
}

The mental model

The key variable isn't "local vs cloud" — it's hardware. Local LLMs on good hardware can be excellent. Local LLMs on inadequate hardware are frustrating.

Here's the honest breakdown by hardware tier:

CPU only or integrated graphics (most laptops): You can run 3B–7B models, slowly. They're fine for experimentation. You wouldn't build a production app on them. Latency is high enough that interactive use is unpleasant for anything longer than a short response.

Apple Silicon Mac with 32GB+ unified memory (M2/M3/M4 Pro or Max): This is the local LLM sweet spot in 2026. The unified memory architecture means the GPU and CPU share the same pool — a 32GB M3 Pro runs a 13B model comfortably, a 64GB M4 Max can run a 34B model well. Speed is genuinely usable. This is where local becomes a real option.

Dedicated GPU with 24GB+ VRAM (RTX 4090, A100, etc.): The fastest local inference. 70B models run at reasonable speeds. This is the hardware for serious local production workloads.

And the cloud fits the other end: no hardware required, best models always available, latency depends on network and provider load, you pay for what you use.

Little tip: if you're on an Apple Silicon Mac with 32GB RAM, local LLMs are genuinely worth experimenting with seriously. If you're on a standard laptop with 16GB RAM and integrated graphics, the cloud is almost certainly the right call for anything beyond hobby projects.

Key terms

Model size (parameters) — how many weights the model has. 3B, 7B, 13B, 34B, 70B are common sizes. Larger = more capable and more RAM required. Rule of thumb: you need roughly 2x the model size in GB of RAM (in 4-bit quantization). A 7B model needs about 4GB VRAM.

Quantization — reducing the precision of model weights to fit in less memory. Q4 (4-bit) is the standard for local use — noticeably smaller than full precision with modest quality loss. Models you pull with Ollama are typically quantized.

VRAM — GPU memory. This is the critical constraint for GPU-accelerated inference. If the model doesn't fit in VRAM, inference falls back to RAM (much slower) or CPU (even slower).

Unified memory — Apple Silicon's approach where CPU and GPU share the same memory pool. This is why M-series Macs punch above their weight for local LLM inference.

Tokens per second (tok/s) — inference speed. For interactive chat, 20+ tok/s is comfortable. Under 10 tok/s starts to feel slow. On a CPU-only machine with a 7B model, you might see 5–10 tok/s.

Ollama — the simplest tool for running local models. Download it, pull a model, run it. Exposes an OpenAI-compatible API so you can use the same SDK client code. Also has a CLI for direct chat.

vLLM — production-grade serving for local models. Handles batching, parallel requests, and quantization configurations. More setup than Ollama but necessary for serious production self-hosting.

Step-by-step

Choosing the right local model for your hardware

Use this as a rough guide (4-bit quantization, Ollama):

| Hardware | Max usable model |
|----------|-----------------|
| 8GB RAM (CPU) | 3B (slow) |
| 16GB RAM (CPU) | 7B (slow) |
| M2 Pro 16GB | 7B–13B usable |
| M3 Pro 36GB | 13B–34B usable |
| M4 Max 64GB | 34B–70B usable |
| RTX 4090 (24GB VRAM) | 34B–70B fast |

"Usable" means response times under 5 seconds for short prompts. It doesn't mean performance-parity with frontier cloud models.

Testing data stays local

The clearest local win: run a sensitive document through a local model and verify no network traffic leaves your machine. Use a network monitor (Little Snitch, Charles, or Wireshark) and confirm Ollama's traffic stays on 127.0.0.1. For apps processing medical records, legal documents, or private source code, local is the compliance-correct path, not just the privacy-preferring one.

// All processing stays on localhost — no network egress
const analysis = await localClient.chat.completions.create({
  model: "llama3.1:8b",
  messages: [{
    role: "user",
    content: `Summarize the key risks in this contract section:

${contractText}`,
  }],
});

Switching between local and cloud based on task type

A practical pattern for apps that need both:

type TaskType = "quick-classification" | "complex-reasoning" | "sensitive-data";

function getClient(task: TaskType): OpenAI {
  if (task === "sensitive-data") {
    return localClient; // stays on machine
  }
  if (task === "quick-classification") {
    return localClient; // fast enough locally, saves cost
  }
  return openaiClient; // complex reasoning — use cloud
}

Little tip: for classification tasks — sentiment, category, intent — smaller local models are often good enough and running them locally means no per-token cost for high-volume routing. Save the cloud API budget for the tasks that actually need frontier model quality.

Patterns / when to use

Use local LLMs when:
- Data privacy requirements prevent sending inputs to third-party APIs (medical, legal, HR)
- You're running high-volume, simple tasks where per-token cloud costs become significant
- You have 32GB+ unified memory or 24GB+ VRAM and want zero infrastructure overhead for development
- You're experimenting and want to run inference without usage costs

Use cloud LLMs when:
- You need frontier model quality — complex reasoning, nuanced writing, hard code generation
- Your hardware is inadequate for the model size you'd need for acceptable quality
- You want managed reliability, uptime, and zero operational overhead
- Your use case doesn't involve genuinely sensitive data

Use both:
The routing pattern above is genuinely useful — sensitive or high-volume-simple tasks go local, quality-critical or complex tasks go cloud. It's more infrastructure than a single-API setup, but the cost savings on volume tasks and the compliance coverage on sensitive tasks can both justify the overhead.

Common mistakes

Evaluating local models on simple prompts — "write a poem about autumn" looks fine from a 7B model. "Refactor this 200-line TypeScript file following these specific constraints" reveals the quality gap. Test on your hardest representative task, not the easy ones.

Ignoring token generation speed — if your app is interactive, 5 tok/s is painful for users, even if the final answer is accurate. Measure tok/s for your hardware and model combination before committing to local for user-facing features.

Assuming open-source = production-ready — Ollama is excellent for development and experimentation. For production, you need proper serving infrastructure, health checks, request queuing, and fallback handling. vLLM and similar tools exist for exactly this, but they require real operational investment.

Not quantifying the cost difference — "cloud costs money" is true but vague. Actually calculate your token volume and projected costs. For many apps, the cloud bill is surprisingly reasonable, and local infrastructure (hardware, electricity, maintenance, engineering time) has real costs too.

Troubleshooting

Ollama is running but responses are very slow — check whether the model is fitting in VRAM or falling back to RAM/CPU. Run ollama ps to see the loaded model and where it's running. If it says "CPU," the model doesn't fit in your GPU memory — try a smaller model or a more aggressive quantization.

Model giving wrong answers on factual questions — smaller models have weaker factual recall. For tasks requiring accurate factual answers, local small models are not reliable. This is an inherent capability limitation, not a configuration problem. Use the cloud or accept lower accuracy.

Out of memory errors when running Ollama — reduce the context window. By default Ollama uses a large context window that consumes significant memory even for small models. Set OLLAMA_NUM_CTX=2048 if you don't need long context. It reduces memory use substantially.

API calls to Ollama failing from a remote machine — Ollama binds to localhost by default. Set OLLAMA_HOST=0.0.0.0 to expose it on your network, but only do this on trusted networks. There's no authentication built into Ollama's API by default.

Checklist

  • [ ] Hardware tier identified and appropriate model size selected
  • [ ] Ollama installed and a test model pulled
  • [ ] Test call to Ollama successful via SDK
  • [ ] Latency measured for your specific task type on your specific hardware
  • [ ] Quality comparison run on your hardest representative task (local vs cloud)
  • [ ] Token generation speed measured and verified as acceptable for UX
  • [ ] Privacy requirements reviewed — is local actually required or just preferred?
  • [ ] Cost calculation done for cloud alternative at projected volume

Practice task

Pick one task from your current project — something specific, not a toy example. Run it locally with Ollama on the largest model your hardware can run at reasonable speed. Run the same task with GPT-4o or Claude Sonnet. Compare three things: output quality, latency, and the dollar cost of 1000 cloud calls at that task's typical token count. The math on cost will surprise you in one direction or the other, and the quality gap will either validate the local approach or clarify that the cloud is the right call. Do the actual numbers.

FAQ

Can local LLMs match cloud quality in 2026?
At the frontier, no — the largest frontier cloud models (GPT-4 class, Claude Opus class) are significantly more capable than what most people can run locally. Capable local models like Llama 3.1 70B on good hardware are genuinely impressive, but the gap to frontier cloud is real on hard tasks.

What's the cheapest hardware to run a useful local model?
An Apple Silicon Mac with 16GB unified memory (M3 or M4 base model) runs 7B–8B models at usable speed. Not impressive by frontier standards, but useful for classification, summarization, and light generation tasks. Cost: whatever the Mac costs — no GPU upgrade needed.

Does Ollama work for production apps?
For low-traffic internal tools, yes. For production apps with meaningful request volume, use vLLM or a similar production-grade serving layer. Ollama is optimized for development experience, not production throughput.

Is my data truly private with local LLMs?
If you're using Ollama on localhost with no network exposure, yes — inference runs entirely on your hardware. Verify this with a network monitor on sensitive data before relying on it for compliance purposes.

What to learn next

  • OpenAI vs Anthropic APIs — if local isn't the right fit, the cloud comparison matters
  • RAG and retrieval-augmented generation — how to give local models access to your data without relying on large context windows
  • vLLM setup for production — if local at scale is the goal, the production serving tools
  • [OpenAI vs Anthropic APIs](/ai/comparisons/openai-vs-anthropic-apis)
  • [Best LLM APIs for apps in 2026](/ai/lists/best-llm-apis-for-apps)
  • [Best free AI tools for builders](/ai/lists/best-free-ai-tools-for-builders)
  • [Prompt engineering for code](/ai/tutorials/prompt-engineering-for-code)

Takeaways

Local LLMs are genuinely good in 2026 on the right hardware. The tooling (Ollama especially) has made setup trivially easy. The quality on capable hardware with larger models is impressive. But the quality gap to frontier cloud models is real, the hardware requirements for truly capable local models are steep, and the cloud is still the right default for most production apps.

Local wins when data privacy requirements are strict, you have the hardware for it, or you're running high-volume simple tasks where per-token costs add up. Cloud wins on quality, zero-hardware-overhead convenience, and complex reasoning tasks. Many production apps end up using both.

If you remember only one thing: test with your actual task on your actual hardware before committing to local. The abstract arguments for local are convincing; the concrete benchmark on your specific use case tells you whether they apply to you.