Local LLMs vs cloud LLMs

The short answer

A local LLM is worth running when a downloaded model meets your real quality and latency target and you need offline operation, tighter data control, or enough steady volume to justify owning the hardware and operations. A cloud LLM is usually easier when you need managed scale, access to a specific hosted model, or do not want to operate inference infrastructure.

Hybrid can be sensible, but only if sensitive requests fail closed. A rule that silently falls back from a local model to a cloud API is not a privacy control.

What you'll learn

This guide gives you a repeatable way to compare local and cloud LLMs across privacy, cost, hardware, speed, context, quality, offline use, maintenance, coding, personal use, and business use. It also shows how to verify that an Ollama request is using a downloaded local model rather than assuming every localhost request stays local.

Who this is for

• Developers deciding whether local inference is practical on their current hardware
• Teams handling confidential code or documents that need an explicit data-flow review
• Builders comparing recurring cloud usage with the full cost of self-hosting
• Anyone who wants evidence from their own workload instead of a universal winner

Local and cloud do not mean one product

A local LLM is a downloaded model whose inference runs on hardware you control. A cloud LLM runs on a provider's infrastructure and receives the request over a network. The same application can use both.

Ollama makes the distinction slightly less obvious because it supports downloaded local models and cloud-hosted models. Calling http://localhost:11434 alone does not prove inference stayed on the machine. For a strict local-only deployment, select a downloaded model and disable Ollama Cloud features.

Cloud providers can offer enterprise controls, regional processing choices, and different retention terms. Review the exact provider, account tier, endpoint, and contract; do not reduce the decision to "cloud is public" or "local is automatically compliant."

Decision table

For coding, test the languages and repository sizes you actually use. For personal work, convenience and offline access may decide it. For a business, add access control, retention, licensing, auditability, uptime, and incident response to the comparison.

Test Ollama with a confirmed local model

1. Install from the current official source

Use the native macOS or Windows download from ollama.com/download. On Linux, the official command is:

curl -fsSL https://ollama.com/install.sh | sh

Choose an exact model tag from the Ollama library, check its artifact size and license, then pull it. This example follows the current quickstart; it is an example, not a hardware promise:

ollama pull gemma3
ollama run gemma3 "Summarize why stream backpressure matters."

2. Disable cloud features when local-only is a requirement

Set OLLAMA_NO_CLOUD=1 for the Ollama server process, restart Ollama, and confirm its logs report that cloud is disabled. The equivalent persistent setting is "disable_ollama_cloud": true in ~/.ollama/server.json.

This removes Ollama cloud models and web search from that server. It does not audit the rest of your application: tools, logging, analytics, backups, and other dependencies can still create network or retention risks.

3. Call the local OpenAI-compatible endpoint

Ollama's OpenAI-compatible API is available at /v1/. The OpenAI client requires a key value syntactically, but the local Ollama API does not authenticate it:

import OpenAI from "openai";

const localClient = new OpenAI({
  baseURL: "http://localhost:11434/v1/",
  apiKey: "ollama",
});

const response = await localClient.chat.completions.create({
  model: "gemma3",
  messages: [{
    role: "user",
    content: "Explain stream backpressure with one Node.js example.",
  }],
});

console.log(response.choices[0].message.content);

The local API requires no authentication on localhost. Keep it bound to loopback unless remote access is genuinely required. If you expose OLLAMA_HOST=0.0.0.0:11434, protect it with network controls plus an authenticated TLS proxy; a bare internet-facing endpoint is unsafe.

Benchmark the workload, not the marketing label

Create a small evaluation set from real inputs. Use the same prompt, context, temperature, output limit, and scoring rubric for every candidate. Run enough repetitions to separate a warm result from a pattern, and record:

• task pass rate or reviewer score
• time to first useful output and total latency
• cold-start and warm-request behavior
• generated throughput and concurrent-request behavior
• peak RAM/VRAM, CPU/GPU split, and failure rate
• input/output tokens and current cloud price for the same calls

Ollama's native API response includes eval_count and eval_duration in nanoseconds. Generated tokens per second can be calculated from eval_count / (eval_duration / 1e9); use the returned values instead of publishing a guessed speed.

curl http://localhost:11434/api/generate -d '{
  "model": "gemma3",
  "prompt": "Classify this support request using the supplied rubric.",
  "stream": false
}'

Run ollama ps while testing. Its PROCESSOR column shows whether the loaded model is on CPU, GPU, or split across both. If performance misses the target, test a smaller artifact, a different quantization, or a shorter context before buying hardware.

Hardware and context: why fixed tables mislead

Parameter count is not enough to predict whether a model will fit or run well. The exact artifact and quantization determine weight size; context length adds memory use; parallel requests multiply context allocation; and partial CPU offload changes latency. Driver, accelerator, and runtime support matter too.

Ollama's current context defaults vary by available VRAM. Set a deliberate value only after measuring the task. For example:

OLLAMA_CONTEXT_LENGTH=8192 ollama serve

That value is an example, not a recommendation for every model. A longer context may be necessary for code or retrieval, but it costs memory. Inspect the allocated context and processor split with ollama ps.

Compare total cost

For cloud, use the provider's current official input, cached-input, output, tool, storage, and batch rates where applicable. Multiply them by measured tokens and request volume; do not copy an old price from a comparison post.

For local, include:

• hardware purchase or rental amortization
• electricity and storage
• engineering time for setup, upgrades, security, and observability
• spare capacity, queues, and failure recovery
• the cost of a cloud fallback if one exists

A local request may have no per-token provider bill and still be the more expensive system. A cloud bill may be small at prototype volume and become material later. Calculate both at today's volume and at a realistic growth case.

Privacy and confidential code

A strong local-only path has several controls working together:

1. The application uses a downloaded, reviewed model ID from an allowlist.
2. Ollama Cloud is disabled on the inference server.
3. The endpoint stays on loopback or a protected private network.
4. Logs, traces, crash reports, tools, and backups follow the same data policy.
5. The model license and deployment controls match the intended business use.
6. Sensitive requests fail closed instead of falling back to cloud.

Ollama states that it does not see prompts or data for local runs; cloud-hosted models process prompts and responses remotely. That product statement is one input to a privacy review, not a replacement for your own architecture, legal, and compliance checks.

Choose LOCAL when...

• a tested downloaded model meets the quality and latency requirement
• offline operation or control of the inference boundary is necessary
• expected steady volume makes the measured total cost favorable
• your team can own capacity, security, monitoring, and upgrades
• confidential inputs are blocked from all cloud fallbacks and external tools

Choose CLOUD when...

• a particular hosted model performs materially better on the evaluation set
• traffic is uncertain or bursty and managed scaling reduces operational risk
• your current hardware misses the latency, context, or concurrency target
• the provider's reviewed privacy and retention terms fit the workload
• shipping and maintaining model-serving infrastructure is not a good use of the team

Choose HYBRID when...

• separate task classes have measured reasons to use different backends
• routing is explicit, observable, and covered by tests
• sensitive routes use a confirmed local-model allowlist and fail closed
• cloud fallback is limited to data classes approved for that provider
• both paths share an evaluation contract, not merely an SDK interface

Do not route only from a label such as "sensitive-data" and assume a localhost client is proof. Validate the selected model and deployment controls before the request is sent.

Common mistakes

Declaring a winner from one easy prompt — extraction, coding, long-context review, and tool use stress different capabilities. Use representative inputs and a scoring rubric.

Publishing universal hardware or speed numbers — artifact, quantization, context, concurrency, offload, and software versions change the result. Report the tested configuration if you publish a benchmark.

Treating local as free — provider usage fees may disappear, but hardware and operations do not.

Treating local as automatically private — a cloud model can be reached through Ollama's local API, and the surrounding application can still send data elsewhere.

Ignoring production behavior — load-test queueing, parallel requests, memory pressure, overload responses, timeouts, authentication, monitoring, and recovery before promising capacity.

Troubleshooting

Responses are slower than expected: run ollama ps and check the processor split, context length, loaded artifact, and concurrent load. Compare cold and warm requests before changing hardware.

The process runs out of memory: reduce the selected model or context, lower concurrency, and retest. Use OLLAMA_CONTEXT_LENGTH for the server default or the API's num_ctx option for a request; OLLAMA_NUM_CTX is not the server setting.

Remote calls fail: Ollama binds to 127.0.0.1:11434 by default. Exposing 0.0.0.0:11434 is an intentional security change, not a troubleshooting shortcut.

A local-only test reaches the internet: confirm the selected model is downloaded, cloud features are disabled, the process was restarted, and the application has no tools or fallback client enabled. Then inspect logs and network egress.

Checklist

• [ ] Representative prompt set and pass rubric defined
• [ ] Exact local model tag, artifact, quantization, context, and license recorded
• [ ] Same inputs and generation settings tested on local and cloud candidates
• [ ] Cold/warm latency, throughput, concurrency, memory, and failures measured
• [ ] Cloud cost calculated from current official pricing and measured tokens
• [ ] Local cost includes hardware, power, storage, operations, and fallback
• [ ] Data flow, logs, retention, tools, access control, and model license reviewed
• [ ] Local-only server has cloud disabled and sensitive paths fail closed
• [ ] Production path has load, overload, timeout, monitoring, and recovery tests

Practice task

Choose ten real examples from one feature. Run them through one downloaded local model and one approved cloud model using the same settings. Score correctness before looking at latency. Then record cold and warm latency, Ollama's returned usage metrics, peak memory, cloud token usage, and every network destination. Write the decision from that evidence, including the conditions that would make you revisit it.

FAQ

How good are local LLMs?
Good enough is task-specific. A local model can be the right classifier and the wrong coding assistant, or the reverse. The honest answer comes from the exact model, artifact, context, hardware, and evaluation set.

Is it worth running a local LLM?
Yes when it passes your workload and offline, privacy-boundary, or steady-volume benefits outweigh ownership cost. No when the tested output misses the bar or serving it costs more time and money than the approved cloud option.

Can Ollama serve a production app?
It can process concurrent requests and queue work, but suitability depends on your load and reliability target. Test capacity, memory, overload behavior, access control, observability, upgrades, and recovery; choose a different serving stack if those results do not meet the requirement.

Is localhost proof that data stayed local?
No. Ollama cloud models can be called through the local API. Use a downloaded model, disable cloud features for a strict local-only server, and audit the rest of the application's network paths.

Official references checked

• Ollama quickstart
• OpenAI compatibility
• Context length
• Ollama FAQ and local-only setting
• Ollama API usage metrics

Takeaways

Local is not automatically private, cheap, fast, or lower quality. Cloud is not automatically expensive, unsafe, or better. Each label hides the model, configuration, workload, data controls, and operating model that determine the result.

If you remember one thing: test the same real tasks, calculate the full cost, and verify the data path. Choose local, cloud, or hybrid from those measurements — not from a benchmark copied from someone else's machine.