Open-Weight LLMs on Windows: When a Quantized Local Model Is a Credible Production Choice

One of the production systems I have built never talks to the internet. It is a retrieval-augmented assistant for an organisation whose documents are not allowed to leave the building, so every embedding and every answer comes from an open-weight model running on the same machine as the user. When people ask whether that is a toy, my answer is no — but only because of a decision most tutorials treat as a footnote: how the model is quantized, and whether it actually fits in memory with the context the job needs.

The argument of this article is simple. For workloads bound by privacy, data residency, offline operation or predictable cost, a well-quantized open-weight model on ordinary Windows hardware is now a credible production option. Not for everything, and not because local is cheaper — it usually is not. It is credible because three things have matured at once: quantization formats that lose almost nothing at 4–5 bits, runtimes that install with one command and expose an OpenAI-compatible API, and open-weight models released under permissive licences, sometimes already quantized by the vendor. The corollary is the part people get wrong: success depends less on which model you pick than on the quantization and memory budget you fit it into.

Here is the route. I start with why local inference matters now and what the honest economics look like. I then lay out the evidence on quantization — size, quality and speed — including a result that overturns a popular rule of thumb. I offer a decision framework, then go deep on four dimensions: how quantization actually works, the memory arithmetic, hardware tiers on Windows, and runtimes and serving. I close with cases, counterpoints, implications by role, a 90-day plan and the limits of what this evidence can tell you.

Context: why this matters now

There are four legitimate reasons to run a model locally, and only one of them is cost.

Residency and privacy. In regulated settings — health, legal, finance, government — sending document text to a third-party API is often a compliance question before it is an engineering one. A local model removes the question: no egress, no processor agreement, no cross-border transfer. Offline and edge operation. Field teams, ships, clinics with poor connectivity and air-gapped networks cannot depend on an API at all. Latency and control. A local model has no rate limits, no surprise deprecations and no queue you do not own. Cost at very high, steady volume — but this is the weakest of the four, and I want to be precise about why.

The supply side has shifted decisively. Vendors now ship models designed to be quantized. Google released quantization-aware-trained (QAT) versions of Gemma 3 in April 2025, cutting the weights of the 27B model from 54 GB to 14.1 GB so that it runs on a single 24 GB RTX 3090 (Google). OpenAI’s gpt-oss models were post-trained with MXFP4 quantization of their mixture-of-experts weights; the 20B model (21B parameters, 3.6B active) is stated to run within 16 GB of memory, under Apache-2.0 (model card). The 2026 open-weight field — Qwen, Gemma 4 with official QAT q4_0 GGUFs, Mistral Small 4, DeepSeek, gpt-oss — is large and mostly permissively licensed.

Exhibit 1

Gemma 3 27BBF16: 54 GB14.1 GB

Gemma 3 12BBF16: 24 GB6.6 GB

Gemma 3 4BBF16: 8 GB2.6 GB

Gemma 3 1BBF16: 2 GB0.5 GB

Weights only, as Google notes: the KV cache for your context length comes on top. Google names the RTX 3090 (24 GB) for 27B and an RTX 4060 Laptop GPU (8 GB) for 12B.

Source: Google Developers Blog, 18 Apr 2025

View data table

Model	BF16 (GB)	int4 QAT (GB)
Gemma 3 27B	54	14.1
Gemma 3 12B	24	6.6
Gemma 3 4B	8	2.6
Gemma 3 1B	2	0.5

Now the economics, honestly. Small hosted models have become extraordinarily cheap. At October 2026 list prices, Claude Haiku 5.5 costs $0.10 per million input tokens and $0.50 per million output tokens (Anthropic); Gemini 3.5 Flash-Lite is $0.30 and $2.50 (Google). Take a typical retrieval-augmented query — about 3,000 tokens of prompt and retrieved context, 300 tokens of answer — and a 16 GB RTX 5060 Ti, which launched at $429 (PCGamesN). Exhibit 2 shows how many queries it takes before the card has paid for itself in avoided API spend.

Exhibit 2

vs Opus 5.5~24K

vs Sonnet 5.5~48K

vs Gemini 3.5 Flash-Lite~260K

vs Haiku 5.5~953K

Author’s estimate. Ignores electricity, the rest of the PC, engineering time and — most importantly — the quality gap: a local 8B model is not equivalent to Opus. The point is the order of magnitude, not the exact figure.

Source: Author’s estimate from list prices (Anthropic, Google; Oct 2026) and RTX 5060 Ti 16 GB launch price (PCGamesN)

View data table

API (input / output per 1M tokens)	Cost per query	Queries to $429
Claude Opus 5.5 ($4 / $20)	$0.0180	~23,800
Claude Sonnet 5.5 ($2 / $10)	$0.0090	~47,700
Gemini 3.5 Flash-Lite ($0.30 / $2.50)	$0.00165	~260,000
Claude Haiku 5.5 ($0.10 / $0.50)	$0.00045	~953,000

Read Exhibit 2 carefully, because it cuts both ways. Against frontier-priced models, a modest GPU pays back within tens of thousands of queries — a few months for a busy internal tool. Against the cheapest hosted tier, it takes close to a million. So the cost case for local is narrow: high, steady volume on tasks where a small model is good enough. The stronger case is almost always the first three reasons — residency, offline, control — where the hosted option is not on the table at all. If you are choosing local to save money on a low-volume workload with no data constraints, you are probably making a mistake.

The evidence: what quantization costs and what it buys

Quantization stores a model’s weights in fewer bits than the 16 used at full precision. The format you will meet on Windows is GGUF, the file format of llama.cpp and of the tools built on that ecosystem. Three questions decide whether a quantized model is fit for production: how much smaller it gets, how much quality it loses, and how much faster it runs.

Size: a 4-bit quant is under a third of full precision

llama.cpp’s quantize README publishes the effective bits per weight (bpw) of each type, measured on Llama-3.1-8B (README). Effective bpw is always a little above the nominal figure because each block of weights also stores scales.

Exhibit 3

F16 (full precision)16.0

Q8_08.50

Q6_K6.56

Q5_K_M5.70

Q4_K_Mthe usual starting point4.89

IQ4_XS4.46

Q3_K_M4.00

Q2_K3.16

IQ2_XS2.59

Source: llama.cpp, tools/quantize/README.md

View data table

Type	Bits per weight	Size (GiB)
F16	16.00	14.96
Q8_0	8.50	7.95
Q6_K	6.56	6.14
Q5_K_M	5.70	5.33
Q4_K_M	4.89	4.58
IQ4_XS	4.46	4.17
Q3_K_M	4.00	3.74
Q2_K	3.16	2.95
IQ2_XS	2.59	2.42

Quality: almost free to 5 bits, a small tax at 4, a cliff below 3

The standard quality measure is the increase in perplexity — roughly, how surprised the model is by held-out text — relative to full precision. llama.cpp’s quantize.cpp records these deltas for Llama-3-8B (source). The curve is the whole story.

Exhibit 4

Q8_0+0.003

Q6_K+0.02

Q5_K_M+0.06

Q4_K_M+0.18

Q4_0+0.47

Q3_K_M+0.66

Q3_K_S+1.63

Q2_K+3.52

Source: llama.cpp, tools/quantize/quantize.cpp

View data table

Type	+ perplexity
Q8_0	+0.0026
Q6_K	+0.0217
Q5_K_M	+0.0569
Q5_K_S	+0.1049
Q4_K_M	+0.1754
Q4_K_S	+0.2689
Q4_0	+0.4685
Q3_K_L	+0.5562
Q3_K_M	+0.6569
Q3_K_S	+1.6321
Q2_K	+3.5199

Two details matter. First, the “K” types beat the older plain types at similar size: Q4_K_M adds 0.18 perplexity while Q4_0 adds 0.47, for a file that is only marginally larger. Second, the gap between Q3_K_M (+0.66) and Q3_K_S (+1.63) shows that the mix of precisions across tensors matters as much as the headline bit count — a point I return to in the deep dive.

Size versus bits: the old rule, and why it broke

For years the community rule of thumb was: at a fixed memory budget, take the bigger model at fewer bits. The original k-quants pull request, which introduced the Q2_K–Q6_K family in June 2023, contains the data behind that rule (PR #1684). On LLaMA-1, every 13B quant — even Q2_K at 5.13 GB — had lower perplexity than the 7B model at full precision (13.0 GB).

Exhibit 5

LLaMA-1 7BLLaMA-1 13B

File sizes from the same table: 7B F16 13.0 GB, 7B Q6_K 5.15 GB; 13B Q2_K 5.13 GB, 13B Q4_K_M 7.32 GB. At about 5 GB, 13B Q2_K (5.85) beat 7B Q6_K (5.91).

Source: llama.cpp PR #1684 (k-quants), June 2023

View data table

Type	7B	13B
Q2_K	6.7764	5.8545
Q3_K_M	6.1503	5.4498
Q4_K_M	5.9601	5.3002
Q5_K_M	5.9208	5.2638
Q6_K	5.911	5.2568
F16	5.9066	5.2543

That rule does not transfer cleanly to modern models. Compare the perplexity penalties for the same quant types on LLaMA-1 7B (from PR #1684) and on Llama-3-8B (from quantize.cpp). Llama 3 pays roughly three to four times as much at every level, and at Q2_K the penalty jumps from 0.87 to 3.52. An academic study of Llama 3 quantization reached the same qualitative conclusion: “non-negligible degradation”, “particularly under ultra-low bit widths” (Huang et al.). The usual explanation — which I offer as a hypothesis, not a finding — is that models trained on far more tokens pack more information into each weight, leaving less redundancy for quantization to discard.

Exhibit 6

Q4_K_M · LLaMA-1 7B+0.05

Q4_K_M · Llama-3-8B+0.18

Q3_K_M · LLaMA-1 7B+0.24

Q3_K_M · Llama-3-8B+0.66

Q2_K · LLaMA-1 7B+0.87

Q2_K · Llama-3-8B+3.52

Directional comparison only: different tokenizers, evaluation runs and llama.cpp versions. Perplexities are not comparable across models in absolute terms; the deltas give a rough sense of fragility.

Source: llama.cpp PR #1684 (LLaMA-1 7B, computed from published perplexities); llama.cpp quantize.cpp (Llama-3-8B)

View data table

Type	LLaMA-1 7B	Llama-3-8B
Q4_K_M	+0.054	+0.175
Q3_K_M	+0.244	+0.657
Q2_K	+0.870	+3.520

The practical consequence is a revised rule: the largest model that fits at 4–5 bits, never below about 3.5 bits without measuring on your own task. “Bigger at 2 bits” was good advice for 2023 models; for 2025–2026 models it is a gamble.

Speed: smaller weights, faster tokens

Generating each token requires reading essentially every weight from memory once, so on consumer hardware generation speed is largely limited by memory bandwidth. Fewer bytes per weight means more tokens per second. In llama.cpp’s current README, Llama-3.1-8B generated 71.9 tokens per second at Q4_K_M against 29.2 at F16 — about 2.5 times faster — on hardware the README does not specify. The k-quants PR gives a hardware breakdown for the 7B model (Exhibit 7).

Exhibit 7

RTX 4080 · Q4_K_M62.5 t/s

RTX 4080 · F1616.7 t/s

M2 Max · Q4_K_M25.0 t/s

M2 Max · F169.0 t/s

Ryzen 7950X · Q4_K_M14.1 t/s

Ryzen 7950X · F164.7 t/s

2023 llama.cpp build; current builds and models are faster in absolute terms. Ryzen figures use 4 threads; M2 Max uses 8. The ratio, not the absolute number, is the point.

Source: llama.cpp PR #1684 (k-quants), June 2023

View data table

Hardware	F16 (ms/tok)	Q4_K_M (ms/tok)	Q4_K_M (t/s)
RTX 4080	60	16	62.5
Apple M2 Max (8 threads)	111	40	25
Ryzen 9 7950X (4 threads)	214	71	14.1

Speed is a quality attribute in production. A model that streams at 60 tokens per second feels instant; one at 5 tokens per second makes a 300-token answer take a minute, and users abandon it. That is why the CPU rows matter: a 7B model at Q4_K_M on a desktop CPU is usable for a single user, while the same model at F16 is not.

A framework: fit first, then choose

Most local-LLM projects I see start with “which model is best?” and end with a model that does not fit, spills into system RAM and crawls. I reverse the order. The decision stack below starts from the workload and the memory budget, and only then picks a model.

Exhibit 8

1. 01EligibilityIs there a residency, offline, control or high-volume reason? If not, use a hosted API.
2. 02Memory budgetVRAM (or RAM) minus KV cache for your real context and concurrency, minus ~1 GB of buffers.
3. 03Quant floorQ4_K_M or IQ4_XS as default; Q5_K_M–Q6_K if it fits; below ~3.5 bpw only with evidence.
4. 04Largest model that fitsPick the strongest licence-compatible model whose file fits the remaining budget.
5. 05Runtime and servingOllama, LM Studio or llama.cpp behind an authenticated OpenAI-compatible endpoint.
6. 06Evaluate on your task50–100 real questions, scored against the hosted baseline. Perplexity is not your metric.

The framework encodes the thesis. Steps two and three — the memory budget and the quant floor — determine what is possible before any model is named. Step six is the guard rail: perplexity is a useful proxy for comparing quants of one model, but it says little about whether a model can follow your instructions, cite your documents or refuse when it does not know.

Deep dive 1: how quantization actually works

It helps to know what the letters mean, because they predict behaviour.

Block-wise quantization. GGUF quant types do not use a single scale for a whole tensor. They split weights into small blocks and store a scale per block, so an outlier in one block does not ruin precision everywhere else. The simplest types are “type-0”, where a weight is reconstructed as w = d × q (a block scale times a small integer), and “type-1”, which adds a block minimum: w = d × q + m. Q4_0 and Q8_0 are the classic plain types.

K-quants. The k-quants introduced in PR #1684 add a second level. Weights are grouped into super-blocks of 256, split into sub-blocks of 16 or 32; the sub-block scales (and minimums) are themselves quantized to 4–8 bits, with one higher-precision scale per super-block (PR #1684). That is why Q4_K is about 4.5 bpw and Q6_K about 6.56. The suffixes _S, _M and _L are mixes: the same base type for most tensors, but higher precision for the ones that hurt most when compressed. In the PR’s definition, Q4_K_M uses Q6_K for half of the attention.wv and feed_forward.w2 tensors and Q4_K elsewhere; Q3_K_M promotes attention.wv, attention.wo and feed_forward.w2 to Q4_K. This is the mechanism behind Exhibit 4’s gap between Q3_K_S and Q3_K_M: spending a few extra bits in the right tensors buys a large share of the quality back.

I-quants. The IQ types, introduced in early 2024, use a different trick at very low bit rates. IQ2_XXS, for example, quantizes weights in groups of eight and encodes their magnitudes as an index into a 256-entry codebook drawn from the E8 lattice, with signs stored using a parity trick — ideas the author credits partly to the QuIP# research (PR #4773). The effect is better quality per bit at 2–4 bpw, at the cost of slower decoding on some hardware. IQ4_XS (4.46 bpw) is a useful alternative to Q4_K_M when a model is just slightly too big.

Importance matrices. Not all weights matter equally. llama.cpp’s imatrix tool runs calibration text through the model and records, for each weight column, the average squared activation; the quantizer then minimises error weighted by that importance rather than treating every weight alike (PR #4861; imatrix README). Importance matrices matter most at low bit rates. The practical upshot: when downloading 3-bit or IQ quants, prefer publishers who state they used an imatrix, and if you quantize yourself, calibrate on text that resembles your workload.

# Build an importance matrix from calibration text, then quantize with it
llama-imatrix -m model-f16.gguf -f calibration-data.txt -ngl 99
llama-quantize --imatrix imatrix.gguf model-f16.gguf model-IQ4_XS.gguf IQ4_XS

Quantization-aware training. QAT, as in Google’s Gemma 3 release, goes further: the model is trained while simulating low-precision arithmetic, so it learns weights that survive quantization. Google reports that QAT reduced the perplexity drop at Q4_0 by 54% (Google). When a vendor ships an official QAT checkpoint, it is usually the best 4-bit option available.

Failure mode and how to measure. The characteristic failure of over-quantization is not gibberish; it is subtle drift — slightly worse instruction following, more formatting mistakes, weaker recall of rare facts. You will not see it in a demo. Measure it with a fixed evaluation set run against two or three quants of the same model, and compare against a hosted baseline.

Deep dive 2: the memory arithmetic

A model needs memory for three things: its weights, its key-value (KV) cache, and working buffers.

Weights are approximately parameters × bits per weight ÷ 8. For Llama 3.1 8B at Q4_K_M: about 8 billion × 4.89 ÷ 8 ≈ 4.9 GB, matching the 4.92 GB file on Hugging Face (bartowski). For Llama 3.3 70B at Q4_K_M the file is 42.52 GB; at IQ2_XS it is 21.14 GB; at F16 it is 141.12 GB (bartowski).

KV cache is the part people forget. For each token of context, the model stores a key and a value vector in every layer:

KV bytes per token = 2 (K and V) × layers × KV heads × head dimension × bytes per element

The model configs give the inputs. Llama 3.1 8B has 32 layers, 8 KV heads and a head dimension of 128; Llama 3.3 70B has 80 layers, 8 KV heads and a head dimension of 128 (8B config; 70B config). At FP16 (2 bytes), that is 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KiB per token, for the 8B model, and 2 × 80 × 8 × 128 × 2 = 327,680 bytes, or 320 KiB per token, for the 70B model. Multiply by context length.

Exhibit 9

8B · 8K1.1 GB

8B · 32K4.3 GB

8B · 128K17.2 GB

70B · 8K2.7 GB

70B · 32K10.7 GB

70B · 128K43.0 GB

Estimate for one sequence. Parallel requests multiply it. A q8_0 KV cache roughly halves these figures, per Ollama’s documentation.

Source: Author’s calculation from model configs (Hugging Face) and the standard KV-cache formula

View data table

Model · context	KV cache (GB)
8B · 8K	1.07
8B · 32K	4.29
8B · 128K	17.18
70B · 8K	2.68
70B · 32K	10.74
70B · 128K	42.95

Buffers and headroom. Runtimes also allocate compute buffers, and Windows reserves some VRAM for the desktop. My rule of thumb is 0.5–1 GB, more on cards that also drive displays. Put together, the worked examples look like this:

Exhibit 10

	Weights	KV cache	Total (est.)	Realistic home
8B Q4_K_M · 8K	4.9 GB	1.1 GB	~7 GB	8 GB GPU (tight) or 12 GB
8B Q4_K_M · 32K	4.9 GB	4.3 GB	~10 GB	12–16 GB GPU; ~8 GB with q8_0 KV
8B Q8_0 · 8K	8.5 GB	1.1 GB	~10.5 GB	12–16 GB GPU
8B Q4_K_M · 128K	4.9 GB	17.2 GB	~23 GB	24 GB GPU — context costs more than the model
70B IQ2_XS · 8K	21.1 GB	2.7 GB	~25 GB	Just over 24 GB, and below the quality floor
70B Q4_K_M · 8K	42.5 GB	2.7 GB	~46 GB	Two 24 GB GPUs, or a slow GPU+CPU split

Estimates for a single request. With OLLAMA_NUM_PARALLEL or llama.cpp slots above 1, multiply the KV column.

Source: Weights: bartowski GGUF repos (Hugging Face); KV cache: author’s calculation; buffers: author’s rule of thumb

The most important row is the fourth. Teams often pick a small model to save memory and then set a 128K context “because the model supports it”, quietly spending four times the model’s size on cache. Set context to what the task needs. For retrieval-augmented generation, 8K–16K is usually enough: you are sending the top few chunks, not the library.

Failure mode and measurement. When a model does not fit in VRAM, runtimes offload layers to the CPU. It still works, so nobody notices — except that generation speed drops sharply. On Ollama, ollama ps shows how much of a loaded model sits on the GPU versus the CPU; anything other than 100% GPU on a latency-sensitive service is a bug.

Deep dive 3: hardware tiers on Windows

The runtime documentation sets the floor. Ollama requires Windows 10 22H2 or newer (Ollama, Windows). For NVIDIA, it supports compute capability 5.0 and above with driver 550 or newer; cards of compute capability 5.0–6.2 need driver 570 or newer (Ollama, GPU) — the Windows page itself lists 551.61 as the minimum, so simply install a current driver. For AMD on Windows, ROCm acceleration needs a ROCm v7 / HIP7-capable driver and a supported card — RX 7900 XTX, 7900 XT, 7900 GRE, 7800 XT, 7700 XT, 7600 XT, 7600 or the Radeon PRO W7000 series. Everything else can use Vulkan, which Ollama enables by default when the backend is installed (set OLLAMA_VULKAN=0 to disable it). llama.cpp itself supports CUDA, HIP, Vulkan and SYCL (for Intel GPUs), among other backends (llama.cpp). LM Studio requires AVX2 on x64 CPUs, runs on Snapdragon X Elite machines, and recommends 16 GB of RAM and 4 GB of dedicated VRAM (LM Studio).

Exhibit 11

	Typical hardware	What runs well	Start with
CPU only, 16–32 GB RAM	AVX2 laptop or desktop, no discrete GPU	1–4B models interactively; 7–8B for a single patient user	Q4_K_M
8 GB VRAM	RTX 4060 / laptop GPUs	7–9B at 4–8K context; Gemma 3 12B QAT per Google	Q4_K_M or vendor QAT
12–16 GB VRAM	RTX 5060 Ti 16 GB class	8B at Q8_0 or long context; 12–14B at Q4–Q5; gpt-oss-20b	Q5_K_M–Q6_K
24–32 GB VRAM	RTX 3090 / 4090 / 5090 class	24–32B at Q4 (Gemma 3 27B QAT is 14.1 GB)	Q4_K_M, IQ4_XS
AMD Radeon	RX 7600–7900 XTX	As per VRAM tier, via ROCm v7; older cards via Vulkan	Same as NVIDIA tier
70B ambitions	Single consumer GPU	Does not fit at 4 bits (~46 GB with 8K context)	Two GPUs, a workstation card, or a smaller model

Author’s guidance. Exact fit depends on architecture, context length and concurrency; verify with the arithmetic in Exhibit 10.

Two practical notes. First, laptop GPUs share thermal budgets with the CPU; sustained throughput on a thin laptop is lower than its first minute suggests, so benchmark for ten minutes, not ten seconds. Second, if you are buying hardware for this, VRAM capacity matters more than raw compute: a 16 GB mid-range card runs more useful models than a faster 8 GB one.

Deep dive 4: runtimes, setup and serving

Three runtimes cover almost every Windows use case. They share the GGUF format, so a model file works across them.

Exhibit 12

	Ollama	LM Studio	llama.cpp
Install	winget install Ollama.Ollama, or OllamaSetup.exe (no admin rights)	winget install ElementLabs.LMStudio	winget install llama.cpp, or the PowerShell install script
Interface	Background service + CLI	Desktop GUI with model browser	CLI and server binaries
API	OpenAI-compatible at localhost:11434/v1	OpenAI-compatible at localhost:1234/v1	OpenAI-compatible chat, responses and embeddings routes
Concurrency	OLLAMA_NUM_PARALLEL (default 1); queue of 512	Managed by the app	Server slots (-np), continuous batching
Auth	None built in; binds 127.0.0.1 by default	Local server; treat as unauthenticated	--api-key; binds 127.0.0.1 by default
Best for	A shared internal service on one box	Comparing quants and prompts quickly	Tuning, custom quants, unusual hardware

Defaults as documented in October 2026; check release notes, as these projects change monthly.

Step 1 — install and run a first model (Ollama). Ollama installs into your user account without administrator rights and serves an API on http://localhost:11434 (Ollama, Windows). You can pull a specific GGUF quant straight from Hugging Face:

winget install Ollama.Ollama
# or: irm https://ollama.com/install.ps1 | iex

ollama run hf.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M
ollama ps   # check the model is 100% on GPU

Step 2 — set context and memory deliberately. Ollama’s documented default context window is 4,096 tokens, overridable with OLLAMA_CONTEXT_LENGTH or per request with num_ctx (Ollama FAQ). The KV cache can be quantized with OLLAMA_KV_CACHE_TYPE when Flash Attention is on: q8_0 uses about half the memory of f16 “with a very small loss in precision”; q4_0 about a quarter, with a loss that “may be more noticeable at higher context sizes”. Models live in C:\Users\%USERNAME%\.ollama\models unless you set OLLAMA_MODELS. On Windows, set these as user environment variables, then quit Ollama from the system tray and restart it:

setx OLLAMA_CONTEXT_LENGTH 16384
setx OLLAMA_FLASH_ATTENTION 1
setx OLLAMA_KV_CACHE_TYPE q8_0
setx OLLAMA_NUM_PARALLEL 2
setx OLLAMA_MODELS "D:\models\ollama"

Step 3 — or use LM Studio to compare quants. LM Studio is the fastest way to try three quants of the same model side by side with a GUI. Since July 2025 it has been free for use at work as well as at home (LM Studio).

winget install ElementLabs.LMStudio

Step 4 — or llama.cpp for full control. llama.cpp’s README now documents a unified llama command, plus a PowerShell installer (irm https://llama.app/install.ps1 | iex); older releases name the binaries llama-cli and llama-server. The server supports an API key, a context size, parallel slots and KV-cache types (-c, -np, --api-key, -ctk/-ctv), and binds to 127.0.0.1 by default (server README).

winget install llama.cpp
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF      # quick interactive test
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF    # OpenAI-compatible server

Step 5 — serve it to applications. All three expose OpenAI-compatible endpoints, which means existing client code needs only a base URL change. With Ollama the API key is required by the client library but ignored by the server (Ollama):

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1/", api_key="ollama")
reply = client.chat.completions.create(
    model="hf.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M",
    messages=[{"role": "user", "content": "Summarise this clause in one sentence: ..."}],
)
print(reply.choices[0].message.content)

That portability is strategically important. If your application talks to an OpenAI-compatible interface, you can move a workload between local and hosted models by configuration — and you should design for exactly that, because the right answer for a workload changes as models and prices move.

Production concerns

Concurrency. Ollama processes one request per model at a time by default and queues up to 512 more; raising OLLAMA_NUM_PARALLEL increases the context memory proportionally (Ollama FAQ). A single consumer GPU is a service for a team, not for a company. Load-test with realistic prompt lengths before you promise anyone a response time.

Context length. Set it explicitly and test at the limit. A silently truncated prompt is one of the most common causes of “the local model is worse” complaints in retrieval systems: the retrieved passages never reached the model.

Network exposure. Ollama binds to 127.0.0.1 by default and has no built-in authentication. Changing OLLAMA_HOST to expose it on a network without putting authentication in front is how incidents happen. In September 2025, Cisco researchers used Shodan to find 1,139 exposed Ollama servers, 214 of them actively hosting models (Cisco). Put a reverse proxy with authentication, or llama.cpp’s --api-key, in front of anything reachable from another machine, and restrict it to trusted networks.

Model-file security. A GGUF file is data, but the parser that reads it is code. In February 2024, Cisco Talos disclosed heap-overflow vulnerabilities in llama.cpp’s GGUF parsing, including CVE-2024-23605 (CVSS 8.8), where “a specially crafted .gguf file can lead to code execution” (Talos). Treat model files like executables: download from known publishers, record a hash (Get-FileHash model.gguf -Algorithm SHA256), store approved files in a controlled location and keep runtimes updated.

Licences. “Open weights” is not one licence. Gemma 4, Qwen3.8, Mistral Small 4 and gpt-oss are Apache-2.0; DeepSeek V4.1-Flash is MIT; Llama models use Meta’s own licences. The Llama 3.1 Community License, for example, requires a separate licence if your products exceeded 700 million monthly active users, asks you to “prominently display ‘Built with Llama’” and binds you to an acceptable use policy (Meta). Read the model card before production, and record the licence alongside the hash.

Updates. Pin versions. A new runtime release can change defaults (context length, KV handling, GPU selection); a re-uploaded GGUF can change behaviour. Upgrade on a schedule, re-run your evaluation set, then promote.

Case examples

Google’s Gemma 3 QAT release (April 2025). Google trained 4-bit-aware versions of Gemma 3 and published them for consumer GPUs, cutting the 27B model’s weights from 54 GB to 14.1 GB and reducing the Q4_0 perplexity drop by 54% compared with naive quantization (Google). Lesson: when a vendor ships a QAT checkpoint, it moves the quality-size frontier; start there.

OpenAI’s gpt-oss (August 2025). OpenAI post-trained its open-weight models with MXFP4 quantization of the mixture-of-experts weights and evaluated them in that form; the 20B model runs within 16 GB of memory under Apache-2.0 (model card). Lesson: quantization is no longer something the community does to a model after release — it is part of how frontier labs design open models for local hardware.

Exposed Ollama servers (September 2025). Cisco’s scan found over a thousand Ollama instances on the public internet without authentication (Cisco). Lesson: the easiest local runtime is also easy to misconfigure. The model was never the risk; the endpoint was.

Field notes: an offline RAG system. The privacy-first RAG system I mentioned at the start runs entirely on local hardware: Ollama serves Gemma 2 (2B or 9B) for generation and nomic-embed-text for embeddings, ChromaDB stores vectors on local disk, a FastAPI backend handles ingestion and queries, DVC versions each ingestion run, MLflow tracks retrieval experiments and Evidently monitors embedding and query drift. It uses CUDA where there is an NVIDIA GPU and Metal on a Mac, and it makes no outbound calls. Because the retrieval interface is identical to the cloud version, the only thing that changes between local and hosted is a model name.

Counterpoints

Implications by stakeholder

Executives and risk owners. Local inference converts a data-protection question into an infrastructure question. That is a good trade for regulated data, and a poor one for everything else. Ask for the eligibility case (step one of Exhibit 8) in writing, and expect to fund evaluation and security work, not just hardware.

Engineering leads. Design every LLM call behind an OpenAI-compatible interface with the model name in configuration, so workloads can move between local and hosted without code changes. Own the operational basics: authenticated endpoints, pinned model hashes, scheduled upgrades with regression runs, and monitoring of GPU residency and latency.

Practitioners. Learn the memory arithmetic and the quant ladder well enough to size a deployment on a napkin. Build a small, honest evaluation set early; it is the only thing that will settle arguments about which quant or model to use.

Security and compliance teams. Treat GGUF files and runtimes as software supply chain: approved sources, hashes, version control and patching. Treat a local LLM endpoint as an unauthenticated API until proven otherwise.

What to do next: a 90-day plan

Days 1–30: decide and size.

1. List candidate workloads and mark which have a residency, offline, control or volume reason. Drop the rest.
2. For each survivor, record the real context length and concurrency. Run the Exhibit 10 arithmetic.
3. Build an evaluation set of 50–100 real inputs with expected answers or rubrics, and score the hosted baseline.
4. On existing hardware, test two or three quants (Q4_K_M, Q5_K_M, Q6_K) of one or two licence-compatible models in LM Studio or Ollama.

Days 31–60: pilot. 5. Stand up Ollama or llama.cpp on one machine with explicit context, KV-cache type and parallelism settings. 6. Put authentication in front of it, restrict it to the internal network, and record model hashes and licences. 7. Run a pilot with real users; log latency, GPU residency and failure cases, and re-score the evaluation set.

Days 61–90: harden or hand back. 8. If the local model is within an acceptable margin of the baseline on your evaluation set, harden: monitoring, upgrade cadence, runbooks. 9. If not, try the next tier up (bigger model or higher quant) once. If it still fails, return the workload to a hosted model — and keep the interface, so you can try again when the next open-weight generation arrives.

Limitations and methodology

The quality evidence is perplexity, measured mostly on two models: Llama-3-8B for the current deltas and LLaMA-1 7B/13B for the size comparison. Perplexity is a proxy; it does not measure instruction following, grounding or reasoning, and its values are not comparable across models with different tokenizers. The cross-generation comparison in Exhibit 6 mixes evaluation runs and llama.cpp versions and should be read as directional.

Speed figures are hardware-dependent. The README’s 2.5× speed-up does not state its hardware; the PR #1684 timings are from a 2023 build on a 2023 model. Absolute speeds on current builds will differ; ratios are more stable. The memory figures are calculations from published configs and file sizes plus my own rule of thumb for buffers; your runtime will report slightly different numbers. The break-even estimate uses list prices on 14 October 2026, one GPU’s launch price, and an assumed token mix; it ignores electricity, the rest of the machine and the capability gap. I have not presented benchmarks of my own here, and the field note above is flagged for confirmation.

Closing

The question “can I run a real model on a Windows PC?” has been answered: yes, and with the right quant it is fast enough to use. The more useful question is whether you should, and if so, how to fit it. For workloads where data cannot leave, networks cannot be relied on, or volume is high and steady, a well-quantized open-weight model is a credible production choice. Make the decision in the right order — eligibility, memory budget, quant floor, then the largest model that fits — secure the endpoint like any other service, and let your own evaluation set, not a leaderboard, have the final word.