Running large language models (LLMs) like DeepSeek-R1, Llama-4, or Qwen-2.5 locally on consumer GPUs often hits a frustrating wall: RuntimeError: CUDA error: out of memory or sudden drops from 65 tokens/sec down to a painful 4 tokens/sec. In most cases, your GPU actually has enough raw VRAM for the model weights, but silent allocation spikes in the KV (Key-Value) Cache, unoptimized context windows, or WSL2 memory leaks push usage past physical limits.
1. Why Ollama and vLLM Crash With CUDA Out of Memory
When you load a 7B, 14B, or 32B parameter model, total GPU memory consumption is governed by three separate components:
- Model Weights: Fixed size determined by parameter count and quantization (e.g., Q4_K_M vs FP16).
- KV Cache (Context Buffer): Scales linearly or quadratically with context length (
num_ctx) and parallel request slots. - Compute & Activation Overhead: Temporary CUDA kernels and FlashAttention buffers (typically 0.8 GB to 2.2 GB).
If your weights + KV cache exceed physical VRAM by even 150 MB on Windows 11, the NVIDIA driver spills shared memory over PCIe into system DDR5 RAM, causing generation speeds to plummet by 85% or triggering an immediate OOM crash on Linux.
2. VRAM Usage Breakdown by Model Size & Quantization
| Model Size | FP16 Weights | Q8_0 Quant | Q4_K_M Quant | 8K Context KV Cache (FP16 / Q8) | Recommended Minimum GPU VRAM |
|---|---|---|---|---|---|
| 8B Parameters | 16.2 GB | 8.6 GB | 4.9 GB | 1.2 GB / 0.6 GB | 8 GB (RTX 4060 / RX 7600) |
| 14B Parameters | 28.4 GB | 15.1 GB | 8.9 GB | 2.1 GB / 1.1 GB | 12 GB (RTX 4070 / RTX 3060 12GB) |
| 32B Parameters | 64.5 GB | 34.2 GB | 19.8 GB | 4.4 GB / 2.2 GB | 24 GB (RTX 3090 / RTX 4090) |
| 70B Parameters | 140.0 GB | 74.5 GB | 42.5 GB | 8.8 GB / 4.4 GB | 48 GB (Dual RTX 3090 / Cloud A6000) |
3. Fix #1: Enable Flash Attention & Quantize the KV Cache in Ollama
By default, Ollama allocates the KV cache in 16-bit floating point (FP16). Switching KV cache storage to q8_0 or q4_0 cuts context memory usage by 50% to 75% with virtually zero perplexity loss when paired with Flash Attention.
On Linux (systemd service), edit your Ollama environment configuration:
RAM Speed & True Latency Calculator
Calculate exact real response times in nanoseconds (ns) and dual-channel bandwidth for AMD Ryzen & Intel PC builds.
Reload and restart the service:
Setting OLLAMA_NUM_PARALLEL=1 is critical for single-user workstations. Without it, Ollama pre-allocates 4 parallel context slots, multiplying your KV cache footprint by 4x automatically.
4. Fix #2: Cap Context Window (`num_ctx`) & Prevent System RAM Spillover
Many third-party WebUIs (Open WebUI, AnythingLLM, or IDE coding extensions like Continue and Roo Code) request a 32,768 or 65,536 token context window. On a 32B model, a 32K context window demands over 12 GB of VRAM for the KV cache alone. Lock a custom Modelfile with a predictable context boundary:
Build and run the tuned profile:
5. Fix #3: Tune vLLM `--gpu-memory-utilization` and `--max-model-len`
When deploying production endpoints with vLLM, the engine pre-allocates 90% of total GPU VRAM on startup (`--gpu-memory-utilization 0.90`). If your display server, browser, or background Docker container is using 1.5 GB of VRAM, vLLM will crash during CUDA graph capture.
--enforce-eager: Disables heavy CUDA graph pre-allocation, saving 1.2 GB to 2.5 GB of VRAM on 12 GB and 16 GB cards.--max-model-len 8192: Prevents vLLM from trying to reserve KV blocks for the model's native 128K maximum length.
6. Fix #4: Disable NVIDIA Sysmem Fallback on Windows 11 & WSL2
On Windows 11, NVIDIA Driver 560+ automatically overflows full VRAM into system RAM instead of throwing an error, dropping speeds from 50 tok/s to 2 tok/s. Open NVIDIA Control Panel → Manage 3D Settings → CUDA - Sysmem Fallback Policy and set it to Prefer No Sysmem Fallback. In WSL2, create %UserProfile%\.wslconfig to release zombie cached memory:
Apply these four optimizations and your local inference stack will maintain peak memory bandwidth without unexpected CUDA OOM terminations.