Tag: PyTorch

Three different “vLLM CUDA OOM” problems

The phrase “vLLM won’t stop throwing CUDA OOM” covers three failures with different causes and different fixes. Identify which one you have before changing anything.

1. Startup: not enough memory for the KV cache

The server refuses to start and prints a ValueError like:

ValueError: To serve at least one request with the model's max seq len (40960), (5.62 GiB KV cache is needed, which is larger than the available KV cache memory (4.89 GiB). Based on the available memory, the estimated maximum model length is 35600. Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.

(The mismatched parenthesis after the sequence length is in vLLM’s own message.) It means: after loading the weights and reserving overhead, the memory left for the KV cache cannot hold even one request at max_model_len tokens. This is a sizing problem, fixed with init arguments — covered below.

2. Runtime: an allocation failure under load

The server is up and serving, then a request or a burst of them fails. There are two distinct sub-cases, and only one is graceful:

• KV-cache block pressure. Too many concurrent sequences need more KV cache blocks than the pool has. On the V1 engine the scheduler preempts the newest sequences and later recomputes them (the default RECOMPUTE mode), so this shows up as higher latency and lower throughput, not errors. Nothing crashes.
• Allocator OOM outside the pool. A prefill spike, activation buffers, CUDA-graph capture, or a second process on the GPU pushes transient usage past the free margin, and PyTorch raises:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU 0 has a total capacity of 23.99 GiB of which 1.10 GiB is free. ... If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.

Recompute preemption does not protect against this one — it only rations KV cache blocks, not activation or graph memory. An allocator OOM here can take down the EngineCore subprocess.

3. The persistent loop: every request fails after one bad one

One large request triggers an OOM, and from then on every request — even trivial ones — returns a 500. Three sub-causes, and they need different responses:

• A dead engine (most common). The OOM exception escaped into the engine’s core loop and killed it. On current vLLM the GPU subprocess is gone and every request raises EngineDeadError (with EngineCore ... died messages in the logs). On an old pinned vLLM (before the V0 engine was removed around v0.10) the async background task died instead, raising AsyncEngineDeadError: Background loop has errored already. Neither is recoverable in place — the maintainers’ guidance is to restart the process. No allocator setting brings a dead engine back.
• Allocator fragmentation. The engine is alive but PyTorch is holding reserved-but-unusable memory in fragmented pools; nvidia-smi shows free memory in aggregate while no single block is large enough. This is the case expandable_segments:True addresses.
• Under-provisioning. The spike that OOMs keeps recurring because the server is sized past its GPU. The fix is the startup sizing covered further down, not recovery.

Sources: vLLM issue #38516 (KV cache ValueError), vLLM issue #16118 (estimate max-model-len)

Can you fix it without restarting? Mostly no — here is what is live

Be clear-eyed about what is actually adjustable while the server runs.

Init-only — changing these means restarting vllm serve: gpu_memory_utilization, max_model_len, max_num_seqs, max_num_batched_tokens, kv_cache_dtype, enforce_eager, tensor_parallel_size, pipeline_parallel_size, quantization, swap_space. These are all EngineArgs, baked in when the engine starts.

Live, right now, no restart:

• Client-side pressure. The only real serving-side lever on a running engine is sending less: lower client concurrency, cap max_tokens per request, shorten prompts. This immediately reduces peak KV cache and activation memory and will pull a server back from the edge.
• LoRA adapters. If the pressure comes from loading many fine-tunes, start with VLLM_ALLOW_RUNTIME_LORA_UPDATING=True and add or drop adapters through POST /v1/load_lora_adapter and POST /v1/unload_lora_adapter, with no interruption to in-flight traffic. vLLM’s docs warn against enabling this outside a trusted, isolated environment, since it can load arbitrary adapters.

One restart, then durable:

• PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True — an environment variable read by PyTorch’s CUDA allocator, not by vLLM, so it is picked up on the next process start. It switches the allocator to growable segments that resist fragmentation, and the PyTorch OOM message itself recommends it. After that one restart it greatly reduces the fragmentation form of the loop without needing further restarts.

Automatic, nothing to set:

• KV-cache preemption. The V1 scheduler’s default RECOMPUTE preemption lets the server ration KV cache blocks under load instead of erroring. It does not cover allocator OOMs from activation or graph memory. This is automatic on current vLLM, which ships only the V1 engine.

Sources: vLLM optimization docs, vLLM conserving memory docs

Fixing the startup KV-cache OOM

For the startup ValueError, the goal is to make the KV cache fit. In rough order of preference:

• Lower --max-model-len to what your workload actually needs. vLLM prints an estimate (“estimated maximum model length is 35600”) — use that as a ceiling. The startup check requires enough KV cache memory to serve one request at the full max_model_len; a context window you never use just raises that bar.
• Raise --gpu-memory-utilization if the GPU is dedicated to this server. The default is 0.9; 0.92–0.95 is safe on a card doing nothing else. Do not go to 1.0 — PyTorch’s allocator and prefill spikes need headroom.
• Compress the KV cache with --kv-cache-dtype fp8 (or fp8_e4m3 / fp8_e5m2). This halves KV cache bytes versus fp16 for a small, usually acceptable quality cost, and often single-handedly resolves the error.
• Cap concurrency with --max-num-seqs (and --max-num-batched-tokens). Fewer simultaneous sequences means a smaller KV cache reservation.
• Shard the model with --tensor-parallel-size N (splits weights across N GPUs, freeing room for KV cache on each) or --pipeline-parallel-size N (splits layers).
• Quantize the weights (AWQ, GPTQ, FP8) so the model itself occupies less, leaving more for the cache.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.92 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 64

Sources: vLLM optimization docs, vLLM issue #16118

Fixing runtime OOM under load

For torch.OutOfMemoryError while serving, the KV cache pool is sized fine but peak usage overflows the margin around it.

• Lower --gpu-memory-utilization to 0.85–0.90. Counter-intuitively, giving vLLM less of the card leaves more raw headroom for the transient allocations — prefill, activations, CUDA graphs — that live outside the reserved pool. On a shared host, 0.80.
• Add --enforce-eager. CUDA-graph capture holds a few hundred MB of static memory and can spike while capturing. Disabling it (at a modest decode-throughput cost) reclaims that room. If you want graphs but less memory, shrink compilation_config.cudagraph_capture_sizes instead.
• Cap --max-num-seqs / --max-num-batched-tokens. These bound how many sequences and how much prefill can be in flight at once, the main driver of runtime spikes.
• Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True before launch so a spike that is recovered does not strand fragmented memory behind it.
• Check for other processes on the GPU (nvidia-smi) — a second CUDA process that grabs memory after vLLM’s pre-allocation is a classic cause of “it worked yesterday” OOMs.

Watch the distinction drawn earlier: if you are only seeing latency climb and throughput sag with no errors, that is KV-cache preemption doing its job under load, and the answer is to reduce traffic or resize the deployment rather than to keep tuning the allocator margin.

Sources: vLLM optimization docs, vLLM OOM root-cause diagnosis

When every request fails after one bad one

When every request fails after one bad one, work through this in order:

1. Check whether the engine is dead. Grep the logs for EngineDeadError and EngineCore ... died, or AsyncEngineDeadError: Background loop has errored already on an old pinned vLLM. If you see either, the engine loop is gone and no in-place setting revives it. Restart the server — and make that restart cheap: put vLLM behind a load balancer or a Kubernetes Deployment with a readiness probe so a rolling restart drains and replaces the instance without dropping traffic. “Without restarting” is the wrong goal here; “restart without downtime” is the achievable one.
2. If the engine is alive but every request still OOMs, suspect fragmentation. Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and restart once. With expandable segments PyTorch grows and shrinks one pool instead of stranding memory in fragmented blocks, which greatly reduces this form of the loop. After that restart you should not need another for fragmentation.
3. Make sure the V0 engine is not pinned. Current vLLM only has the V1 engine, whose GPU-subprocess isolation keeps a crash from propagating straight into the request handler. If an old deployment still sets VLLM_USE_V1=0, upgrading off V0 removes the AsyncEngineDeadError failure mode entirely.
4. If it keeps recurring, you are under-provisioned. Apply the startup-sizing fixes from the earlier section (lower max_model_len, fp8 KV cache, lower max_num_seqs) so the spike never happens, rather than recovering from it again and again.

Sources: vLLM GPU OOM CrashLoopBackOff runbook

Preventing it: a starting configuration

A starting point that avoids most OOM incidents on a single dedicated GPU:

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

vllm serve <model> \
  --max-model-len <your real p99 context, not the model max> \
  --max-num-seqs <your real concurrency ceiling> \
  --gpu-memory-utilization 0.90 \
  --kv-cache-dtype fp8

The principle: size max_model_len and max_num_seqs to the workload you actually have, not the model’s maximums; keep gpu_memory_utilization around 0.90 so there is headroom outside the pool; turn on fp8 KV cache unless you have measured a quality regression; and set the allocator env var so a recovered spike leaves less fragmentation behind. Load-test at your real concurrency before production — the KV cache math depends on numbers you only know at runtime.

Sources: vLLM optimization docs

Closing thoughts

The honest version of “solve the vLLM OOM loop without restarting the server” is: prevent the OOM with correct init sizing, set expandable_segments:True once so fragmentation is far less likely to make it persistent, and lean on the V1 engine’s subprocess isolation and recompute preemption so KV-cache pressure costs you latency instead of a wedged server.

What you genuinely do live is narrow — client-side load reduction now, LoRA hot-swap, and allocator config that applies on the next start. Everything else is an engine-init argument, so the practical target is not “never restart” but “restart without downtime”: a load balancer or a rolling Kubernetes deployment turns a required restart into a non-event. Chasing in-place recovery of a dead engine — EngineDeadError, or AsyncEngineDeadError on an old pinned V0 — is effort better spent on sizing the server so it does not die.

Frequently Asked Questions

How do I fix “CUDA out of memory” in vLLM?

First identify which OOM it is. A startup ValueError: To serve at least one request ... is a KV-cache sizing problem: lower --max-model-len, raise --gpu-memory-utilization, or add --kv-cache-dtype fp8. A torch.OutOfMemoryError while serving is a headroom problem: lower --gpu-memory-utilization to 0.85, add --enforce-eager, cap --max-num-seqs, and set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. All of those except the env var require restarting the server.

What does “To serve at least one request with the model’s max seq len” mean?

After loading weights and reserving overhead, the memory left for the KV cache cannot hold a single request at max_model_len tokens. vLLM prints an estimated maximum model length; set --max-model-len at or below it, raise --gpu-memory-utilization, or use --kv-cache-dtype fp8 to make each token’s KV entry smaller.

Can I change gpu_memory_utilization without restarting vLLM?

No. It is an engine-init argument, fixed when vllm serve starts. The same is true of max_model_len, max_num_seqs, kv_cache_dtype, enforce_eager, and tensor_parallel_size. Changing any of them means a restart.

What is a good gpu_memory_utilization value?

0.90 on a dedicated GPU, 0.92–0.95 if the card is doing nothing else, 0.80–0.85 on a shared host or if you get runtime OOMs. Never 1.0 — PyTorch’s allocator and prefill spikes need memory outside vLLM’s reserved pool. The default is 0.9.

Does --swap-space help with OOM on the vLLM V1 engine?

Effectively no. V1’s default preemption mode is RECOMPUTE, and V1 dropped host-memory swap for the KV cache, so --swap-space does not offload blocks to CPU the way it did on V0. Rely on recompute preemption and correct sizing instead.

How do I stop vLLM OOM errors from persisting across every request?

First check the logs: EngineDeadError (or AsyncEngineDeadError: Background loop has errored already on an old pinned vLLM) means the engine loop is dead and only a restart fixes it. If the engine is alive but every request still OOMs, set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and restart once — that greatly reduces the allocator-fragmentation form of the loop. Current vLLM’s recompute preemption also keeps sustained KV-cache pressure from escalating into a crash.

Does vLLM restart itself after a CUDA OOM crash?

No. A dead engine (EngineDeadError, or AsyncEngineDeadError on old pinned vLLM) stays dead until the process is restarted externally. Run vLLM behind a load balancer or a Kubernetes Deployment with a readiness probe so that restart drains and replaces the instance without dropping traffic.

Keep reading: LLM Inference & Serving

• QuaRot: Enabling End-to-End 4-Bit LLM Inference via Randomized Rotations — Learn how QuaRot achieves end-to-end 4-bit LLM inference across weights, activations, and KV cache by eliminating outliers with randomized Hadamard…
• Troubleshooting vLLM Startup: Resolving KV Cache Capacity Exceeded Errors — Fix vLLM ValueError: model’s max seq len is larger than KV cache capacity. Learn how adjusting max_model_len, gpu_memory_utilization, and FP8 solves…
• Multi-Head Latent Attention: Scaling LLM Serving Throughput via Low-Rank KV Compression — Discover Multi-Head Latent Attention (MLA) and how low-rank KV cache compression boosts LLM inference throughput while slashing memory footprint.
• Troubleshooting ‘ValueError: Some modules are dispatched on the CPU or the disk’ in Transformers — Fix ‘ValueError: Some modules are dispatched on the CPU or the disk’ in Hugging Face Transformers. Learn why quantized models fail to fit in GPU VRAM.

All guides in this series: LLM Inference & Serving Guides: vLLM, Ollama & Hugging Face