Tag: GPU Memory Management

  • QuaRot: Enabling End-to-End 4-Bit LLM Inference via Randomized Rotations

    QuaRot: Enabling End-to-End 4-Bit LLM Inference via Randomized Rotations

    TL;DR

    • QuaRot enables full 4-bit quantization across weights, activations, and the KV cache without retaining high-precision channels.
    • The framework suppresses disruptive activation outliers by using Randomized Hadamard Rotations via computational invariance to disperse peak magnitudes across dimensions.
    • Offline orthogonal rotations fuse directly into weights and normalization layers with zero inference overhead, while lightweight online Hadamard transforms handle internal activations and attention states.

    The Challenge of Outliers in Low-Bit LLM Quantization

    While weight-only 4-bit quantization (W4A16) is widely adopted, true end-to-end low-precision Post-Training Quantization (PTQ)—quantizing weights, activations, and the KV Cache to 4 bits (W4A4 Quantization with 4-bit KV Cache)—has historically resulted in severe accuracy degradation.

    The primary difficulty stems from Activation Outliers: persistent, extreme-value activations in specific channels with magnitudes orders of magnitude larger than standard features. Conventional uniform quantizers cannot allocate sufficient dynamic range to represent these channels without crushing non-outlier features to zero.

    Prior Post-Training Quantization approaches handle outliers through differing mechanisms that carry distinct trade-offs:

    • Outlier Channel Migration vs. Mixed-Precision Retention: SmoothQuant uses per-channel scaling factors to migrate quantization difficulty from activations to weights, targeting 8-bit quantization without keeping mixed-precision channels. In contrast, frameworks like LLM.int8() (and later works like QUIK and Atom) detect outliers dynamically or offline and retain those sensitive channels in higher precision (such as FP16).
    • Kernel and Hardware Overhead: For methods retaining mixed-precision channels, specialized execution kernels are required, introducing memory layout overhead, non-contiguous indexing, and hardware underutilization during matrix multiplication.

    Sources: GitHub – spcl/QuaRot: Code for Neurips24 paper: QuaRot, an end-to-end 4-bit inference of large language models, microsoft.com/en-us/research/publication/quarot-outlier-free-4-bit-in…, QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs, openreview.net/forum, Vercel Security Checkpoint

    What Is QuaRot and How Randomized Rotations Eliminate Outliers

    To eliminate the need for mixed-precision channels, QuaRot introduces an end-to-end Post-Training Quantization (PTQ) framework that achieves full W4A4 Quantization and 4-bit KV Cache compression. Rather than keeping outlier channels in FP16 during inference, QuaRot suppresses them across the model using Randomized Hadamard Rotations applied via Computational Invariance.

    A randomized Hadamard transformation multiplies a standard Hadamard matrix H by a diagonal matrix diag(s), where s is a vector of random signs drawn from {+1, -1}. A weight matrix W is defined as mu-incoherent if:

    max(W) <= mu * ||W||_F / sqrt(m * n)

    When incoherence is high, large individual elements emerge as outliers relative to the average magnitude. Multiplying by orthogonal randomized Hadamard matrices spreads high-energy spikes across all dimensions, systematically lowering incoherence and shaping activation distributions into uniform forms amenable to low-bit uniform quantizers.

    Under the principle of Computational Invariance, transformers remain invariant to orthogonal transformations across linear and normalization layers. Specifically, multiplying an input projection weight matrix on the left by Q^T and post-multiplying the preceding output projection matrix on the right by Q yields mathematically equivalent outputs. This invariance holds across RMSNorm operations because normalization commutes with orthogonal rotations when rescaling parameters (diag(alpha)) are fused into the subsequent weights:

    RMSNorm(X) = RMSNorm(X * Q^T) * Q

    QuaRot structures its modifications into two stages, illustrated below via mathematical pseudocode:

    # Stage 1: Transformations and Invariance (Mathematical Pseudocode)
# Applied to all input projections following RMSNorm (W_q, W_k, W_v, W_gate, W_up):
W_in      <- Q^T * diag(alpha) * W_in

# Output projections post-multiplied by Q; FFN down-proj absorbs H:
W_down    <- H * W_down * Q

# Attention value and output projections:
W_v       <- W_v * (I (x) H_d_h)
W_out     <- (H_n_h (x) I) * (I (x) H_d_h) * W_out = H * W_out

# Online post-RoPE transformations for queries and keys:
Q_rot     <- Pos(X * W_q) * (I (x) H_d_h)
K_rot     <- Pos(X * W_k) * (I (x) H_d_h)

# Stage 2: Quantization
Scale     <- max(|row|) / 7
Quantized <- round(row / Scale)

    In the first stage, RMSNorm scale parameters (diag(alpha)) are absorbed into adjacent input projection weights (including W_q, W_k, W_v, W_gate, and W_up), and randomized Hadamard rotation Q matches the hidden dimension to rotate hidden states (X <- X * Q). Because Q is merged directly into static weights offline, these inter-block rotations incur zero runtime inference overhead. Within the Feed-Forward Network (FFN), an online Hadamard transform is applied before the down-projection layer and cancelled out by fusing H into W_down. In the Multi-Head Attention module, head-wise Hadamard transforms H_d_h are applied to W_v and W_out, and an online Hadamard heads block applies (H_n_h (x) I) to attention activations Z. For keys and queries, online head-wise Hadamard rotations are applied post-RoPE, preserving attention scores while eliminating outliers from the KV Cache.

    In the second stage, weights are quantized (via GPTQ or round-to-nearest), while activations undergo online symmetric per-token quantization scaled by dividing the absolute maximum by 7. The KV Cache is quantized using asymmetric 4-bit quantization with a group size of 128.

    Empirically, on LLaMA-2 70B, QuaRot achieves full W4A4KV4 inference with a WikiText-2 perplexity loss of at most 0.47 (degrading from 3.32 to 3.79), while retaining over 99% of original zero-shot accuracy. In hardware benchmarks, it achieves prefill speedups of up to 2.16x on LLaMA-2 7B and 3.33x on LLaMA-2 70B (batch size 64, sequence length 2048), alongside up to 3.89x memory savings during decoding due to 4-bit KV caching.

    Sources: QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs

    Closing thoughts

    By leveraging computational invariance, QuaRot treats activation outliers not as an unavoidable feature of large language models, but as a coordinate-alignment artifact that can be systematically rotated away. This contrasts with mixed-precision retention strategies such as LLM.int8(), QUIK, or Atom, which keep sensitive channels in FP16 or INT8 and require specialized, non-contiguous matrix multiplication kernels, as well as SmoothQuant, which migrates difficulty through channel scaling.

    A key distinction in QuaRot is that offline rotations are fused directly into linear weights and normalization layers at zero runtime cost, while online Hadamard operations are restricted to intermediate points like the FFN down-projection and post-RoPE attention heads. Structurally neutralizing incoherence prior to uniform quantization allows standard integer GEMM kernels to execute across all weights, activations, and KV cache states with minimal accuracy degradation.

    Frequently Asked Questions

    What is QuaRot?

    QuaRot is an end-to-end Post-Training Quantization framework that enables 4-bit weight, activation, and KV Cache quantization without retaining high-precision channels. It eliminates activation outliers by applying Randomized Hadamard Rotations through computational invariance.

    Why do activation outliers cause issues in low-bit LLM quantization?

    Activation outliers are extreme values in specific channels that are orders of magnitude larger than standard features. Conventional uniform quantizers cannot allocate sufficient dynamic range for these channels without crushing the remaining non-outlier features to zero.

    How does QuaRot differ from approaches like SmoothQuant and LLM.int8()?

    SmoothQuant uses per-channel scaling to migrate quantization difficulty from activations to weights for 8-bit quantization without keeping mixed-precision channels. LLM.int8() isolates outlier channels dynamically to compute them in FP16. QuaRot takes a different approach: it applies orthogonal Hadamard rotations to disperse peak magnitudes across dimensions, allowing uniform 4-bit quantization across all matrix operations without mixed-precision channels.

    How does QuaRot quantize the KV Cache?

    QuaRot applies online head-wise Randomized Hadamard Rotations post-RoPE to keys and queries to remove outliers directly from the KV Cache. The cache is then compressed using asymmetric 4-bit quantization with a group size of 128.


    Keep reading: LLM Inference & Serving

    All guides in this series: LLM Inference & Serving Guides: vLLM, Ollama & Hugging Face

  • Troubleshooting vLLM Startup: Resolving KV Cache Capacity Exceeded Errors

    Troubleshooting vLLM Startup: Resolving KV Cache Capacity Exceeded Errors

    TL;DR

    • vLLM aborts engine initialization with a ValueError when the allocated KV cache capacity cannot accommodate the model’s maximum sequence length for at least one request.

    • The issue is commonly resolved by decreasing max_model_len, increasing gpu_memory_utilization, or applying tensor parallelism and quantization.

    • In tight-margin boundary cases, switching to the FlashInfer attention backend—either explicitly or implicitly via FP8 KV cache—has been observed to leave less available KV cache memory, triggering this error under otherwise identical settings.

    Diagnosing the KV Cache Exceeded ValueError during Engine Initialization

    During engine initialization and cache profiling, vLLM manages key-value activations in fixed-size blocks via PagedAttention. Before processing inference requests, the engine evaluates available memory and performs a sanity check: the total number of tokens that can fit into the allocated KV cache blocks must be at least as large as the configured maximum sequence length. If the KV cache capacity is smaller than that limit, the engine aborts initialization with a traceback:

    ValueError: The model's max seq len (<X>) is larger than the maximum number of tokens that can be stored in KV cache (<Y>). Try increasing gpu_memory_utilization or decreasing max_model_len when initializing the engine.

    This error prevents runtime Out-of-Memory (OOM) faults by ensuring the engine has enough memory to process at least a single full-context request. The available space for cache blocks is roughly calculated as:

    KV Cache Memory ≈ (Total GPU Memory * gpu_memory_utilization) - (Weights + Activations + CUDA Graphs)

    Two primary factors trigger this error during startup:

    • Unset or high default context window: If max_model_len (CLI flag --max-model-len) is omitted, vLLM falls back to the native context limit in the model’s Hugging Face config.json (such as max_position_embeddings, which can reach 32,768, 65,536, or 131,072+ tokens). On GPU memory-constrained hardware, the engine cannot allocate enough blocks to cover such large sequences.
    • Model weight and runtime overhead: Serving models in unquantized FP16 or BF16 formats or enabling CUDA graph capture consumes the majority of the GPU memory budget, leaving insufficient headroom for the remaining KV cache pool.

    To resolve the error during engine initialization, adjust the memory configuration via the following flags:

    vllm serve <model_name> --max-model-len 4096 --gpu-memory-utilization 0.95

    Setting --max-model-len to an integer lower than the reported capacity directly caps the required sequence ceiling. Raising gpu_memory_utilization (CLI flag --gpu-memory-utilization) above its default of 0.9 (for example, to 0.95) expands the engine’s reserved GPU memory quota, provided sufficient headroom remains to avoid runtime OOM spikes.

    Additional memory can be freed by distributing weights and attention heads across multiple GPUs using tensor parallelism via --tensor-parallel-size, applying model weight quantization (such as FP8, AWQ, or GPTQ), or specifying --kv-cache-dtype as fp8 (or fp8_e4m3 / fp8_e5m2 on supported architectures) to cut the per-token KV footprint by approximately 50% compared to 16-bit floats. However, operators should be aware that enabling FP8 KV cache can implicitly select the FlashInfer backend on certain GPUs, which in tight memory margin scenarios may reduce the available KV cache pool and trigger this initialization check.

    Sources: [Usage]: What's the relationship between KV cache and MAX_SEQUENCE_LENGTH. · Issue #10517 · vllm-project/vllm , worker – vLLM, docs.vllm.ai/en/latest/design/hybrid_kv_cache_manager, vLLM KV Cache OOM: The Flags That Actually Fix It, Conserving Memory – vLLM

    Resolving Hidden Memory Overhead from FLASHINFER and FP8 KV Cache Configurations

    Beyond standard sequence length and utilization tuning, a ValueError during KV cache initialization can also arise in tight-margin boundary conditions when switching the attention backend via --attention-backend, even when GPU memory, --max-model-len (max_model_len), and --gpu-memory-utilization (gpu_memory_utilization) settings remain identical. In an issue reported on the vLLM repository, serving RedHatAI/Qwen3.8-27B-INT4 (a hybrid Mamba/GDN architecture) on two 16 GB RTX 5060 Ti GPUs (PyTorch 2.13.0+cu130, flashinfer-python 0.6.18, vLLM 0.28.0 and 0.28.1rc1.dev446+g798544433) succeeded under the default FlashAttention (FLASH_ATTN) backend, creating a pool of 105,594 tokens on the development build (108,589 tokens on 0.28.0).

    However, explicitly passing --attention-backend FLASHINFER caused engine initialization to fail. Note that the reproduction command included specialized flags tailored for this hybrid model setup, such as --mamba-cache-mode align, rather than standard generic parameters:

    vllm serve RedHatAI/Qwen3.8-27B-INT4 --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90 --max-model-len 100352 --max-num-seqs 32 \
  --mamba-cache-mode align --max-num-batched-tokens 3135 \
  --attention-backend FLASHINFER

    This configuration failed during the memory check with the following error:

    ValueError: To serve at least one request with the model's max seq len (100352),
3.21 GiB KV cache is needed, which is larger than the available KV cache memory
(3.06 GiB). Try increasing `gpu_memory_utilization` ... or decreasing `max_model_len` ...

    On vLLM 0.28.0, the same message reported 3.15 GiB available memory instead of 3.06 GiB. Users can also encounter this issue without explicitly setting --attention-backend FLASHINFER, as setting --kv-cache-dtype fp8 implicitly selects FlashInfer on some GPUs. While the exact root cause was not definitively settled in the issue report, the reporter suggested that the FlashInfer workspace memory might not be reserved prior to calculating the KV cache pool size. Because the original error message only advised adjusting gpu_memory_utilization or max_model_len, it never informed users that the attention backend was the variable that changed.

    To address this ambiguity, a pull request has been proposed (still open) to identify the active attention backend and include that information directly within the existing ValueError.

    Sources: [Bug]: –attention-backend FLASHINFER fails the startup KV-cache check at a max-model-len that FLASH_ATTN serves; the message never names the backend, kv cache Value Erorr msg improvement Attention Backend

    Closing thoughts

    Ultimately, while trimming --max-model-len or nudging gpu_memory_utilization upward provides a quick mechanical fix, this error illustrates that KV cache headroom is governed by far more than just model weights and user-defined sequence caps. As the observed FlashInfer boundary case demonstrates, backend selection—whether configured explicitly or selected implicitly through options like FP8 KV caching—can alter the memory available for cache blocks and break razor-thin initialization budgets. Rather than reflexively sacrificing context length when this check fails, operators on memory-constrained GPUs should consider backend selection as an active variable in their memory budget. In this context, the proposed pull request to expose the active attention backend directly within the ValueError traceback represents a welcome diagnostic improvement, clarifying what variable actually impacted the available memory pool.

    Frequently Asked Questions

    Why does vLLM throw a ValueError regarding KV cache capacity during engine initialization?

    The error occurs during memory profiling if the total number of tokens that can fit into the allocated KV cache blocks is smaller than the model’s configured maximum sequence length. This sanity check aborts startup to prevent runtime Out-of-Memory faults, ensuring the engine has enough memory to handle at least a single full-context request.

    What are the primary ways to fix the KV cache ValueError?

    You can resolve the error by decreasing the maximum sequence ceiling with --max-model-len or increasing --gpu-memory-utilization (for example, to 0.95) to allocate more GPU memory. Additional remedies include enabling tensor parallelism via --tensor-parallel-size, applying model weight quantization, or setting --kv-cache-dtype to fp8 to reduce the per-token KV footprint (bearing in mind that FP8 KV cache may implicitly select FlashInfer).

    How can changing the attention backend trigger this KV cache error?

    In narrow memory margin setups, switching the attention backend to FLASHINFER has been observed to leave less available KV cache memory compared to FLASH_ATTN (suspected to stem from how backend workspace memory is reserved relative to KV cache pool sizing). This difference can cause initialization to fail even when GPU memory, --max-model-len, and --gpu-memory-utilization remain completely unchanged.

    Can the FLASHINFER backend be activated without passing --attention-backend FLASHINFER?

    Yes, setting --kv-cache-dtype fp8 implicitly selects the FlashInfer backend on some GPUs, which can alter available KV cache headroom in tight boundary cases and trigger the initialization error.


    Keep reading: LLM Inference & Serving

    All guides in this series: LLM Inference & Serving Guides: vLLM, Ollama & Hugging Face

  • Multi-Head Latent Attention: Scaling LLM Serving Throughput via Low-Rank KV Compression

    Multi-Head Latent Attention: Scaling LLM Serving Throughput via Low-Rank KV Compression

    TL;DR

    • Multi-Head Latent Attention (MLA) overcomes serving memory bottlenecks by compressing Key-Value states into compact latent vectors via low-rank projections rather than relying on coarse head-sharing.

    • During inference, MLA absorbs key and value up-projection matrices into query and output projections; it pairs this with Decoupled RoPE so that position embeddings do not interfere with matrix absorption.

    • In DeepSeek-V2, MLA enables a 93.3% reduction in KV cache footprint compared to DeepSeek 67B. Together with sparse MoE computation (activating 21B out of 236B parameters), this contributes to a 5.76-fold boost in maximum generation throughput, while separate architectural ablations show MLA outperforms standard Multi-Head Attention.

    The Memory Wall: Why the KV Cache Constrains LLM Serving

    During Autoregressive Generation in Large Language Models (LLMs), Inference Throughput is fundamentally bottlenecked by the Key-Value (KV) Cache. In standard Multi-Head Attention (MHA), the model must retain the Key and Value tensors of all previously generated tokens across every layer. These cached states reside in High-Bandwidth Memory (HBM) and must be repeatedly transferred into on-chip memory at every decoding step to compute attention for each new token. This persistent, high-volume memory traffic creates severe HBM bandwidth contention, capping decoding throughput and strictly limiting both the serving batch size and the practical context window length.

    Prior attempts to alleviate this memory wall rely primarily on Multi-Query Attention (MQA) and Grouped-Query Attention (GQA). Both techniques reduce memory requirements by forcing multiple query heads to share a reduced number of KV heads. While this head-sharing structure lowers the absolute volume of data stored in HBM and transferred to on-chip memory, it introduces an architectural trade-off: sharing KV projections across disparate query heads compromises per-head representational diversity and impairs retrieval expressiveness.

    Consequently, LLM serving faces a structural tension between memory bandwidth efficiency and model capacity. Standard MHA preserves distinct per-head projections at the cost of high HBM bandwidth consumption, whereas MQA and GQA reduce cache overhead at the expense of representational richness. Overcoming this bottleneck requires moving beyond coarse head-sharing, addressing the memory wall via Low-Rank Joint Compression of the stored representations into a Compressed KV Latent Vector rather than sacrificing individual per-head projection capacity.

    Sources: [2405.04434] DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model, arxiv.org/pdf/2405.04434, vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQF73W_INb…, Not Acceptable!, Paper page – TransMLA: Multi-head Latent Attention Is All You Need

    Empirical Gains: KV Footprint Reduction and Serving Speedups in Practice

    The practical efficacy of this low-rank compression strategy is clearly demonstrated in modern architectures. Compared to the dense DeepSeek 67B baseline, DeepSeek-V2 reduces the Key-Value (KV) Cache footprint by 93.3% and boosts maximum generation throughput up to 5.76 times. This throughput gain is a system-wide result: the KV cache reduction stems directly from MLA’s low-rank compression, while overall serving efficiency is further multiplied by the DeepSeekMoE architecture, which activates only 21B parameters out of 236B total parameters per token.

    Standard Multi-Head Attention (MHA) caches all uncompressed keys and values across every attention head, requiring 2 * n_h * d_h * l elements per token, where n_h represents the head count, d_h is the per-head dimension, and l denotes the layer count. Alternative mechanisms like Grouped-Query Attention (GQA) and Multi-Query Attention (MQA) scale memory down to 2 * n_g * d_h * l elements (for n_g groups) and 2 * d_h * l elements, respectively, but sacrifice modeling capability.

    In contrast, MLA projects keys and values into a single low-rank latent vector c_t^KV of dimension d_c. However, standard Rotary Position Embedding (RoPE) is position-sensitive and couples directly with keys and queries. If RoPE were applied directly to the content keys, the key up-projection matrix W^UK could not be absorbed into the query projection during inference because a token-dependent rotation matrix would sit between them, destroying associativity and forcing the system to recompute prefix keys. MLA resolves this by introducing Decoupled RoPE: the content keys k_t^C remain unrotated (enabling matrix absorption), while a separate, low-dimensional shared key k_t^R of dimension d_h^R carries the positional embeddings.

    During inference, MLA caches only the compressed latent vector c_t^KV alongside the decoupled key k_t^R, requiring (d_c + d_h^R) * l elements per token. In the DeepSeek-V2 configuration (n_h = 128, d_h = 128, d_c = 512 = 4 * d_h, and d_h^R = 64 = d_h / 2), the cache requirement is (d_c + d_h^R) * l = 4.5 * d_h * l elements. In terms of per-token cache storage footprint, this is equivalent to GQA configured with only 2.25 groups. In architectural evaluations, MLA nevertheless achieves stronger performance than full MHA.

    Furthermore, during inference, the key up-projection matrix W^UK is absorbed directly into the query projection W^Q, and the value up-projection matrix W^UV is absorbed into the output projection W^O. As a result, the full per-head keys and values never need to be materialized in memory for attention computation, greatly reducing HBM bandwidth traffic.

    Sources: DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

    Closing thoughts

    These architectural efficiencies highlight a practical approach to addressing memory bottlenecks in LLM serving. Rather than relying entirely on coarse head-sharing mechanisms like GQA or MQA that compromise representational diversity, Multi-Head Latent Attention demonstrates that memory bandwidth contention can be significantly mitigated through low-rank joint compression.

    A key design element is decoupling positional embeddings from content projections. Because standard RoPE operations are position-dependent and prevent key up-projection matrices from being absorbed into queries, Decoupled RoPE isolates the positional stream. This allows the content stream to remain purely linear, enabling inference engines to absorb W^UK into W^Q and W^UV into W^O without materializing uncompressed per-head Key-Value states in memory.

    MLA does not eliminate all architectural trade-offs: practitioners must still select appropriate compression dimensions d_c, allocate cache for decoupled positional keys, and navigate specialized CUDA kernel implementations. Nevertheless, its empirical impact in DeepSeek-V2—where MLA’s 93.3% KV cache reduction relative to DeepSeek 67B combines with sparse MoE execution to deliver up to a 5.76-fold throughput improvement, alongside ablation results outperforming standard MHA—makes latent compression a compelling pattern for high-throughput serving systems.

    Frequently Asked Questions

    Why does the standard Key-Value (KV) cache create a bottleneck in LLM serving?

    In standard Multi-Head Attention, retaining uncompressed Key and Value tensors across all heads and layers requires repeatedly transferring massive volumes of data from High-Bandwidth Memory to on-chip memory at each decoding step. This creates severe memory bandwidth contention that restricts decoding throughput, serving batch sizes, and context window lengths.

    How does Multi-Head Latent Attention (MLA) differ from MQA and GQA?

    MQA and GQA reduce memory by forcing multiple query heads to share a reduced count of KV heads, which compromises representational diversity and retrieval expressiveness. In contrast, MLA compresses stored representations into a compact latent vector via low-rank joint compression without relying on coarse head-sharing.

    What does MLA store in the KV cache per token?

    MLA caches only the Compressed KV Latent Vector alongside a decoupled key carrying Decoupled Rotary Position Embedding (Decoupled RoPE). This requires (d_c + d_h^R) * l elements per token rather than materializing full uncompressed keys and values across every head.

    How does MLA optimize computation during inference?

    During inference, the key up-projection matrix is absorbed directly into the query projection, and the value up-projection matrix is absorbed into the output projection. Because Decoupled RoPE keeps the content stream position-free, this linear absorption avoids having to compute or materialize full per-head Key and Value tensors in memory during attention.

    What empirical throughput and memory gains were achieved by DeepSeek-V2 using MLA?

    Compared to DeepSeek 67B, DeepSeek-V2 reduced the KV cache footprint by 93.3% and increased maximum generation throughput up to 5.76 times. The KV cache reduction is driven by MLA, while the throughput gains reflect both the compressed cache and the sparse DeepSeekMoE architecture activating 21B out of 236B parameters. In terms of cache memory footprint per token, MLA is equivalent to GQA with just 2.25 groups, while separate paper ablations show it delivers stronger modeling performance than full Multi-Head Attention.


    Keep reading: LLM Inference & Serving

    All guides in this series: LLM Inference & Serving Guides: vLLM, Ollama & Hugging Face

  • Troubleshooting ‘ValueError: Some modules are dispatched on the CPU or the disk’ in Transformers

    Troubleshooting ‘ValueError: Some modules are dispatched on the CPU or the disk’ in Transformers

    TL;DR

    • The dispatch error acts as a safety check when quantized models cannot fit entirely on GPUs because low-precision kernels cannot natively execute on the CPU or disk.
    • Past memory calculation bugs in Accelerate integrations—such as under-sized leaf module buffers and assuming uniform layer sizes in no-split modules—previously caused CPU offloading even on systems with ample GPU VRAM.
    • These issues (tracked in issue #46823) were resolved in Transformers through PR #47203 and PR #47211, superseding earlier proposals such as PR #47061.

    Symptom and Trigger: When the Quantization Dispatch Error Occurs

    In Hugging Face Transformers (working alongside Accelerate and bitsandbytes), the dispatch error occurs during model initialization via from_pretrained() when attempting to load a quantized model—such as with load_in_8bit or load_in_4bit—under automatic or custom device placement using device_map (such as device_map="auto").

    When this condition is met, Hugging Face Transformers raises the following exception:

    ValueError: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model.

    This error serves as a safety check triggered by the following sequence of constraints and runtime checks:

    • Kernel Device Constraints: The underlying quantization engine (bitsandbytes) executes its low-precision matrix operations using specialized GPU kernels.
    • VRAM Exhaustion and Placement Calculation: When the dispatch planner calculates module placement across available hardware, it may determine that available GPU VRAM—after accounting for CUDA kernel context overhead—is insufficient to host every quantized layer.
    • Automatic Fallback: Upon exhausting GPU VRAM, the dispatch planner attempts CPU offloading ("cpu") or disk offloading ("disk") for the overflow layers.
    • Execution Safety Check: Because quantized modules cannot execute their low-precision kernels natively on CPU or disk without explicit offload configurations, Hugging Face Transformers halts execution with ValueError: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model to prevent immediate failure during forward passes or inference.

    The error directly indicates that the planned quantization configuration cannot fit entirely within the targeted GPU VRAM under the current device_map strategy.

    Sources: huggingface error2 #error – Qiita, PR #31098 Enable BNB multi-backend support – SemanticDiff, Bitsandbytes · Hugging Face, Bitsandbytes · Hugging Face, Loading big models into memory · Hugging Face

    Accelerate Memory Calculation Bugs: Why Ample VRAM Models Still Spill to CPU

    While this dispatch error is designed to catch genuine out-of-memory conditions, models loaded with load_in_4bit and device_map="auto" historically dispatched modules to CPU despite having an explicit and ample max_memory configuration. Under these circumstances, ValueError: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model was raised from quantizer_bnb_4bit.validate_environment (via _get_device_map in integrations/accelerate.py).

    This behavior, tracked in issue #46823, stemmed from how device memory caps and layer sizes were calculated in integrations/accelerate.py:

    • Leaf module buffer under-sizing: get_balanced_memory calculates device caps with a buffer that previously only measured _no_split_modules. However, childless leaf modules (such as embed_tokens and lm_head, which are not reduced by quantization) act as un-splittable atoms. When a leaf module was larger than the balanced cap, device placement skipped earlier GPUs and collapsed onto the remaining capacity, spilling excess modules to CPU offloading even when total GPU VRAM was six times the model size.
    • Variable layer sizes in no-split modules: The block size for _no_split_modules was historically computed by sampling only the first matching layer. This caused under-packing when layers of the same class varied significantly in size. For instance, loading MiniMaxAI/MiniMax-M3 (a 428B sparse MoE VL model) with load_in_4bit, device_map="auto", and max_memory={i: "165GiB" for i in range(8)} on an 8x B200 setup triggered the dispatch ValueError because auto-placement under-packed the variable layers.

    While an initial proposal (PR #47061) attempted to size buffers via get_max_layer_size, maintainers determined it was not the correct fix and closed it in favor of targeted solutions that have since been merged into transformers main:

    • PR #47203 (merged July 9, 2026) addressed the variable layer sizing bug in _no_split_modules.
    • PR #47211 (merged September 18, 2026) corrected the leaf-module buffer calculation by adopting an updated sizing formula: buffer = int(1.25 * max(buffer, mean_leaves, max(leave_modules_sizes.values(), default=0))).

    Sources: Fix device_map="auto" under-packing that spills quantized models to cpu (#46823), 4-bit load_in_4bit + device_map='auto' + explicit max_memory still dispatches modules to CPU on a model that fits -> ValueError

    Closing thoughts

    While Transformers frames this error as a straightforward physical memory shortage, the underlying cause in certain multi-GPU quantized workflows was rooted in placement calculations. The handling of variable-sized MoE layers and unquantized leaf modules previously caused device_map="auto" to bypass available GPUs and spill modules to CPU. These issues have since been officially resolved in Transformers through PR #47203 and PR #47211. Developers working on older versions who encounter this error on ample hardware should update their transformers installation or inspect per-module allocations and explicit device maps.

    Frequently Asked Questions

    What causes the ‘ValueError: Some modules are dispatched on the CPU or the disk’ error in Hugging Face Transformers?

    This error occurs during model initialization with quantization settings like load_in_8bit or load_in_4bit when available GPU VRAM is insufficient to hold all layers. Because quantized modules require specialized GPU kernels and cannot execute natively on CPU or disk, Transformers halts execution as a safety check when modules are offloaded.

    Why did this dispatch error occur on systems with ample GPU VRAM?

    Historically, memory calculation bugs in Accelerate integrations caused auto-placement to under-pack GPUs. If variable-sized layers or un-splittable leaf modules exceeded the calculated balanced memory buffer, earlier GPUs were skipped, causing remaining layers to collapse onto the last GPU or spill to CPU.

    Which types of modules commonly triggered unintended CPU offloading during quantization?

    Childless leaf modules not reduced by quantization (such as embed_tokens and lm_head) and variable-sized layers in no-split modules (such as those in sparse MoE architectures) commonly caused under-packing when buffer reservations were under-calculated.

    How were these quantization memory calculation issues fixed in Transformers?

    An early proposed PR (#47061) was superseded by two merged fixes: PR #47203 resolved the variable layer size estimation in no-split modules, and PR #47211 updated the buffer calculation in get_balanced_memory using leaf module sizes (buffer = int(1.25 * max(buffer, mean_leaves, max(leave_modules_sizes.values(), default=0)))).


    Keep reading: LLM Inference & Serving

    All guides in this series: LLM Inference & Serving Guides: vLLM, Ollama & Hugging Face

  • Troubleshooting bitsandbytes: Resolving ‘RuntimeError: CUDA Setup failed despite GPU being available’

    Troubleshooting bitsandbytes: Resolving ‘RuntimeError: CUDA Setup failed despite GPU being available’

    TL;DR

    • The CUDA Setup failed error occurs when bitsandbytes cannot dynamically locate runtime libraries, encounters ABI mismatches, or lacks sufficient GPU compute capability, all of which can be inspected using python -m bitsandbytes.
    • Linux initialization failures caused by dynamic linking issues or missing libstdc++ symbols can be resolved by configuring dynamic library paths or compiling bitsandbytes from source against the local CUDA environment using CMake.
    • Native Windows environments frequently fail due to the library searching for Linux-specific paths and shared objects; running inside WSL2 or installing compatible Windows wheels provides a practical workaround.

    Understanding Why bitsandbytes Fails to Initialize CUDA

    The RuntimeError: CUDA Setup failed despite GPU being available error occurs when the Python environment successfully detects an NVIDIA GPU through PyTorch, but the underlying bitsandbytes C++/CUDA extension fails to load, locate, or execute its compiled GPU kernels. Because bitsandbytes operates as a Python wrapper over custom CUDA and C++ kernels designed for memory-efficient deep learning operations, initialization fails if the low-level execution environment does not satisfy its dynamic linking or hardware requirements.

    CUDA initialization failures are driven by three primary root causes:

    • Dynamic CUDA Library Discovery: At runtime, bitsandbytes searches system search paths for CUDA dynamic runtime shared libraries, such as libcudart.so on Linux or nvcuda.dll on Windows. If these libraries are absent from the linker search paths or are incompatible with the active PyTorch installation, the C++ extension cannot be loaded.
    • CUDA Version and Symbol Mismatches: When the CUDA version used to compile the installed PyTorch build does not match the system CUDA Toolkit or the pre-compiled binaries distributed within the bitsandbytes package, the runtime environment cannot properly bind the required symbols.
    • Compute Capability and Architecture Requirements: bitsandbytes requires specific minimum NVIDIA compute capability levels to execute its kernel backends. Attempting to initialize the library on an unsupported architecture or on a build missing compiled kernels for that specific architecture triggers a setup failure:
      • Compute capability 6.0+ (Pascal or newer): Required to run basic 8-bit optimizers and 4-bit/8-bit quantization routines.
      • Compute capability 7.5+ (Turing, Ampere, Ada Lovelace, Hopper, or newer): Required for accelerated LLM.int8() mixed-precision matrix multiplication.

    Sources: mistralai/Mixtral-8x7B-Instruct-v0.1 · [AUTOMATED] Model Memory Requirements, CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment! I, bitsandbytes · PyPI, RuntimeError: CUDA Setup failed despite correct environment paths on Windows · Issue #976 · bitsandbytes-found, stackoverflow.com/questions/79913258/cuda-setup-error-missing-libnvji…

    Diagnosing CUDA and Runtime Mismatches Using 'python -m bitsandbytes'

    To pinpoint which of these underlying root causes is responsible for the error in a specific environment, running the diagnostic command provided in the error message outputs detailed information about the environment, detected NVIDIA GPU hardware, and library search stages:

    python -m bitsandbytes

    Executing this command triggers bitsandbytes’ internal CUDA setup routine to inspect system paths, compute capability levels, and dynamic library dependencies.

    The resulting diagnostic log reveals the exact sequence of checks bitsandbytes performs. In one developer’s environment report, the diagnostic output captured the following path inspections and library loading stages:

    CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 121
CUDA SETUP: Loading binary /home/yed_dev/.local/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda121.so...
/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /home/yed_dev/.local/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda121.so)
CUDA SETUP: Something unexpected happened. Please compile from source:
git clone [email protected]:TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=121
python setup.py install

    As demonstrated in this log, running python -m bitsandbytes allows you to inspect several diagnostic points:

    • Directory and runtime discovery: It identifies missing or non-existent directories listed in environment paths (such as /usr/local/cuda/lib missing expected shared libraries like libcudart.so, libcudart.so.11.0, or libcudart.so.12.0) and notes when fallback searches find candidates like /usr/local/cuda/lib64/libcudart.so.
    • Hardware and version detection: It prints the detected CUDA version (e.g., CUDA 121) and the highest detected NVIDIA GPU compute capability (e.g., 7.5).
    • Binary loading and ABI mismatches: It shows which specific pre-compiled binary is being loaded (such as libbitsandbytes_cuda121.so) and captures underlying system loader failures, such as a missing C++ ABI symbol in the host’s libstdc++.so (e.g., version 'CXXABI_1.3.9' not found).

    If the setup encounters an unrecoverable failure during binary loading, diagnostic tools often suggest compiling bitsandbytes directly from source for the detected CUDA version.

    Sources: help!!!RuntimeError: CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    Fixing Linux Setup Errors: Configuring CUDA Paths and Missing libstdc++ Symbols

    Once the diagnostic output reveals the exact nature of the failure, addressing it requires targeting the specific platform constraints. In Linux environments, bitsandbytes can detect the local CUDA runtime and hardware specifications successfully but still fail to initialize due to missing system library symbols. In one reported setup log, the initialization routine correctly located the CUDA runtime path at /usr/local/cuda/lib64/libcudart.so, detected CUDA version 117, and identified an NVIDIA GPU compute capability of 7.5, yet failed when attempting to load the pre-compiled binary:

    CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 117
CUDA SETUP: Loading binary /usr/local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda117.so...
/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /usr/local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda117.so)

    This log demonstrates an incompatibility where the system’s libstdc++.so lacks the required CXXABI_1.3.9 symbol needed by libbitsandbytes_cuda117.so. When this dynamic linking error occurs during execution—such as importing dependent libraries like peft that call import bitsandbytes as bnb—compiling the package directly from source against the local environment resolves the ABI mismatch.

    While legacy runtime diagnostic outputs in older package versions recommended running make and python setup.py install from the original repository, current official bitsandbytes documentation maintains source builds through bitsandbytes-foundation/bitsandbytes using CMake:

    git clone https://github.com/bitsandbytes-foundation/bitsandbytes.git
cd bitsandbytes
cmake -DCOMPUTE_BACKEND=cuda -S .
make
pip install -e .

    This compilation workflow builds the CUDA backend kernels directly with your host compiler and active CUDA SDK, ensuring complete symbol compatibility with your system’s libstdc++ runtime.

    Sources: CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment!

    Resolving Initialization Failures on Windows via Pre-compiled Wheels

    While Linux environments primarily encounter ABI and dynamic path issues, running bitsandbytes in native Windows environments presents a different set of challenges because the initialization routines often look for Linux-specific shared libraries and paths. In user reports running setups such as CUDA 11.8 with PyTorch 2.0.1 or CUDA 12.2.2, the initialization routine scans for .so binaries rather than Windows-compatible dynamic libraries, producing path warnings and failing to register the NVIDIA GPU:

    bin C:\...\bitsandbytes\libbitsandbytes_cpu.so
UserWarning: C:\... did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed?
UserWarning: WARNING: No GPU detected! Check your CUDA paths. Proceeding to load CPU-only library...
CUDA SETUP: Loading binary ...\libbitsandbytes_cpu.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA library was not detected.
CUDA SETUP: Solution 1): Your paths are probably not up-to-date. You can update them via: sudo ldconfig.

    This log excerpt illustrates bitsandbytes attempting to search Linux directories like /usr/local/cuda/lib64 and falling back to a CPU-only .so binary on Windows, ultimately raising a TypeError (argument of type 'WindowsPath' is not iterable) and presenting Linux-specific solutions like sudo ldconfig.

    In cases reported across Windows installations using tools like threestudio or stable-diffusion-webui (with dependencies like xformers 0.0.20), the setup routine fails when it cannot resolve Windows-native runtime bindings. For threestudio, project collaborators recommended running the workflow inside WSL2 to bypass native Windows library resolution failures. In native Windows environments, users typically require dedicated Windows-compatible wheel builds to properly load the CUDA dynamic libraries.

    Sources: CUDA Setup failed despite GPU being available, CUDA Setup failed despite GPU being available. Please run the following command to get more information

    Closing thoughts

    Ultimately, these platform-specific failures illustrate that PyTorch detecting a GPU is only half the battle; bitsandbytes relies on strict low-level dynamic linking that fails whenever host runtime libraries, ABI symbols, or operating system paths diverge from its pre-built binaries.

    Running the python -m bitsandbytes diagnostic tool is essential because it immediately clarifies whether the issue is an unlocated libcudart.so, an incompatible host libstdc++ version, or a Windows environment trying to load Linux-specific paths. When these mismatches occur, targeted source compilation via CMake on Linux or operating within WSL2 / utilizing Windows-compatible builds provides the most reliable way to align the library’s custom kernels with your active CUDA environment.

    Frequently Asked Questions

    Why does bitsandbytes throw a CUDA Setup failed error even when PyTorch detects a GPU?

    This error occurs because the underlying C++/CUDA extension fails to load, locate, or execute its GPU kernels. Common root causes include dynamic CUDA library discovery issues, CUDA version and C++ ABI symbol mismatches, or an unsupported GPU compute capability.

    How can I diagnose CUDA and runtime issues with bitsandbytes?

    You can run the command ‘python -m bitsandbytes’ in your terminal. This diagnostic tool inspects environment paths, checks for required shared libraries like libcudart.so, reports detected CUDA and compute capability versions, and identifies binary loading failures.

    What compute capability is required to run bitsandbytes?

    Bitsandbytes requires a minimum compute capability of 6.0+ (Pascal or newer) for basic 8-bit optimizers and 4-bit/8-bit quantization routines. For accelerated LLM.int8() mixed-precision matrix multiplication, a compute capability of 7.5+ (Turing or newer) is required.

    Why does bitsandbytes encounter initialization errors on Windows?

    On native Windows environments, bitsandbytes often defaults to searching for Linux-specific file paths and .so shared libraries rather than Windows-compatible dynamic libraries. Running inside WSL2 or installing compatible Windows wheels provides a practical workaround to avoid these path and dynamic linking issues.


    Keep reading: LLM Inference & Serving

    All guides in this series: LLM Inference & Serving Guides: vLLM, Ollama & Hugging Face

  • vLLM CUDA Out of Memory: Fixing Startup, Runtime, and Persistent OOM Loops

    vLLM CUDA Out of Memory: Fixing Startup, Runtime, and Persistent OOM Loops

    TL;DR

    • “vLLM CUDA OOM” is three different problems: a startup failure (ValueError: To serve at least one request with the model's max seq len ...), a runtime failure under load (torch.OutOfMemoryError: CUDA out of memory. Tried to allocate ...), and a persistent loop where every request 500s after one bad one.
    • Almost every memory knob in vLLM — gpu_memory_utilization, max_model_len, max_num_seqs, kv_cache_dtype, enforce_eager, tensor_parallel_size — is an engine-init argument. You cannot change it on a running server, so most real fixes require a restart.
    • The only truly live lever is client-side: send fewer concurrent requests, smaller max_tokens, shorter prompts. PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True needs one restart to take effect but then greatly reduces the fragmentation form of the loop for good. LoRA adapters can be hot-swapped via /v1/load_lora_adapter.
    • The persistent loop means the engine’s error-handling has given up. On current vLLM (V1 engine) a dead GPU subprocess raises EngineDeadError; on an old pinned vLLM (before ~v0.10, when the V0 engine still existed) it was AsyncEngineDeadError: Background loop has errored already. Neither recovers in place — you restart. The practical goal is “restart without downtime” (load balancer, rolling deploy), not “never restart”.

    Three different “vLLM CUDA OOM” problems

    The phrase “vLLM won’t stop throwing CUDA OOM” covers three failures with different causes and different fixes. Identify which one you have before changing anything.

    1. Startup: not enough memory for the KV cache

    The server refuses to start and prints a ValueError like:

    ValueError: To serve at least one request with the model's max seq len (40960), (5.62 GiB KV cache is needed, which is larger than the available KV cache memory (4.89 GiB). Based on the available memory, the estimated maximum model length is 35600. Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.

    (The mismatched parenthesis after the sequence length is in vLLM’s own message.) It means: after loading the weights and reserving overhead, the memory left for the KV cache cannot hold even one request at max_model_len tokens. This is a sizing problem, fixed with init arguments — covered below.

    2. Runtime: an allocation failure under load

    The server is up and serving, then a request or a burst of them fails. There are two distinct sub-cases, and only one is graceful:

    • KV-cache block pressure. Too many concurrent sequences need more KV cache blocks than the pool has. On the V1 engine the scheduler preempts the newest sequences and later recomputes them (the default RECOMPUTE mode), so this shows up as higher latency and lower throughput, not errors. Nothing crashes.
    • Allocator OOM outside the pool. A prefill spike, activation buffers, CUDA-graph capture, or a second process on the GPU pushes transient usage past the free margin, and PyTorch raises:
    torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU 0 has a total capacity of 23.99 GiB of which 1.10 GiB is free. ... If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.

    Recompute preemption does not protect against this one — it only rations KV cache blocks, not activation or graph memory. An allocator OOM here can take down the EngineCore subprocess.

    3. The persistent loop: every request fails after one bad one

    One large request triggers an OOM, and from then on every request — even trivial ones — returns a 500. Three sub-causes, and they need different responses:

    • A dead engine (most common). The OOM exception escaped into the engine’s core loop and killed it. On current vLLM the GPU subprocess is gone and every request raises EngineDeadError (with EngineCore ... died messages in the logs). On an old pinned vLLM (before the V0 engine was removed around v0.10) the async background task died instead, raising AsyncEngineDeadError: Background loop has errored already. Neither is recoverable in place — the maintainers’ guidance is to restart the process. No allocator setting brings a dead engine back.
    • Allocator fragmentation. The engine is alive but PyTorch is holding reserved-but-unusable memory in fragmented pools; nvidia-smi shows free memory in aggregate while no single block is large enough. This is the case expandable_segments:True addresses.
    • Under-provisioning. The spike that OOMs keeps recurring because the server is sized past its GPU. The fix is the startup sizing covered further down, not recovery.

    Sources: vLLM issue #38516 (KV cache ValueError), vLLM issue #16118 (estimate max-model-len)

    Can you fix it without restarting? Mostly no — here is what is live

    Be clear-eyed about what is actually adjustable while the server runs.

    Init-only — changing these means restarting vllm serve: gpu_memory_utilization, max_model_len, max_num_seqs, max_num_batched_tokens, kv_cache_dtype, enforce_eager, tensor_parallel_size, pipeline_parallel_size, quantization, swap_space. These are all EngineArgs, baked in when the engine starts.

    Live, right now, no restart:

    • Client-side pressure. The only real serving-side lever on a running engine is sending less: lower client concurrency, cap max_tokens per request, shorten prompts. This immediately reduces peak KV cache and activation memory and will pull a server back from the edge.
    • LoRA adapters. If the pressure comes from loading many fine-tunes, start with VLLM_ALLOW_RUNTIME_LORA_UPDATING=True and add or drop adapters through POST /v1/load_lora_adapter and POST /v1/unload_lora_adapter, with no interruption to in-flight traffic. vLLM’s docs warn against enabling this outside a trusted, isolated environment, since it can load arbitrary adapters.

    One restart, then durable:

    • PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True — an environment variable read by PyTorch’s CUDA allocator, not by vLLM, so it is picked up on the next process start. It switches the allocator to growable segments that resist fragmentation, and the PyTorch OOM message itself recommends it. After that one restart it greatly reduces the fragmentation form of the loop without needing further restarts.

    Automatic, nothing to set:

    • KV-cache preemption. The V1 scheduler’s default RECOMPUTE preemption lets the server ration KV cache blocks under load instead of erroring. It does not cover allocator OOMs from activation or graph memory. This is automatic on current vLLM, which ships only the V1 engine.

    Sources: vLLM optimization docs, vLLM conserving memory docs

    Fixing the startup KV-cache OOM

    For the startup ValueError, the goal is to make the KV cache fit. In rough order of preference:

    • Lower --max-model-len to what your workload actually needs. vLLM prints an estimate (“estimated maximum model length is 35600”) — use that as a ceiling. The startup check requires enough KV cache memory to serve one request at the full max_model_len; a context window you never use just raises that bar.
    • Raise --gpu-memory-utilization if the GPU is dedicated to this server. The default is 0.9; 0.92–0.95 is safe on a card doing nothing else. Do not go to 1.0 — PyTorch’s allocator and prefill spikes need headroom.
    • Compress the KV cache with --kv-cache-dtype fp8 (or fp8_e4m3 / fp8_e5m2). This halves KV cache bytes versus fp16 for a small, usually acceptable quality cost, and often single-handedly resolves the error.
    • Cap concurrency with --max-num-seqs (and --max-num-batched-tokens). Fewer simultaneous sequences means a smaller KV cache reservation.
    • Shard the model with --tensor-parallel-size N (splits weights across N GPUs, freeing room for KV cache on each) or --pipeline-parallel-size N (splits layers).
    • Quantize the weights (AWQ, GPTQ, FP8) so the model itself occupies less, leaving more for the cache.
    vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.92 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 64

    Sources: vLLM optimization docs, vLLM issue #16118

    Fixing runtime OOM under load

    For torch.OutOfMemoryError while serving, the KV cache pool is sized fine but peak usage overflows the margin around it.

    • Lower --gpu-memory-utilization to 0.85–0.90. Counter-intuitively, giving vLLM less of the card leaves more raw headroom for the transient allocations — prefill, activations, CUDA graphs — that live outside the reserved pool. On a shared host, 0.80.
    • Add --enforce-eager. CUDA-graph capture holds a few hundred MB of static memory and can spike while capturing. Disabling it (at a modest decode-throughput cost) reclaims that room. If you want graphs but less memory, shrink compilation_config.cudagraph_capture_sizes instead.
    • Cap --max-num-seqs / --max-num-batched-tokens. These bound how many sequences and how much prefill can be in flight at once, the main driver of runtime spikes.
    • Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True before launch so a spike that is recovered does not strand fragmented memory behind it.
    • Check for other processes on the GPU (nvidia-smi) — a second CUDA process that grabs memory after vLLM’s pre-allocation is a classic cause of “it worked yesterday” OOMs.

    Watch the distinction drawn earlier: if you are only seeing latency climb and throughput sag with no errors, that is KV-cache preemption doing its job under load, and the answer is to reduce traffic or resize the deployment rather than to keep tuning the allocator margin.

    Sources: vLLM optimization docs, vLLM OOM root-cause diagnosis

    When every request fails after one bad one

    When every request fails after one bad one, work through this in order:

    1. Check whether the engine is dead. Grep the logs for EngineDeadError and EngineCore ... died, or AsyncEngineDeadError: Background loop has errored already on an old pinned vLLM. If you see either, the engine loop is gone and no in-place setting revives it. Restart the server — and make that restart cheap: put vLLM behind a load balancer or a Kubernetes Deployment with a readiness probe so a rolling restart drains and replaces the instance without dropping traffic. “Without restarting” is the wrong goal here; “restart without downtime” is the achievable one.
    2. If the engine is alive but every request still OOMs, suspect fragmentation. Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and restart once. With expandable segments PyTorch grows and shrinks one pool instead of stranding memory in fragmented blocks, which greatly reduces this form of the loop. After that restart you should not need another for fragmentation.
    3. Make sure the V0 engine is not pinned. Current vLLM only has the V1 engine, whose GPU-subprocess isolation keeps a crash from propagating straight into the request handler. If an old deployment still sets VLLM_USE_V1=0, upgrading off V0 removes the AsyncEngineDeadError failure mode entirely.
    4. If it keeps recurring, you are under-provisioned. Apply the startup-sizing fixes from the earlier section (lower max_model_len, fp8 KV cache, lower max_num_seqs) so the spike never happens, rather than recovering from it again and again.

    Sources: vLLM GPU OOM CrashLoopBackOff runbook

    Preventing it: a starting configuration

    A starting point that avoids most OOM incidents on a single dedicated GPU:

    export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

vllm serve <model> \
  --max-model-len <your real p99 context, not the model max> \
  --max-num-seqs <your real concurrency ceiling> \
  --gpu-memory-utilization 0.90 \
  --kv-cache-dtype fp8

    The principle: size max_model_len and max_num_seqs to the workload you actually have, not the model’s maximums; keep gpu_memory_utilization around 0.90 so there is headroom outside the pool; turn on fp8 KV cache unless you have measured a quality regression; and set the allocator env var so a recovered spike leaves less fragmentation behind. Load-test at your real concurrency before production — the KV cache math depends on numbers you only know at runtime.

    Sources: vLLM optimization docs

    Closing thoughts

    The honest version of “solve the vLLM OOM loop without restarting the server” is: prevent the OOM with correct init sizing, set expandable_segments:True once so fragmentation is far less likely to make it persistent, and lean on the V1 engine’s subprocess isolation and recompute preemption so KV-cache pressure costs you latency instead of a wedged server.

    What you genuinely do live is narrow — client-side load reduction now, LoRA hot-swap, and allocator config that applies on the next start. Everything else is an engine-init argument, so the practical target is not “never restart” but “restart without downtime”: a load balancer or a rolling Kubernetes deployment turns a required restart into a non-event. Chasing in-place recovery of a dead engine — EngineDeadError, or AsyncEngineDeadError on an old pinned V0 — is effort better spent on sizing the server so it does not die.

    Frequently Asked Questions

    How do I fix “CUDA out of memory” in vLLM?

    First identify which OOM it is. A startup ValueError: To serve at least one request ... is a KV-cache sizing problem: lower --max-model-len, raise --gpu-memory-utilization, or add --kv-cache-dtype fp8. A torch.OutOfMemoryError while serving is a headroom problem: lower --gpu-memory-utilization to 0.85, add --enforce-eager, cap --max-num-seqs, and set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. All of those except the env var require restarting the server.

    What does “To serve at least one request with the model’s max seq len” mean?

    After loading weights and reserving overhead, the memory left for the KV cache cannot hold a single request at max_model_len tokens. vLLM prints an estimated maximum model length; set --max-model-len at or below it, raise --gpu-memory-utilization, or use --kv-cache-dtype fp8 to make each token’s KV entry smaller.

    Can I change gpu_memory_utilization without restarting vLLM?

    No. It is an engine-init argument, fixed when vllm serve starts. The same is true of max_model_len, max_num_seqs, kv_cache_dtype, enforce_eager, and tensor_parallel_size. Changing any of them means a restart.

    What is a good gpu_memory_utilization value?

    0.90 on a dedicated GPU, 0.92–0.95 if the card is doing nothing else, 0.80–0.85 on a shared host or if you get runtime OOMs. Never 1.0 — PyTorch’s allocator and prefill spikes need memory outside vLLM’s reserved pool. The default is 0.9.

    Does --swap-space help with OOM on the vLLM V1 engine?

    Effectively no. V1’s default preemption mode is RECOMPUTE, and V1 dropped host-memory swap for the KV cache, so --swap-space does not offload blocks to CPU the way it did on V0. Rely on recompute preemption and correct sizing instead.

    How do I stop vLLM OOM errors from persisting across every request?

    First check the logs: EngineDeadError (or AsyncEngineDeadError: Background loop has errored already on an old pinned vLLM) means the engine loop is dead and only a restart fixes it. If the engine is alive but every request still OOMs, set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and restart once — that greatly reduces the allocator-fragmentation form of the loop. Current vLLM’s recompute preemption also keeps sustained KV-cache pressure from escalating into a crash.

    Does vLLM restart itself after a CUDA OOM crash?

    No. A dead engine (EngineDeadError, or AsyncEngineDeadError on old pinned vLLM) stays dead until the process is restarted externally. Run vLLM behind a load balancer or a Kubernetes Deployment with a readiness probe so that restart drains and replaces the instance without dropping traffic.


    Keep reading: LLM Inference & Serving

    All guides in this series: LLM Inference & Serving Guides: vLLM, Ollama & Hugging Face