Troubleshooting ‘ValueError: Some modules are dispatched on the CPU or the disk’ in Transformers

Symptom and Trigger: When the Quantization Dispatch Error Occurs

In Hugging Face Transformers (working alongside Accelerate and bitsandbytes), the dispatch error occurs during model initialization via from_pretrained() when attempting to load a quantized model—such as with load_in_8bit or load_in_4bit—under automatic or custom device placement using device_map (such as device_map="auto").

When this condition is met, Hugging Face Transformers raises the following exception:

ValueError: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model.

This error serves as a safety check triggered by the following sequence of constraints and runtime checks:

• Kernel Device Constraints: The underlying quantization engine (bitsandbytes) executes its low-precision matrix operations using specialized GPU kernels.
• VRAM Exhaustion and Placement Calculation: When the dispatch planner calculates module placement across available hardware, it may determine that available GPU VRAM—after accounting for CUDA kernel context overhead—is insufficient to host every quantized layer.
• Automatic Fallback: Upon exhausting GPU VRAM, the dispatch planner attempts CPU offloading ("cpu") or disk offloading ("disk") for the overflow layers.
• Execution Safety Check: Because quantized modules cannot execute their low-precision kernels natively on CPU or disk without explicit offload configurations, Hugging Face Transformers halts execution with ValueError: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model to prevent immediate failure during forward passes or inference.

The error directly indicates that the planned quantization configuration cannot fit entirely within the targeted GPU VRAM under the current device_map strategy.

Sources: huggingface error2 #error – Qiita, PR #31098 Enable BNB multi-backend support – SemanticDiff, Bitsandbytes · Hugging Face, Bitsandbytes · Hugging Face, Loading big models into memory · Hugging Face

Accelerate Memory Calculation Bugs: Why Ample VRAM Models Still Spill to CPU

While this dispatch error is designed to catch genuine out-of-memory conditions, models loaded with load_in_4bit and device_map="auto" historically dispatched modules to CPU despite having an explicit and ample max_memory configuration. Under these circumstances, ValueError: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model was raised from quantizer_bnb_4bit.validate_environment (via _get_device_map in integrations/accelerate.py).

This behavior, tracked in issue #46823, stemmed from how device memory caps and layer sizes were calculated in integrations/accelerate.py:

• Leaf module buffer under-sizing: get_balanced_memory calculates device caps with a buffer that previously only measured _no_split_modules. However, childless leaf modules (such as embed_tokens and lm_head, which are not reduced by quantization) act as un-splittable atoms. When a leaf module was larger than the balanced cap, device placement skipped earlier GPUs and collapsed onto the remaining capacity, spilling excess modules to CPU offloading even when total GPU VRAM was six times the model size.
• Variable layer sizes in no-split modules: The block size for _no_split_modules was historically computed by sampling only the first matching layer. This caused under-packing when layers of the same class varied significantly in size. For instance, loading MiniMaxAI/MiniMax-M3 (a 428B sparse MoE VL model) with load_in_4bit, device_map="auto", and max_memory={i: "165GiB" for i in range(8)} on an 8x B200 setup triggered the dispatch ValueError because auto-placement under-packed the variable layers.

While an initial proposal (PR #47061) attempted to size buffers via get_max_layer_size, maintainers determined it was not the correct fix and closed it in favor of targeted solutions that have since been merged into transformers main:

• PR #47203 (merged July 9, 2026) addressed the variable layer sizing bug in _no_split_modules.
• PR #47211 (merged September 18, 2026) corrected the leaf-module buffer calculation by adopting an updated sizing formula: buffer = int(1.25 * max(buffer, mean_leaves, max(leave_modules_sizes.values(), default=0))).

Sources: Fix device_map="auto" under-packing that spills quantized models to cpu (#46823), 4-bit load_in_4bit + device_map='auto' + explicit max_memory still dispatches modules to CPU on a model that fits -> ValueError

Closing thoughts

While Transformers frames this error as a straightforward physical memory shortage, the underlying cause in certain multi-GPU quantized workflows was rooted in placement calculations. The handling of variable-sized MoE layers and unquantized leaf modules previously caused device_map="auto" to bypass available GPUs and spill modules to CPU. These issues have since been officially resolved in Transformers through PR #47203 and PR #47211. Developers working on older versions who encounter this error on ample hardware should update their transformers installation or inspect per-module allocations and explicit device maps.

Frequently Asked Questions

What causes the ‘ValueError: Some modules are dispatched on the CPU or the disk’ error in Hugging Face Transformers?

This error occurs during model initialization with quantization settings like load_in_8bit or load_in_4bit when available GPU VRAM is insufficient to hold all layers. Because quantized modules require specialized GPU kernels and cannot execute natively on CPU or disk, Transformers halts execution as a safety check when modules are offloaded.

Why did this dispatch error occur on systems with ample GPU VRAM?

Historically, memory calculation bugs in Accelerate integrations caused auto-placement to under-pack GPUs. If variable-sized layers or un-splittable leaf modules exceeded the calculated balanced memory buffer, earlier GPUs were skipped, causing remaining layers to collapse onto the last GPU or spill to CPU.

Which types of modules commonly triggered unintended CPU offloading during quantization?

Childless leaf modules not reduced by quantization (such as embed_tokens and lm_head) and variable-sized layers in no-split modules (such as those in sparse MoE architectures) commonly caused under-packing when buffer reservations were under-calculated.

How were these quantization memory calculation issues fixed in Transformers?

An early proposed PR (#47061) was superseded by two merged fixes: PR #47203 resolved the variable layer size estimation in no-split modules, and PR #47211 updated the buffer calculation in get_balanced_memory using leaf module sizes (buffer = int(1.25 * max(buffer, mean_leaves, max(leave_modules_sizes.values(), default=0)))).

Keep reading: LLM Inference & Serving

• QuaRot: Enabling End-to-End 4-Bit LLM Inference via Randomized Rotations — Learn how QuaRot achieves end-to-end 4-bit LLM inference across weights, activations, and KV cache by eliminating outliers with randomized Hadamard…
• Troubleshooting vLLM Startup: Resolving KV Cache Capacity Exceeded Errors — Fix vLLM ValueError: model’s max seq len is larger than KV cache capacity. Learn how adjusting max_model_len, gpu_memory_utilization, and FP8 solves…
• Multi-Head Latent Attention: Scaling LLM Serving Throughput via Low-Rank KV Compression — Discover Multi-Head Latent Attention (MLA) and how low-rank KV cache compression boosts LLM inference throughput while slashing memory footprint.
• Troubleshooting bitsandbytes: Resolving ‘RuntimeError: CUDA Setup failed despite GPU being available’ — Fix the bitsandbytes ‘RuntimeError: CUDA Setup failed despite GPU being available’ by resolving dynamic linking, CUDA ABI mismatches, and library…

All guides in this series: LLM Inference & Serving Guides: vLLM, Ollama & Hugging Face