Tag: bitsandbytes

  • Troubleshooting ‘ValueError: Some modules are dispatched on the CPU or the disk’ in Transformers

    Troubleshooting ‘ValueError: Some modules are dispatched on the CPU or the disk’ in Transformers

    TL;DR

    • The dispatch error acts as a safety check when quantized models cannot fit entirely on GPUs because low-precision kernels cannot natively execute on the CPU or disk.
    • Past memory calculation bugs in Accelerate integrations—such as under-sized leaf module buffers and assuming uniform layer sizes in no-split modules—previously caused CPU offloading even on systems with ample GPU VRAM.
    • These issues (tracked in issue #46823) were resolved in Transformers through PR #47203 and PR #47211, superseding earlier proposals such as PR #47061.

    Symptom and Trigger: When the Quantization Dispatch Error Occurs

    In Hugging Face Transformers (working alongside Accelerate and bitsandbytes), the dispatch error occurs during model initialization via from_pretrained() when attempting to load a quantized model—such as with load_in_8bit or load_in_4bit—under automatic or custom device placement using device_map (such as device_map="auto").

    When this condition is met, Hugging Face Transformers raises the following exception:

    ValueError: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model.

    This error serves as a safety check triggered by the following sequence of constraints and runtime checks:

    • Kernel Device Constraints: The underlying quantization engine (bitsandbytes) executes its low-precision matrix operations using specialized GPU kernels.
    • VRAM Exhaustion and Placement Calculation: When the dispatch planner calculates module placement across available hardware, it may determine that available GPU VRAM—after accounting for CUDA kernel context overhead—is insufficient to host every quantized layer.
    • Automatic Fallback: Upon exhausting GPU VRAM, the dispatch planner attempts CPU offloading ("cpu") or disk offloading ("disk") for the overflow layers.
    • Execution Safety Check: Because quantized modules cannot execute their low-precision kernels natively on CPU or disk without explicit offload configurations, Hugging Face Transformers halts execution with ValueError: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model to prevent immediate failure during forward passes or inference.

    The error directly indicates that the planned quantization configuration cannot fit entirely within the targeted GPU VRAM under the current device_map strategy.

    Sources: huggingface error2 #error – Qiita, PR #31098 Enable BNB multi-backend support – SemanticDiff, Bitsandbytes · Hugging Face, Bitsandbytes · Hugging Face, Loading big models into memory · Hugging Face

    Accelerate Memory Calculation Bugs: Why Ample VRAM Models Still Spill to CPU

    While this dispatch error is designed to catch genuine out-of-memory conditions, models loaded with load_in_4bit and device_map="auto" historically dispatched modules to CPU despite having an explicit and ample max_memory configuration. Under these circumstances, ValueError: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model was raised from quantizer_bnb_4bit.validate_environment (via _get_device_map in integrations/accelerate.py).

    This behavior, tracked in issue #46823, stemmed from how device memory caps and layer sizes were calculated in integrations/accelerate.py:

    • Leaf module buffer under-sizing: get_balanced_memory calculates device caps with a buffer that previously only measured _no_split_modules. However, childless leaf modules (such as embed_tokens and lm_head, which are not reduced by quantization) act as un-splittable atoms. When a leaf module was larger than the balanced cap, device placement skipped earlier GPUs and collapsed onto the remaining capacity, spilling excess modules to CPU offloading even when total GPU VRAM was six times the model size.
    • Variable layer sizes in no-split modules: The block size for _no_split_modules was historically computed by sampling only the first matching layer. This caused under-packing when layers of the same class varied significantly in size. For instance, loading MiniMaxAI/MiniMax-M3 (a 428B sparse MoE VL model) with load_in_4bit, device_map="auto", and max_memory={i: "165GiB" for i in range(8)} on an 8x B200 setup triggered the dispatch ValueError because auto-placement under-packed the variable layers.

    While an initial proposal (PR #47061) attempted to size buffers via get_max_layer_size, maintainers determined it was not the correct fix and closed it in favor of targeted solutions that have since been merged into transformers main:

    • PR #47203 (merged July 9, 2026) addressed the variable layer sizing bug in _no_split_modules.
    • PR #47211 (merged September 18, 2026) corrected the leaf-module buffer calculation by adopting an updated sizing formula: buffer = int(1.25 * max(buffer, mean_leaves, max(leave_modules_sizes.values(), default=0))).

    Sources: Fix device_map="auto" under-packing that spills quantized models to cpu (#46823), 4-bit load_in_4bit + device_map='auto' + explicit max_memory still dispatches modules to CPU on a model that fits -> ValueError

    Closing thoughts

    While Transformers frames this error as a straightforward physical memory shortage, the underlying cause in certain multi-GPU quantized workflows was rooted in placement calculations. The handling of variable-sized MoE layers and unquantized leaf modules previously caused device_map="auto" to bypass available GPUs and spill modules to CPU. These issues have since been officially resolved in Transformers through PR #47203 and PR #47211. Developers working on older versions who encounter this error on ample hardware should update their transformers installation or inspect per-module allocations and explicit device maps.

    Frequently Asked Questions

    What causes the ‘ValueError: Some modules are dispatched on the CPU or the disk’ error in Hugging Face Transformers?

    This error occurs during model initialization with quantization settings like load_in_8bit or load_in_4bit when available GPU VRAM is insufficient to hold all layers. Because quantized modules require specialized GPU kernels and cannot execute natively on CPU or disk, Transformers halts execution as a safety check when modules are offloaded.

    Why did this dispatch error occur on systems with ample GPU VRAM?

    Historically, memory calculation bugs in Accelerate integrations caused auto-placement to under-pack GPUs. If variable-sized layers or un-splittable leaf modules exceeded the calculated balanced memory buffer, earlier GPUs were skipped, causing remaining layers to collapse onto the last GPU or spill to CPU.

    Which types of modules commonly triggered unintended CPU offloading during quantization?

    Childless leaf modules not reduced by quantization (such as embed_tokens and lm_head) and variable-sized layers in no-split modules (such as those in sparse MoE architectures) commonly caused under-packing when buffer reservations were under-calculated.

    How were these quantization memory calculation issues fixed in Transformers?

    An early proposed PR (#47061) was superseded by two merged fixes: PR #47203 resolved the variable layer size estimation in no-split modules, and PR #47211 updated the buffer calculation in get_balanced_memory using leaf module sizes (buffer = int(1.25 * max(buffer, mean_leaves, max(leave_modules_sizes.values(), default=0)))).


    Keep reading: LLM Inference & Serving

    All guides in this series: LLM Inference & Serving Guides: vLLM, Ollama & Hugging Face

  • Troubleshooting bitsandbytes: Resolving ‘RuntimeError: CUDA Setup failed despite GPU being available’

    Troubleshooting bitsandbytes: Resolving ‘RuntimeError: CUDA Setup failed despite GPU being available’

    TL;DR

    • The CUDA Setup failed error occurs when bitsandbytes cannot dynamically locate runtime libraries, encounters ABI mismatches, or lacks sufficient GPU compute capability, all of which can be inspected using python -m bitsandbytes.
    • Linux initialization failures caused by dynamic linking issues or missing libstdc++ symbols can be resolved by configuring dynamic library paths or compiling bitsandbytes from source against the local CUDA environment using CMake.
    • Native Windows environments frequently fail due to the library searching for Linux-specific paths and shared objects; running inside WSL2 or installing compatible Windows wheels provides a practical workaround.

    Understanding Why bitsandbytes Fails to Initialize CUDA

    The RuntimeError: CUDA Setup failed despite GPU being available error occurs when the Python environment successfully detects an NVIDIA GPU through PyTorch, but the underlying bitsandbytes C++/CUDA extension fails to load, locate, or execute its compiled GPU kernels. Because bitsandbytes operates as a Python wrapper over custom CUDA and C++ kernels designed for memory-efficient deep learning operations, initialization fails if the low-level execution environment does not satisfy its dynamic linking or hardware requirements.

    CUDA initialization failures are driven by three primary root causes:

    • Dynamic CUDA Library Discovery: At runtime, bitsandbytes searches system search paths for CUDA dynamic runtime shared libraries, such as libcudart.so on Linux or nvcuda.dll on Windows. If these libraries are absent from the linker search paths or are incompatible with the active PyTorch installation, the C++ extension cannot be loaded.
    • CUDA Version and Symbol Mismatches: When the CUDA version used to compile the installed PyTorch build does not match the system CUDA Toolkit or the pre-compiled binaries distributed within the bitsandbytes package, the runtime environment cannot properly bind the required symbols.
    • Compute Capability and Architecture Requirements: bitsandbytes requires specific minimum NVIDIA compute capability levels to execute its kernel backends. Attempting to initialize the library on an unsupported architecture or on a build missing compiled kernels for that specific architecture triggers a setup failure:
      • Compute capability 6.0+ (Pascal or newer): Required to run basic 8-bit optimizers and 4-bit/8-bit quantization routines.
      • Compute capability 7.5+ (Turing, Ampere, Ada Lovelace, Hopper, or newer): Required for accelerated LLM.int8() mixed-precision matrix multiplication.

    Sources: mistralai/Mixtral-8x7B-Instruct-v0.1 · [AUTOMATED] Model Memory Requirements, CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment! I, bitsandbytes · PyPI, RuntimeError: CUDA Setup failed despite correct environment paths on Windows · Issue #976 · bitsandbytes-found, stackoverflow.com/questions/79913258/cuda-setup-error-missing-libnvji…

    Diagnosing CUDA and Runtime Mismatches Using 'python -m bitsandbytes'

    To pinpoint which of these underlying root causes is responsible for the error in a specific environment, running the diagnostic command provided in the error message outputs detailed information about the environment, detected NVIDIA GPU hardware, and library search stages:

    python -m bitsandbytes

    Executing this command triggers bitsandbytes’ internal CUDA setup routine to inspect system paths, compute capability levels, and dynamic library dependencies.

    The resulting diagnostic log reveals the exact sequence of checks bitsandbytes performs. In one developer’s environment report, the diagnostic output captured the following path inspections and library loading stages:

    CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 121
CUDA SETUP: Loading binary /home/yed_dev/.local/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda121.so...
/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /home/yed_dev/.local/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda121.so)
CUDA SETUP: Something unexpected happened. Please compile from source:
git clone [email protected]:TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=121
python setup.py install

    As demonstrated in this log, running python -m bitsandbytes allows you to inspect several diagnostic points:

    • Directory and runtime discovery: It identifies missing or non-existent directories listed in environment paths (such as /usr/local/cuda/lib missing expected shared libraries like libcudart.so, libcudart.so.11.0, or libcudart.so.12.0) and notes when fallback searches find candidates like /usr/local/cuda/lib64/libcudart.so.
    • Hardware and version detection: It prints the detected CUDA version (e.g., CUDA 121) and the highest detected NVIDIA GPU compute capability (e.g., 7.5).
    • Binary loading and ABI mismatches: It shows which specific pre-compiled binary is being loaded (such as libbitsandbytes_cuda121.so) and captures underlying system loader failures, such as a missing C++ ABI symbol in the host’s libstdc++.so (e.g., version 'CXXABI_1.3.9' not found).

    If the setup encounters an unrecoverable failure during binary loading, diagnostic tools often suggest compiling bitsandbytes directly from source for the detected CUDA version.

    Sources: help!!!RuntimeError: CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    Fixing Linux Setup Errors: Configuring CUDA Paths and Missing libstdc++ Symbols

    Once the diagnostic output reveals the exact nature of the failure, addressing it requires targeting the specific platform constraints. In Linux environments, bitsandbytes can detect the local CUDA runtime and hardware specifications successfully but still fail to initialize due to missing system library symbols. In one reported setup log, the initialization routine correctly located the CUDA runtime path at /usr/local/cuda/lib64/libcudart.so, detected CUDA version 117, and identified an NVIDIA GPU compute capability of 7.5, yet failed when attempting to load the pre-compiled binary:

    CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 117
CUDA SETUP: Loading binary /usr/local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda117.so...
/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /usr/local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda117.so)

    This log demonstrates an incompatibility where the system’s libstdc++.so lacks the required CXXABI_1.3.9 symbol needed by libbitsandbytes_cuda117.so. When this dynamic linking error occurs during execution—such as importing dependent libraries like peft that call import bitsandbytes as bnb—compiling the package directly from source against the local environment resolves the ABI mismatch.

    While legacy runtime diagnostic outputs in older package versions recommended running make and python setup.py install from the original repository, current official bitsandbytes documentation maintains source builds through bitsandbytes-foundation/bitsandbytes using CMake:

    git clone https://github.com/bitsandbytes-foundation/bitsandbytes.git
cd bitsandbytes
cmake -DCOMPUTE_BACKEND=cuda -S .
make
pip install -e .

    This compilation workflow builds the CUDA backend kernels directly with your host compiler and active CUDA SDK, ensuring complete symbol compatibility with your system’s libstdc++ runtime.

    Sources: CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment!

    Resolving Initialization Failures on Windows via Pre-compiled Wheels

    While Linux environments primarily encounter ABI and dynamic path issues, running bitsandbytes in native Windows environments presents a different set of challenges because the initialization routines often look for Linux-specific shared libraries and paths. In user reports running setups such as CUDA 11.8 with PyTorch 2.0.1 or CUDA 12.2.2, the initialization routine scans for .so binaries rather than Windows-compatible dynamic libraries, producing path warnings and failing to register the NVIDIA GPU:

    bin C:\...\bitsandbytes\libbitsandbytes_cpu.so
UserWarning: C:\... did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed?
UserWarning: WARNING: No GPU detected! Check your CUDA paths. Proceeding to load CPU-only library...
CUDA SETUP: Loading binary ...\libbitsandbytes_cpu.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA library was not detected.
CUDA SETUP: Solution 1): Your paths are probably not up-to-date. You can update them via: sudo ldconfig.

    This log excerpt illustrates bitsandbytes attempting to search Linux directories like /usr/local/cuda/lib64 and falling back to a CPU-only .so binary on Windows, ultimately raising a TypeError (argument of type 'WindowsPath' is not iterable) and presenting Linux-specific solutions like sudo ldconfig.

    In cases reported across Windows installations using tools like threestudio or stable-diffusion-webui (with dependencies like xformers 0.0.20), the setup routine fails when it cannot resolve Windows-native runtime bindings. For threestudio, project collaborators recommended running the workflow inside WSL2 to bypass native Windows library resolution failures. In native Windows environments, users typically require dedicated Windows-compatible wheel builds to properly load the CUDA dynamic libraries.

    Sources: CUDA Setup failed despite GPU being available, CUDA Setup failed despite GPU being available. Please run the following command to get more information

    Closing thoughts

    Ultimately, these platform-specific failures illustrate that PyTorch detecting a GPU is only half the battle; bitsandbytes relies on strict low-level dynamic linking that fails whenever host runtime libraries, ABI symbols, or operating system paths diverge from its pre-built binaries.

    Running the python -m bitsandbytes diagnostic tool is essential because it immediately clarifies whether the issue is an unlocated libcudart.so, an incompatible host libstdc++ version, or a Windows environment trying to load Linux-specific paths. When these mismatches occur, targeted source compilation via CMake on Linux or operating within WSL2 / utilizing Windows-compatible builds provides the most reliable way to align the library’s custom kernels with your active CUDA environment.

    Frequently Asked Questions

    Why does bitsandbytes throw a CUDA Setup failed error even when PyTorch detects a GPU?

    This error occurs because the underlying C++/CUDA extension fails to load, locate, or execute its GPU kernels. Common root causes include dynamic CUDA library discovery issues, CUDA version and C++ ABI symbol mismatches, or an unsupported GPU compute capability.

    How can I diagnose CUDA and runtime issues with bitsandbytes?

    You can run the command ‘python -m bitsandbytes’ in your terminal. This diagnostic tool inspects environment paths, checks for required shared libraries like libcudart.so, reports detected CUDA and compute capability versions, and identifies binary loading failures.

    What compute capability is required to run bitsandbytes?

    Bitsandbytes requires a minimum compute capability of 6.0+ (Pascal or newer) for basic 8-bit optimizers and 4-bit/8-bit quantization routines. For accelerated LLM.int8() mixed-precision matrix multiplication, a compute capability of 7.5+ (Turing or newer) is required.

    Why does bitsandbytes encounter initialization errors on Windows?

    On native Windows environments, bitsandbytes often defaults to searching for Linux-specific file paths and .so shared libraries rather than Windows-compatible dynamic libraries. Running inside WSL2 or installing compatible Windows wheels provides a practical workaround to avoid these path and dynamic linking issues.


    Keep reading: LLM Inference & Serving

    All guides in this series: LLM Inference & Serving Guides: vLLM, Ollama & Hugging Face

  • Fixing Quality Degradation When Merging 4-Bit QLoRA Adapters in PEFT

    Fixing Quality Degradation When Merging 4-Bit QLoRA Adapters in PEFT

    TL;DR

    • In QLoRA, the base model weights are stored in 4-bit precision (NormalFloat4 / NF4 via bitsandbytes) while the trained LoRA adapter matrices stay in 16-bit floating point (torch.float16 or torch.bfloat16).
    • Calling merge_and_unload directly on a model that is still loaded in 4-bit forces PEFT to dequantize each weight to 16-bit, add the adapter delta, and then re-quantize the result back into NF4.
    • The dequantize and add steps are effectively lossless; the final re-quantization is not. It rounds every merged weight to the nearest NF4 bucket, across every layer at once, and PEFT warns this “may get different generations due to rounding errors.”
    • The fix is to load the base model in 16-bit, run merge_and_unload there, save, and — only if deployment needs it — quantize the merged checkpoint once afterward.

    Understanding the Pitfall: Why Merging into 4-Bit Weights Hurts Output Quality

    In QLoRA configurations, the base model weights are stored in 4-bit precision formats such as NormalFloat4 (NF4) via bitsandbytes, whereas the trained LoRA adapter matrices A and B are kept in 16-bit floating-point precision (torch.float16 or torch.bfloat16).

    When you call merge_and_unload() on a model that is still loaded in 4-bit, PEFT cannot add a 16-bit delta to a 4-bit weight directly. Instead it dequantizes each base weight back to the compute dtype (bf16/fp16), adds the LoRA update ΔW = (α / r)·BA in floating point, and then re-quantizes the updated weight back into NF4. The dequantize and add steps are well defined and essentially lossless. The problem is the last step: re-quantization snaps every merged weight to the nearest value its NF4 bucket can represent, and that rounding is applied to every weight in every merged layer at once.

    So there is a single mechanism at work here — re-quantization rounding error — not a pile of unrelated failure modes. NF4 and fp16 are not “incompatible” formats; the conversion between them is well defined. What you lose is precision when a 16-bit sum is forced back into NF4’s limited set of representable values.

    PEFT itself flags this rather than forbidding it: merging a LoRA module into a 4-bit linear layer “may get different generations due to rounding errors.” In practice the merged model still runs, but its outputs can shift measurably from the same adapter running unmerged — enough to matter for evaluation or production, and impossible to recover once the merge is saved. Community threads on the PEFT tracker (issue #2321, issue #2105) are users asking why this warning appears and asking for the behavior to be documented, not reports of the model being destroyed. Treat it as a quality regression to design around, not a guaranteed failure.

    Sources: LoRA · Hugging Face, [Warning] `Merge lora module to 4-bit linear may get different generations` · Issue #2321 · huggingface/peft ·, merge_and_unload docs do not clarify behaviour for quantized base models · Issue #2105 · huggingface/peft · Gi

    The Correct Workflow: Reloading in Full Precision Before Merging

    The officially supported path is to merge the adapter into an unquantized 16-bit base model, never into the 4-bit model you trained against.

    1. Load the base model without quantization. Call transformers.AutoModelForCausalLM.from_pretrained with the compute dtype set to torch.bfloat16 or torch.float16 — use dtype= on current Transformers (torch_dtype= still works but is deprecated) — and omit quantization_config / load_in_4bit entirely for this phase. Set device_map="auto", or an explicit CPU/GPU map if memory is tight.
    2. Attach the trained adapter. peft.PeftModel.from_pretrained(base_model, adapter_path) loads the adapter weights onto the unquantized base.
    3. Call merge_and_unload(). This fuses ΔW = (α / r)·BA into the 16-bit base weights and returns a plain Transformers model with no adapter layers left. Useful arguments: safe_merge=True clones each weight matrix one layer at a time to check for NaN before committing that layer (peak overhead is one layer, not a second full model); progressbar=True shows layer-by-layer progress; adapter_names limits the merge to specific adapters.
    4. Save the merged model. model.save_pretrained(output_dir) writes a standalone 16-bit checkpoint that no longer depends on PEFT or bitsandbytes to load.
    5. Re-quantize only if deployment needs it. If you need a 4-bit or 8-bit artifact for inference, quantize this saved 16-bit checkpoint once, after the merge, with a current post-training method (bitsandbytes, torchao, AWQ, or GPTQ). That keeps quantization to a single, measurable step on the final model instead of one buried inside the merge.

    Sources: LoRA · Hugging Face, Quantization · Hugging Face

    System and Memory Requirements for 16-Bit Model Fusion

    The merge arithmetic itself is cheap. The real constraint is holding the base model in 16-bit, since the whole point of the correct workflow is that it is no longer quantized during the merge.

    At bf16/fp16 you need roughly two bytes per parameter, spread across VRAM and system RAM combined, plus headroom for the adapter and for the single weight matrix safe_merge clones per layer as it checks for NaN (one layer at a time, not a full second model). That is several times the footprint of the 4-bit model you trained with, which is the trade-off for avoiding re-quantization.

    You do not need a single GPU large enough to hold the whole model. device_map="auto" offloads layers to CPU RAM, and because the merge is a one-off operation, running it partly or entirely on CPU is fine — fitting matters far more than speed. If system RAM is also tight, pass an explicit device map that keeps most layers on CPU. Keep load_in_4bit off for this whole phase; the low-bit representation for deployment is produced later, from the saved checkpoint.

    Sources: LoRA · Hugging Face

    Comparing Quality: 4-Bit Merge, 16-Bit Merge, and Runtime QLoRA

    Three configurations are worth keeping distinct when you evaluate the result:

    • Naive 4-bit merge — adapter fused into the 4-bit model, with the re-quantization rounding baked into every layer. It runs, but generations can diverge from the trained adapter and the loss is permanent.
    • Proper 16-bit merge — adapter fused into the unquantized base, then saved. Generations track the unmerged adapter up to ordinary floating-point noise, because no re-quantization happens during the merge.
    • Runtime QLoRA inference — base kept in 4-bit, adapter applied on the fly, nothing merged. This is the natural reference point for adapter quality, but it carries adapter overhead on every forward pass and needs bitsandbytes at serving time.

    This is a conceptual comparison, not a numeric benchmark — the right numbers are the ones you measure on your own task. For deployment, do the 16-bit merge first, then quantize the merged checkpoint if you need a low-bit model for inference. Applying quantization once to a clean merged model (bitsandbytes NF4 for a calibration-free load-time path, or AWQ / GPTQ for calibration-based PTQ) gives you one rounding pass you can actually measure, rather than one hidden inside merge_and_unload. Before shipping, validate the deployed model against the runtime-QLoRA setup on your own evaluation set — a single quantization step on the final model is far easier to sign off on than rounding introduced mid-merge.

    Sources: LoRA · Hugging Face

    Closing thoughts

    Merging a QLoRA adapter straight into 4-bit base weights is not a shortcut worth taking. PEFT will do it, but it has to re-quantize every updated weight back into NF4, and that rounding pass — applied across every merged layer — is what pushes the fused model’s outputs away from what you trained.

    The reliable path is boring: load the base model in 16-bit, attach the adapter, run merge_and_unload there, save, and quantize once at the end if deployment needs it. That keeps quantization to a single measurable step on the final model instead of an invisible one inside the merge. When you do need a low-bit artifact, reach for a current post-training quantization method rather than merging inside an already-quantized model.

    Frequently Asked Questions

    What precisions are used for the base model and adapters in QLoRA?

    In QLoRA, the base model weights are stored in 4-bit precision formats like NormalFloat4 (NF4) via bitsandbytes. The trained LoRA adapter matrices are kept in 16-bit floating-point precision, such as torch.float16 or torch.bfloat16.

    Why does calling merge_and_unload directly on a 4-bit model change generation quality?

    PEFT cannot add a 16-bit adapter delta to a 4-bit weight, so it dequantizes each base weight to bf16/fp16, adds the delta, and re-quantizes the result back into NF4. That final re-quantization rounds every merged weight to the nearest NF4 bucket, across every layer at once, and PEFT warns it “may get different generations due to rounding errors.” The dequantize and add steps are lossless; only the re-quantization loses information.

    What stops the LoRA update from integrating exactly during a 4-bit merge?

    Nothing stops the addition itself — it happens in floating point and is accurate. What you lose is precision when the summed weight is forced back into NF4’s limited set of representable values. Merging into an unquantized 16-bit base skips that step entirely, so the adapter integrates without rounding loss.


    Keep reading: Training & Architectures

    All guides in this series: LLM Training & Architecture Guides: Fine-Tuning, RL & New Models