Tag: Hugging Face

  • Fixing Quality Degradation When Merging 4-Bit QLoRA Adapters in PEFT

    Fixing Quality Degradation When Merging 4-Bit QLoRA Adapters in PEFT

    TL;DR

    • In QLoRA, the base model weights are stored in 4-bit precision (NormalFloat4 / NF4 via bitsandbytes) while the trained LoRA adapter matrices stay in 16-bit floating point (torch.float16 or torch.bfloat16).
    • Calling merge_and_unload directly on a model that is still loaded in 4-bit forces PEFT to dequantize each weight to 16-bit, add the adapter delta, and then re-quantize the result back into NF4.
    • The dequantize and add steps are effectively lossless; the final re-quantization is not. It rounds every merged weight to the nearest NF4 bucket, across every layer at once, and PEFT warns this “may get different generations due to rounding errors.”
    • The fix is to load the base model in 16-bit, run merge_and_unload there, save, and — only if deployment needs it — quantize the merged checkpoint once afterward.

    Understanding the Pitfall: Why Merging into 4-Bit Weights Hurts Output Quality

    In QLoRA configurations, the base model weights are stored in 4-bit precision formats such as NormalFloat4 (NF4) via bitsandbytes, whereas the trained LoRA adapter matrices A and B are kept in 16-bit floating-point precision (torch.float16 or torch.bfloat16).

    When you call merge_and_unload() on a model that is still loaded in 4-bit, PEFT cannot add a 16-bit delta to a 4-bit weight directly. Instead it dequantizes each base weight back to the compute dtype (bf16/fp16), adds the LoRA update ΔW = (α / r)·BA in floating point, and then re-quantizes the updated weight back into NF4. The dequantize and add steps are well defined and essentially lossless. The problem is the last step: re-quantization snaps every merged weight to the nearest value its NF4 bucket can represent, and that rounding is applied to every weight in every merged layer at once.

    So there is a single mechanism at work here — re-quantization rounding error — not a pile of unrelated failure modes. NF4 and fp16 are not “incompatible” formats; the conversion between them is well defined. What you lose is precision when a 16-bit sum is forced back into NF4’s limited set of representable values.

    PEFT itself flags this rather than forbidding it: merging a LoRA module into a 4-bit linear layer “may get different generations due to rounding errors.” In practice the merged model still runs, but its outputs can shift measurably from the same adapter running unmerged — enough to matter for evaluation or production, and impossible to recover once the merge is saved. Community threads on the PEFT tracker (issue #2321, issue #2105) are users asking why this warning appears and asking for the behavior to be documented, not reports of the model being destroyed. Treat it as a quality regression to design around, not a guaranteed failure.

    Sources: huggingface.co, github.com, github.com

    The Correct Workflow: Reloading in Full Precision Before Merging

    The officially supported path is to merge the adapter into an unquantized 16-bit base model, never into the 4-bit model you trained against.

    1. Load the base model without quantization. Call transformers.AutoModelForCausalLM.from_pretrained with the compute dtype set to torch.bfloat16 or torch.float16 — use dtype= on current Transformers (torch_dtype= still works but is deprecated) — and omit quantization_config / load_in_4bit entirely for this phase. Set device_map="auto", or an explicit CPU/GPU map if memory is tight.
    2. Attach the trained adapter. peft.PeftModel.from_pretrained(base_model, adapter_path) loads the adapter weights onto the unquantized base.
    3. Call merge_and_unload(). This fuses ΔW = (α / r)·BA into the 16-bit base weights and returns a plain Transformers model with no adapter layers left. Useful arguments: safe_merge=True clones each weight matrix one layer at a time to check for NaN before committing that layer (peak overhead is one layer, not a second full model); progressbar=True shows layer-by-layer progress; adapter_names limits the merge to specific adapters.
    4. Save the merged model. model.save_pretrained(output_dir) writes a standalone 16-bit checkpoint that no longer depends on PEFT or bitsandbytes to load.
    5. Re-quantize only if deployment needs it. If you need a 4-bit or 8-bit artifact for inference, quantize this saved 16-bit checkpoint once, after the merge, with a current post-training method (bitsandbytes, torchao, AWQ, or GPTQ). That keeps quantization to a single, measurable step on the final model instead of one buried inside the merge.

    Sources: huggingface.co, huggingface.co

    System and Memory Requirements for 16-Bit Model Fusion

    The merge arithmetic itself is cheap. The real constraint is holding the base model in 16-bit, since the whole point of the correct workflow is that it is no longer quantized during the merge.

    At bf16/fp16 you need roughly two bytes per parameter, spread across VRAM and system RAM combined, plus headroom for the adapter and for the single weight matrix safe_merge clones per layer as it checks for NaN (one layer at a time, not a full second model). That is several times the footprint of the 4-bit model you trained with, which is the trade-off for avoiding re-quantization.

    You do not need a single GPU large enough to hold the whole model. device_map="auto" offloads layers to CPU RAM, and because the merge is a one-off operation, running it partly or entirely on CPU is fine — fitting matters far more than speed. If system RAM is also tight, pass an explicit device map that keeps most layers on CPU. Keep load_in_4bit off for this whole phase; the low-bit representation for deployment is produced later, from the saved checkpoint.

    Sources: huggingface.co

    Comparing Quality: 4-Bit Merge, 16-Bit Merge, and Runtime QLoRA

    Three configurations are worth keeping distinct when you evaluate the result:

    • Naive 4-bit merge — adapter fused into the 4-bit model, with the re-quantization rounding baked into every layer. It runs, but generations can diverge from the trained adapter and the loss is permanent.
    • Proper 16-bit merge — adapter fused into the unquantized base, then saved. Generations track the unmerged adapter up to ordinary floating-point noise, because no re-quantization happens during the merge.
    • Runtime QLoRA inference — base kept in 4-bit, adapter applied on the fly, nothing merged. This is the natural reference point for adapter quality, but it carries adapter overhead on every forward pass and needs bitsandbytes at serving time.

    This is a conceptual comparison, not a numeric benchmark — the right numbers are the ones you measure on your own task. For deployment, do the 16-bit merge first, then quantize the merged checkpoint if you need a low-bit model for inference. Applying quantization once to a clean merged model (bitsandbytes NF4 for a calibration-free load-time path, or AWQ / GPTQ for calibration-based PTQ) gives you one rounding pass you can actually measure, rather than one hidden inside merge_and_unload. Before shipping, validate the deployed model against the runtime-QLoRA setup on your own evaluation set — a single quantization step on the final model is far easier to sign off on than rounding introduced mid-merge.

    Sources: huggingface.co

    Closing thoughts

    Merging a QLoRA adapter straight into 4-bit base weights is not a shortcut worth taking. PEFT will do it, but it has to re-quantize every updated weight back into NF4, and that rounding pass — applied across every merged layer — is what pushes the fused model’s outputs away from what you trained.

    The reliable path is boring: load the base model in 16-bit, attach the adapter, run merge_and_unload there, save, and quantize once at the end if deployment needs it. That keeps quantization to a single measurable step on the final model instead of an invisible one inside the merge. When you do need a low-bit artifact, reach for a current post-training quantization method rather than merging inside an already-quantized model.

    Frequently Asked Questions

    What precisions are used for the base model and adapters in QLoRA?

    In QLoRA, the base model weights are stored in 4-bit precision formats like NormalFloat4 (NF4) via bitsandbytes. The trained LoRA adapter matrices are kept in 16-bit floating-point precision, such as torch.float16 or torch.bfloat16.

    Why does calling merge_and_unload directly on a 4-bit model change generation quality?

    PEFT cannot add a 16-bit adapter delta to a 4-bit weight, so it dequantizes each base weight to bf16/fp16, adds the delta, and re-quantizes the result back into NF4. That final re-quantization rounds every merged weight to the nearest NF4 bucket, across every layer at once, and PEFT warns it “may get different generations due to rounding errors.” The dequantize and add steps are lossless; only the re-quantization loses information.

    What stops the LoRA update from integrating exactly during a 4-bit merge?

    Nothing stops the addition itself — it happens in floating point and is accurate. What you lose is precision when the summed weight is forced back into NF4’s limited set of representable values. Merging into an unquantized 16-bit base skips that step entirely, so the adapter integrates without rounding loss.

  • Fixing Cannot Copy Out of Meta Tensor Errors in Hugging Face Transformers

    Fixing Cannot Copy Out of Meta Tensor Errors in Hugging Face Transformers

    TL;DR

    • The ‘Cannot copy out of meta tensor’ error happens when attempting manual operations like .to(device) on meta tensors, which serve as memory-free model placeholders.
    • Avoid calling .to() manually when loading models with device_map=’auto’, as Hugging Face Accelerate automatically handles layer placement and device transfers via hooks.
    • To resolve meta tensor errors in custom pipelines, use load_checkpoint_and_dispatch() when dispatching an Accelerate-managed model, or torch.nn.Module.to_empty() when manually allocating a model before loading a state dict — they solve different scenarios, not interchangeable alternatives.

    Understanding Meta Tensors in PyTorch and Hugging Face

    The error NotImplementedError: Cannot copy out of meta tensor; no data! occurs when PyTorch, Hugging Face Transformers, or Accelerate attempts to copy data, perform operations, or execute standard .to(device) calls on a tensor residing on the meta device. In PyTorch, a tensor on the meta device acts as a lightweight placeholder that stores metadata—such as shape, strides, and data type—without allocating physical memory or actual storage bytes. Both PyTorch and Hugging Face Transformers use this mechanism to construct large model architectures instantly without overwhelming CPU RAM or GPU VRAM.

    When loading large models in Hugging Face Transformers using device_map="auto" or low_cpu_mem_usage=True, Accelerate initially instantiates the model skeleton on the meta device. Setting low_cpu_mem_usage=True instructs from_pretrained to load the model via meta-device initialization, which prevents full model weights from being loaded into system memory twice. Accelerate then calculates optimal parameter placement across available hardware (splitting across GPUs, CPU RAM, or disk) before populating real weight data. Data-dependent operations that need real bytes—such as .to("cuda") or .item()—fail on a meta tensor because no physical memory has been allocated (though the exact exception differs: .to() raises this article’s Cannot copy out of meta tensor; no data! error, while .item() raises a separate Tensor.item() cannot be called on meta tensors error). Metadata-only operations like .clone(), by contrast, succeed on a meta tensor and simply return another meta tensor. If custom logic or user code tries to interact with or move these placeholder parameters before they are fully materialized into real tensors, PyTorch throws the error.

    To avoid and fix meta tensor errors, follow these standard practices and API guidelines:

    • Avoid Manual .to() Calls on device_map="auto" Models: Do not call model.to("cuda") or model.to(device) after instantiating a model with device_map="auto". Accelerate automatically manages layer movement and execution across devices using PyTorch forward hooks. Calling manual device transfer methods forces PyTorch to attempt copying empty meta tensors, triggering the exception.
    • Use Native Checkpoint Dispatching for Custom Models: When constructing custom model pipelines using init_empty_weights() to route layer creation onto the meta device, load and place real weights using load_checkpoint_and_dispatch() — this function loads a checkpoint from disk and replaces the empty meta tensors with real weight tensors before attaching execution hooks. dispatch_model() is a narrower tool: it only attaches the execution hooks that route each layer’s forward pass to its assigned device, and it expects the model to already hold real (non-meta) weights — calling it alone on a model still on the meta device will not populate those weights.
    • Use torch.nn.Module.to_empty() for Uninitialized Allocation: If manual model allocation is required before loading state dicts into memory, replace standard .to(device) calls with torch.nn.Module.to_empty(). This native PyTorch method transfers a model off the meta device onto a target device (such as CPU or CUDA) and allocates uninitialized memory without trying to read non-existent meta tensor values.
    • Verify Distributed & Offloading Settings: In distributed training or inference environments (such as DeepSpeed ZeRO-3 or FSDP), ensure training arguments do not conflict with model loading options. Passing explicit offloading configurations or disabling incompatible CPU RAM efficient loading flags ensures model weights are fully materialized before training loops begin.

    Sources: pytorch.org, huggingface.co, huggingface.co, github.com

    Closing thoughts

    Looking at the underlying cause, the “cannot copy out of meta tensor” error most often traces back to a friction point where legacy PyTorch habits—specifically manual .to(device) transfers—conflict with modern, automated memory management. That said, it is not always user error: Accelerate’s automatic dispatch logic can itself leave some submodules on the meta device for certain model architectures, and a later data-dependent operation (such as the .item() call inside model.generate()) then fails with a related meta-tensor exception — evidence that dispatch-logic bugs, not just manual .to() misuse, are a real source of these errors. Meta tensors are essential lightweight blueprints that enable the loading of massive architectures, but they require developers to trust frameworks like Hugging Face Accelerate to handle parameter placement behind the scenes. Using load_checkpoint_and_dispatch() for Accelerate-managed models, or to_empty() when manually allocating a model before loading a state dict, resolves most instantiation hurdles—though architecture-specific dispatch failures may still require filing an upstream issue. Ultimately, respecting the boundary between placeholder metadata and materialized weights ensures clean, exception-free pipelines without causing system memory exhaustion.

    Frequently Asked Questions

    What causes the “Cannot copy out of meta tensor” error in PyTorch and Hugging Face?

    The error occurs when PyTorch, Hugging Face Transformers, or Accelerate attempts to copy data, perform operations, or execute manual .to(device) calls on a tensor residing on the meta device. Because meta tensors are lightweight placeholders with no physical memory allocated, standard data-copying operations fail.

    How does device_map=”auto” utilize meta tensors when loading models?

    When using device_map="auto" or low_cpu_mem_usage=True, Accelerate initially instantiates the model skeleton on the meta device to prevent loading full model weights into system memory twice. It calculates optimal parameter placement across hardware before populating real weight data.

    Why should manual .to() calls be avoided on models instantiated with device_map=”auto”?

    Accelerate automatically manages layer movement and execution across devices using forward hooks. Calling .to("cuda") or .to(device) manually forces PyTorch to attempt copying empty meta tensors, triggering the exception.

    How can you safely allocate or dispatch custom models created on the meta device?

    For custom models, developers should load and place real weights using load_checkpoint_and_dispatch(), which reads a checkpoint from disk and replaces the empty meta tensors with real weight tensors. dispatch_model() is not a substitute for this — it only attaches execution hooks to a model whose weights are already real, so calling it alone on a model still on the meta device will not load any weights. If manual allocation is required before loading state dicts, torch.nn.Module.to_empty() should be used instead of standard .to(device) calls.