Tag: LLM Fine-Tuning

  • Fixing Quality Degradation When Merging 4-Bit QLoRA Adapters in PEFT

    Fixing Quality Degradation When Merging 4-Bit QLoRA Adapters in PEFT

    TL;DR

    • In QLoRA, the base model weights are stored in 4-bit precision (NormalFloat4 / NF4 via bitsandbytes) while the trained LoRA adapter matrices stay in 16-bit floating point (torch.float16 or torch.bfloat16).
    • Calling merge_and_unload directly on a model that is still loaded in 4-bit forces PEFT to dequantize each weight to 16-bit, add the adapter delta, and then re-quantize the result back into NF4.
    • The dequantize and add steps are effectively lossless; the final re-quantization is not. It rounds every merged weight to the nearest NF4 bucket, across every layer at once, and PEFT warns this “may get different generations due to rounding errors.”
    • The fix is to load the base model in 16-bit, run merge_and_unload there, save, and — only if deployment needs it — quantize the merged checkpoint once afterward.

    Understanding the Pitfall: Why Merging into 4-Bit Weights Hurts Output Quality

    In QLoRA configurations, the base model weights are stored in 4-bit precision formats such as NormalFloat4 (NF4) via bitsandbytes, whereas the trained LoRA adapter matrices A and B are kept in 16-bit floating-point precision (torch.float16 or torch.bfloat16).

    When you call merge_and_unload() on a model that is still loaded in 4-bit, PEFT cannot add a 16-bit delta to a 4-bit weight directly. Instead it dequantizes each base weight back to the compute dtype (bf16/fp16), adds the LoRA update ΔW = (α / r)·BA in floating point, and then re-quantizes the updated weight back into NF4. The dequantize and add steps are well defined and essentially lossless. The problem is the last step: re-quantization snaps every merged weight to the nearest value its NF4 bucket can represent, and that rounding is applied to every weight in every merged layer at once.

    So there is a single mechanism at work here — re-quantization rounding error — not a pile of unrelated failure modes. NF4 and fp16 are not “incompatible” formats; the conversion between them is well defined. What you lose is precision when a 16-bit sum is forced back into NF4’s limited set of representable values.

    PEFT itself flags this rather than forbidding it: merging a LoRA module into a 4-bit linear layer “may get different generations due to rounding errors.” In practice the merged model still runs, but its outputs can shift measurably from the same adapter running unmerged — enough to matter for evaluation or production, and impossible to recover once the merge is saved. Community threads on the PEFT tracker (issue #2321, issue #2105) are users asking why this warning appears and asking for the behavior to be documented, not reports of the model being destroyed. Treat it as a quality regression to design around, not a guaranteed failure.

    Sources: huggingface.co, github.com, github.com

    The Correct Workflow: Reloading in Full Precision Before Merging

    The officially supported path is to merge the adapter into an unquantized 16-bit base model, never into the 4-bit model you trained against.

    1. Load the base model without quantization. Call transformers.AutoModelForCausalLM.from_pretrained with the compute dtype set to torch.bfloat16 or torch.float16 — use dtype= on current Transformers (torch_dtype= still works but is deprecated) — and omit quantization_config / load_in_4bit entirely for this phase. Set device_map="auto", or an explicit CPU/GPU map if memory is tight.
    2. Attach the trained adapter. peft.PeftModel.from_pretrained(base_model, adapter_path) loads the adapter weights onto the unquantized base.
    3. Call merge_and_unload(). This fuses ΔW = (α / r)·BA into the 16-bit base weights and returns a plain Transformers model with no adapter layers left. Useful arguments: safe_merge=True clones each weight matrix one layer at a time to check for NaN before committing that layer (peak overhead is one layer, not a second full model); progressbar=True shows layer-by-layer progress; adapter_names limits the merge to specific adapters.
    4. Save the merged model. model.save_pretrained(output_dir) writes a standalone 16-bit checkpoint that no longer depends on PEFT or bitsandbytes to load.
    5. Re-quantize only if deployment needs it. If you need a 4-bit or 8-bit artifact for inference, quantize this saved 16-bit checkpoint once, after the merge, with a current post-training method (bitsandbytes, torchao, AWQ, or GPTQ). That keeps quantization to a single, measurable step on the final model instead of one buried inside the merge.

    Sources: huggingface.co, huggingface.co

    System and Memory Requirements for 16-Bit Model Fusion

    The merge arithmetic itself is cheap. The real constraint is holding the base model in 16-bit, since the whole point of the correct workflow is that it is no longer quantized during the merge.

    At bf16/fp16 you need roughly two bytes per parameter, spread across VRAM and system RAM combined, plus headroom for the adapter and for the single weight matrix safe_merge clones per layer as it checks for NaN (one layer at a time, not a full second model). That is several times the footprint of the 4-bit model you trained with, which is the trade-off for avoiding re-quantization.

    You do not need a single GPU large enough to hold the whole model. device_map="auto" offloads layers to CPU RAM, and because the merge is a one-off operation, running it partly or entirely on CPU is fine — fitting matters far more than speed. If system RAM is also tight, pass an explicit device map that keeps most layers on CPU. Keep load_in_4bit off for this whole phase; the low-bit representation for deployment is produced later, from the saved checkpoint.

    Sources: huggingface.co

    Comparing Quality: 4-Bit Merge, 16-Bit Merge, and Runtime QLoRA

    Three configurations are worth keeping distinct when you evaluate the result:

    • Naive 4-bit merge — adapter fused into the 4-bit model, with the re-quantization rounding baked into every layer. It runs, but generations can diverge from the trained adapter and the loss is permanent.
    • Proper 16-bit merge — adapter fused into the unquantized base, then saved. Generations track the unmerged adapter up to ordinary floating-point noise, because no re-quantization happens during the merge.
    • Runtime QLoRA inference — base kept in 4-bit, adapter applied on the fly, nothing merged. This is the natural reference point for adapter quality, but it carries adapter overhead on every forward pass and needs bitsandbytes at serving time.

    This is a conceptual comparison, not a numeric benchmark — the right numbers are the ones you measure on your own task. For deployment, do the 16-bit merge first, then quantize the merged checkpoint if you need a low-bit model for inference. Applying quantization once to a clean merged model (bitsandbytes NF4 for a calibration-free load-time path, or AWQ / GPTQ for calibration-based PTQ) gives you one rounding pass you can actually measure, rather than one hidden inside merge_and_unload. Before shipping, validate the deployed model against the runtime-QLoRA setup on your own evaluation set — a single quantization step on the final model is far easier to sign off on than rounding introduced mid-merge.

    Sources: huggingface.co

    Closing thoughts

    Merging a QLoRA adapter straight into 4-bit base weights is not a shortcut worth taking. PEFT will do it, but it has to re-quantize every updated weight back into NF4, and that rounding pass — applied across every merged layer — is what pushes the fused model’s outputs away from what you trained.

    The reliable path is boring: load the base model in 16-bit, attach the adapter, run merge_and_unload there, save, and quantize once at the end if deployment needs it. That keeps quantization to a single measurable step on the final model instead of an invisible one inside the merge. When you do need a low-bit artifact, reach for a current post-training quantization method rather than merging inside an already-quantized model.

    Frequently Asked Questions

    What precisions are used for the base model and adapters in QLoRA?

    In QLoRA, the base model weights are stored in 4-bit precision formats like NormalFloat4 (NF4) via bitsandbytes. The trained LoRA adapter matrices are kept in 16-bit floating-point precision, such as torch.float16 or torch.bfloat16.

    Why does calling merge_and_unload directly on a 4-bit model change generation quality?

    PEFT cannot add a 16-bit adapter delta to a 4-bit weight, so it dequantizes each base weight to bf16/fp16, adds the delta, and re-quantizes the result back into NF4. That final re-quantization rounds every merged weight to the nearest NF4 bucket, across every layer at once, and PEFT warns it “may get different generations due to rounding errors.” The dequantize and add steps are lossless; only the re-quantization loses information.

    What stops the LoRA update from integrating exactly during a 4-bit merge?

    Nothing stops the addition itself — it happens in floating point and is accurate. What you lose is precision when the summed weight is forced back into NF4’s limited set of representable values. Merging into an unquantized 16-bit base skips that step entirely, so the adapter integrates without rounding loss.

  • Group Relative Policy Optimization for Efficient Reinforcement Learning in Language Models

    Group Relative Policy Optimization for Efficient Reinforcement Learning in Language Models

    TL;DR

    • Group Relative Policy Optimization (GRPO) eliminates the Critic Model from standard PPO architectures, cutting GPU VRAM usage nearly in half during LLM reinforcement learning.
    • Standard GRPO relies on static sampling and fixed rollouts, which can waste computational resources on easy prompts while under-training difficult reasoning tasks.
    • Combining GRPO with dynamic techniques like Prompt-GDRO and Rollout-GDRO directs compute toward hard tasks, significantly improving reasoning accuracy.

    Understanding Group Relative Policy Optimization (GRPO) in LLM Training

    Reinforcement learning (RL) training methods enhance the alignment and reasoning performance of Large Language Models (LLMs), specifically by improving their capacity to understand human intents, follow user instructions, and strengthen inferential processing.

    Across the broader LLM lifecycle, RL strategies are integrated across several phases: pre-training, alignment fine-tuning, and reinforced reasoning. In particular, RL approaches deployed during the reinforced reasoning phase act as a primary driver for advancing model reasoning limits, with significant focus placed on Reinforcement Learning with Verifiable Rewards (RLVR). Fine-tuning and evaluation within these training frameworks draw upon varied data sources and benchmarks, including human-annotated datasets, AI-assisted preference data, and program-verification-style corpora.

    Sources: Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    Architectural Shift: Eliminating the Critic Model for Memory Efficiency

    To operationalize these reinforced reasoning phases effectively, attention has increasingly turned to optimizing trainer architectures. In standard Proximal Policy Optimization (PPO), reinforcement learning relies on an Actor-Critic architecture. Fine-tuning a Large Language Model (LLM) Actor Model under PPO requires training a separate Critic Model of a similar size. Running this dedicated value network nearly doubles GPU VRAM requirements and compute overhead during post-training.

    Group Relative Policy Optimization (GRPO) executes an architectural shift by completely eliminating the Critic Model. Rather than using a value network to predict output values, GRPO generates a group of outputs {o1, o2, …, oG} for each input prompt q under the current old policy pi_theta_old. Scores for each output are computed using a reward function—such as rule-based checks for answer correctness—and then standardized across the group to determine relative performance. Removing the Critic Model cuts VRAM usage nearly in half during reinforcement learning training. This direct reduction in memory overhead significantly decreases hardware costs and frees up GPU capacity, enabling models to be trained with higher batch sizes.

    Sources: arxiv.org, substack.com, arxiv.org, huggingface.co, huggingface.co

    Impact on Model Performance, Throughput, and Optimization Limitations

    Despite these architectural memory savings, standard Group Relative Policy Optimization (GRPO) suffers from a structural optimization limitation rooted in static uniformity: it relies on uniform prompt sampling and a fixed number of rollouts per prompt. For heterogeneous, heavy-tailed reasoning datasets, this static approach creates inefficiencies by wasting computational resources on already-solved patterns while under-training the long tail of difficult problems.

    To overcome these compute and optimization bottlenecks, Multi-Adversary Group Distributionally Robust Optimization adapts the training distribution dynamically using an Online Difficulty Classifier that partitions prompts into pass@k difficulty groups: Prompt-GDRO employs an EMA-debiased multiplicative-weights bandit sampler to target the intensive difficulty margin and upweight persistently hard prompt groups without introducing frequency bias, while Rollout-GDRO uses a shadow-price controller guided by a variance-proxy analysis to reallocate rollouts across difficulty groups—aiming for a square-root optimal rollout allocation—to maximize gradient variance reduction on hard tasks under a fixed mean budget.

    Because Rollout-GDRO operates under a fixed mean budget, it dynamically reallocates compute resources in a compute-neutral manner rather than increasing total throughput demands. Qualitative evaluations show that this creates an emergent curriculum, shifting optimization resources toward the evolving reasoning frontier as the model learns. When validated on the DAPO 14.1k dataset using Qwen3-Base models across 1.7B, 4B, and 8B parameter scales, these optimization adjustments yield measurable performance improvements over the standard GRPO baseline, with Prompt-GDRO achieving an average relative gain of +10.6% in pass@8 accuracy and Rollout-GDRO achieving an average relative gain of +10.1% in pass@8 accuracy.

    Sources: Group Distributionally Robust Optimization-Driven Reinforcement Learning for LLM Reasoning

    Closing thoughts

    In summary, by eliminating the memory-heavy Critic Model, Group Relative Policy Optimization offers an invaluable structural solution to the severe VRAM bottlenecks inherent in standard PPO frameworks. However, raw memory efficiency alone falls short when static sampling wastes compute on already-solved problems while under-training the long tail of complex tasks.

    In my assessment, the true breakthrough lies in pairing this critic-free architecture with dynamic techniques like Prompt-GDRO and Rollout-GDRO, which establish an emergent curriculum by targeting compute specifically at the model’s evolving reasoning frontier. Ultimately, these findings demonstrate that maximizing reinforcement learning efficacy requires both stripping away architectural redundancy and intelligently shifting optimization resources toward hard, unresolved prompts.

    Frequently Asked Questions

    What is Group Relative Policy Optimization (GRPO)?

    Group Relative Policy Optimization (GRPO) is a reinforcement learning training method for language models that eliminates the separate Critic Model used in standard PPO architectures. Instead of relying on a value network, GRPO generates a group of outputs per prompt and standardizes their reward scores to measure relative performance.

    How does GRPO reduce memory overhead during training?

    Standard PPO requires training a separate Critic Model of similar size to the Actor Model, which nearly doubles GPU VRAM requirements. By completely removing the Critic Model, GRPO cuts VRAM usage nearly in half and allows models to train with higher batch sizes.

    What limitation exists in standard GRPO?

    Standard GRPO relies on uniform prompt sampling and a fixed number of rollouts per prompt. This static approach can waste computational resources on already-solved patterns while under-training difficult, long-tail reasoning problems.

    How do Prompt-GDRO and Rollout-GDRO address the limitations of GRPO?

    Prompt-GDRO upweights persistently hard prompt groups without frequency bias, while Rollout-GDRO dynamically reallocates rollouts across difficulty groups under a fixed mean budget. Together, they shift optimization resources toward the evolving reasoning frontier, improving accuracy over standard GRPO baselines.