TL;DR
- QuaRot enables full 4-bit quantization across weights, activations, and the KV cache without retaining high-precision channels.
- The framework suppresses disruptive activation outliers by using Randomized Hadamard Rotations via computational invariance to disperse peak magnitudes across dimensions.
- Offline orthogonal rotations fuse directly into weights and normalization layers with zero inference overhead, while lightweight online Hadamard transforms handle internal activations and attention states.
The Challenge of Outliers in Low-Bit LLM Quantization
While weight-only 4-bit quantization (W4A16) is widely adopted, true end-to-end low-precision Post-Training Quantization (PTQ)—quantizing weights, activations, and the KV Cache to 4 bits (W4A4 Quantization with 4-bit KV Cache)—has historically resulted in severe accuracy degradation.
The primary difficulty stems from Activation Outliers: persistent, extreme-value activations in specific channels with magnitudes orders of magnitude larger than standard features. Conventional uniform quantizers cannot allocate sufficient dynamic range to represent these channels without crushing non-outlier features to zero.
Prior Post-Training Quantization approaches handle outliers through differing mechanisms that carry distinct trade-offs:
- Outlier Channel Migration vs. Mixed-Precision Retention: SmoothQuant uses per-channel scaling factors to migrate quantization difficulty from activations to weights, targeting 8-bit quantization without keeping mixed-precision channels. In contrast, frameworks like LLM.int8() (and later works like QUIK and Atom) detect outliers dynamically or offline and retain those sensitive channels in higher precision (such as FP16).
- Kernel and Hardware Overhead: For methods retaining mixed-precision channels, specialized execution kernels are required, introducing memory layout overhead, non-contiguous indexing, and hardware underutilization during matrix multiplication.
Sources: GitHub – spcl/QuaRot: Code for Neurips24 paper: QuaRot, an end-to-end 4-bit inference of large language models, microsoft.com/en-us/research/publication/quarot-outlier-free-4-bit-in…, QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs, openreview.net/forum, Vercel Security Checkpoint
What Is QuaRot and How Randomized Rotations Eliminate Outliers
To eliminate the need for mixed-precision channels, QuaRot introduces an end-to-end Post-Training Quantization (PTQ) framework that achieves full W4A4 Quantization and 4-bit KV Cache compression. Rather than keeping outlier channels in FP16 during inference, QuaRot suppresses them across the model using Randomized Hadamard Rotations applied via Computational Invariance.
A randomized Hadamard transformation multiplies a standard Hadamard matrix H by a diagonal matrix diag(s), where s is a vector of random signs drawn from {+1, -1}. A weight matrix W is defined as mu-incoherent if:
max(W) <= mu * ||W||_F / sqrt(m * n)
When incoherence is high, large individual elements emerge as outliers relative to the average magnitude. Multiplying by orthogonal randomized Hadamard matrices spreads high-energy spikes across all dimensions, systematically lowering incoherence and shaping activation distributions into uniform forms amenable to low-bit uniform quantizers.
Under the principle of Computational Invariance, transformers remain invariant to orthogonal transformations across linear and normalization layers. Specifically, multiplying an input projection weight matrix on the left by Q^T and post-multiplying the preceding output projection matrix on the right by Q yields mathematically equivalent outputs. This invariance holds across RMSNorm operations because normalization commutes with orthogonal rotations when rescaling parameters (diag(alpha)) are fused into the subsequent weights:
RMSNorm(X) = RMSNorm(X * Q^T) * Q
QuaRot structures its modifications into two stages, illustrated below via mathematical pseudocode:
# Stage 1: Transformations and Invariance (Mathematical Pseudocode)
# Applied to all input projections following RMSNorm (W_q, W_k, W_v, W_gate, W_up):
W_in <- Q^T * diag(alpha) * W_in
# Output projections post-multiplied by Q; FFN down-proj absorbs H:
W_down <- H * W_down * Q
# Attention value and output projections:
W_v <- W_v * (I (x) H_d_h)
W_out <- (H_n_h (x) I) * (I (x) H_d_h) * W_out = H * W_out
# Online post-RoPE transformations for queries and keys:
Q_rot <- Pos(X * W_q) * (I (x) H_d_h)
K_rot <- Pos(X * W_k) * (I (x) H_d_h)
# Stage 2: Quantization
Scale <- max(|row|) / 7
Quantized <- round(row / Scale)
In the first stage, RMSNorm scale parameters (diag(alpha)) are absorbed into adjacent input projection weights (including W_q, W_k, W_v, W_gate, and W_up), and randomized Hadamard rotation Q matches the hidden dimension to rotate hidden states (X <- X * Q). Because Q is merged directly into static weights offline, these inter-block rotations incur zero runtime inference overhead. Within the Feed-Forward Network (FFN), an online Hadamard transform is applied before the down-projection layer and cancelled out by fusing H into W_down. In the Multi-Head Attention module, head-wise Hadamard transforms H_d_h are applied to W_v and W_out, and an online Hadamard heads block applies (H_n_h (x) I) to attention activations Z. For keys and queries, online head-wise Hadamard rotations are applied post-RoPE, preserving attention scores while eliminating outliers from the KV Cache.
In the second stage, weights are quantized (via GPTQ or round-to-nearest), while activations undergo online symmetric per-token quantization scaled by dividing the absolute maximum by 7. The KV Cache is quantized using asymmetric 4-bit quantization with a group size of 128.
Empirically, on LLaMA-2 70B, QuaRot achieves full W4A4KV4 inference with a WikiText-2 perplexity loss of at most 0.47 (degrading from 3.32 to 3.79), while retaining over 99% of original zero-shot accuracy. In hardware benchmarks, it achieves prefill speedups of up to 2.16x on LLaMA-2 7B and 3.33x on LLaMA-2 70B (batch size 64, sequence length 2048), alongside up to 3.89x memory savings during decoding due to 4-bit KV caching.
Sources: QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
Closing thoughts
By leveraging computational invariance, QuaRot treats activation outliers not as an unavoidable feature of large language models, but as a coordinate-alignment artifact that can be systematically rotated away. This contrasts with mixed-precision retention strategies such as LLM.int8(), QUIK, or Atom, which keep sensitive channels in FP16 or INT8 and require specialized, non-contiguous matrix multiplication kernels, as well as SmoothQuant, which migrates difficulty through channel scaling.
A key distinction in QuaRot is that offline rotations are fused directly into linear weights and normalization layers at zero runtime cost, while online Hadamard operations are restricted to intermediate points like the FFN down-projection and post-RoPE attention heads. Structurally neutralizing incoherence prior to uniform quantization allows standard integer GEMM kernels to execute across all weights, activations, and KV cache states with minimal accuracy degradation.
Frequently Asked Questions
What is QuaRot?
QuaRot is an end-to-end Post-Training Quantization framework that enables 4-bit weight, activation, and KV Cache quantization without retaining high-precision channels. It eliminates activation outliers by applying Randomized Hadamard Rotations through computational invariance.
Why do activation outliers cause issues in low-bit LLM quantization?
Activation outliers are extreme values in specific channels that are orders of magnitude larger than standard features. Conventional uniform quantizers cannot allocate sufficient dynamic range for these channels without crushing the remaining non-outlier features to zero.
How does QuaRot differ from approaches like SmoothQuant and LLM.int8()?
SmoothQuant uses per-channel scaling to migrate quantization difficulty from activations to weights for 8-bit quantization without keeping mixed-precision channels. LLM.int8() isolates outlier channels dynamically to compute them in FP16. QuaRot takes a different approach: it applies orthogonal Hadamard rotations to disperse peak magnitudes across dimensions, allowing uniform 4-bit quantization across all matrix operations without mixed-precision channels.
How does QuaRot quantize the KV Cache?
QuaRot applies online head-wise Randomized Hadamard Rotations post-RoPE to keys and queries to remove outliers directly from the KV Cache. The cache is then compressed using asymmetric 4-bit quantization with a group size of 128.
Keep reading: LLM Inference & Serving
- Troubleshooting vLLM Startup: Resolving KV Cache Capacity Exceeded Errors — Fix vLLM ValueError: model’s max seq len is larger than KV cache capacity. Learn how adjusting max_model_len, gpu_memory_utilization, and FP8 solves…
- Multi-Head Latent Attention: Scaling LLM Serving Throughput via Low-Rank KV Compression — Discover Multi-Head Latent Attention (MLA) and how low-rank KV cache compression boosts LLM inference throughput while slashing memory footprint.
- Troubleshooting ‘ValueError: Some modules are dispatched on the CPU or the disk’ in Transformers — Fix ‘ValueError: Some modules are dispatched on the CPU or the disk’ in Hugging Face Transformers. Learn why quantized models fail to fit in GPU VRAM.
- Troubleshooting bitsandbytes: Resolving ‘RuntimeError: CUDA Setup failed despite GPU being available’ — Fix the bitsandbytes ‘RuntimeError: CUDA Setup failed despite GPU being available’ by resolving dynamic linking, CUDA ABI mismatches, and library…
All guides in this series: LLM Inference & Serving Guides: vLLM, Ollama & Hugging Face
