Troubleshooting bitsandbytes: Resolving ‘RuntimeError: CUDA Setup failed despite GPU being available’

Understanding Why bitsandbytes Fails to Initialize CUDA

The RuntimeError: CUDA Setup failed despite GPU being available error occurs when the Python environment successfully detects an NVIDIA GPU through PyTorch, but the underlying bitsandbytes C++/CUDA extension fails to load, locate, or execute its compiled GPU kernels. Because bitsandbytes operates as a Python wrapper over custom CUDA and C++ kernels designed for memory-efficient deep learning operations, initialization fails if the low-level execution environment does not satisfy its dynamic linking or hardware requirements.

CUDA initialization failures are driven by three primary root causes:

• Dynamic CUDA Library Discovery: At runtime, bitsandbytes searches system search paths for CUDA dynamic runtime shared libraries, such as libcudart.so on Linux or nvcuda.dll on Windows. If these libraries are absent from the linker search paths or are incompatible with the active PyTorch installation, the C++ extension cannot be loaded.
• CUDA Version and Symbol Mismatches: When the CUDA version used to compile the installed PyTorch build does not match the system CUDA Toolkit or the pre-compiled binaries distributed within the bitsandbytes package, the runtime environment cannot properly bind the required symbols.
• Compute Capability and Architecture Requirements: bitsandbytes requires specific minimum NVIDIA compute capability levels to execute its kernel backends. Attempting to initialize the library on an unsupported architecture or on a build missing compiled kernels for that specific architecture triggers a setup failure:
  • Compute capability 6.0+ (Pascal or newer): Required to run basic 8-bit optimizers and 4-bit/8-bit quantization routines.
  • Compute capability 7.5+ (Turing, Ampere, Ada Lovelace, Hopper, or newer): Required for accelerated LLM.int8() mixed-precision matrix multiplication.

Sources: mistralai/Mixtral-8x7B-Instruct-v0.1 · [AUTOMATED] Model Memory Requirements, CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment! I, bitsandbytes · PyPI, RuntimeError: CUDA Setup failed despite correct environment paths on Windows · Issue #976 · bitsandbytes-found, stackoverflow.com/questions/79913258/cuda-setup-error-missing-libnvji…

Diagnosing CUDA and Runtime Mismatches Using 'python -m bitsandbytes'

To pinpoint which of these underlying root causes is responsible for the error in a specific environment, running the diagnostic command provided in the error message outputs detailed information about the environment, detected NVIDIA GPU hardware, and library search stages:

python -m bitsandbytes

Executing this command triggers bitsandbytes’ internal CUDA setup routine to inspect system paths, compute capability levels, and dynamic library dependencies.

The resulting diagnostic log reveals the exact sequence of checks bitsandbytes performs. In one developer’s environment report, the diagnostic output captured the following path inspections and library loading stages:

CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 121
CUDA SETUP: Loading binary /home/yed_dev/.local/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda121.so...
/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /home/yed_dev/.local/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda121.so)
CUDA SETUP: Something unexpected happened. Please compile from source:
git clone [email protected]:TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=121
python setup.py install

As demonstrated in this log, running python -m bitsandbytes allows you to inspect several diagnostic points:

• Directory and runtime discovery: It identifies missing or non-existent directories listed in environment paths (such as /usr/local/cuda/lib missing expected shared libraries like libcudart.so, libcudart.so.11.0, or libcudart.so.12.0) and notes when fallback searches find candidates like /usr/local/cuda/lib64/libcudart.so.
• Hardware and version detection: It prints the detected CUDA version (e.g., CUDA 121) and the highest detected NVIDIA GPU compute capability (e.g., 7.5).
• Binary loading and ABI mismatches: It shows which specific pre-compiled binary is being loaded (such as libbitsandbytes_cuda121.so) and captures underlying system loader failures, such as a missing C++ ABI symbol in the host’s libstdc++.so (e.g., version 'CXXABI_1.3.9' not found).

If the setup encounters an unrecoverable failure during binary loading, diagnostic tools often suggest compiling bitsandbytes directly from source for the detected CUDA version.

Sources: help!!!RuntimeError: CUDA Setup failed despite GPU being available. Please run the following command to get more information:

Fixing Linux Setup Errors: Configuring CUDA Paths and Missing libstdc++ Symbols

Once the diagnostic output reveals the exact nature of the failure, addressing it requires targeting the specific platform constraints. In Linux environments, bitsandbytes can detect the local CUDA runtime and hardware specifications successfully but still fail to initialize due to missing system library symbols. In one reported setup log, the initialization routine correctly located the CUDA runtime path at /usr/local/cuda/lib64/libcudart.so, detected CUDA version 117, and identified an NVIDIA GPU compute capability of 7.5, yet failed when attempting to load the pre-compiled binary:

CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 117
CUDA SETUP: Loading binary /usr/local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda117.so...
/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /usr/local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda117.so)

This log demonstrates an incompatibility where the system’s libstdc++.so lacks the required CXXABI_1.3.9 symbol needed by libbitsandbytes_cuda117.so. When this dynamic linking error occurs during execution—such as importing dependent libraries like peft that call import bitsandbytes as bnb—compiling the package directly from source against the local environment resolves the ABI mismatch.

While legacy runtime diagnostic outputs in older package versions recommended running make and python setup.py install from the original repository, current official bitsandbytes documentation maintains source builds through bitsandbytes-foundation/bitsandbytes using CMake:

git clone https://github.com/bitsandbytes-foundation/bitsandbytes.git
cd bitsandbytes
cmake -DCOMPUTE_BACKEND=cuda -S .
make
pip install -e .

This compilation workflow builds the CUDA backend kernels directly with your host compiler and active CUDA SDK, ensuring complete symbol compatibility with your system’s libstdc++ runtime.

Sources: CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment!

Resolving Initialization Failures on Windows via Pre-compiled Wheels

While Linux environments primarily encounter ABI and dynamic path issues, running bitsandbytes in native Windows environments presents a different set of challenges because the initialization routines often look for Linux-specific shared libraries and paths. In user reports running setups such as CUDA 11.8 with PyTorch 2.0.1 or CUDA 12.2.2, the initialization routine scans for .so binaries rather than Windows-compatible dynamic libraries, producing path warnings and failing to register the NVIDIA GPU:

bin C:\...\bitsandbytes\libbitsandbytes_cpu.so
UserWarning: C:\... did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed?
UserWarning: WARNING: No GPU detected! Check your CUDA paths. Proceeding to load CPU-only library...
CUDA SETUP: Loading binary ...\libbitsandbytes_cpu.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA library was not detected.
CUDA SETUP: Solution 1): Your paths are probably not up-to-date. You can update them via: sudo ldconfig.

This log excerpt illustrates bitsandbytes attempting to search Linux directories like /usr/local/cuda/lib64 and falling back to a CPU-only .so binary on Windows, ultimately raising a TypeError (argument of type 'WindowsPath' is not iterable) and presenting Linux-specific solutions like sudo ldconfig.

In cases reported across Windows installations using tools like threestudio or stable-diffusion-webui (with dependencies like xformers 0.0.20), the setup routine fails when it cannot resolve Windows-native runtime bindings. For threestudio, project collaborators recommended running the workflow inside WSL2 to bypass native Windows library resolution failures. In native Windows environments, users typically require dedicated Windows-compatible wheel builds to properly load the CUDA dynamic libraries.

Sources: CUDA Setup failed despite GPU being available, CUDA Setup failed despite GPU being available. Please run the following command to get more information

Closing thoughts

Ultimately, these platform-specific failures illustrate that PyTorch detecting a GPU is only half the battle; bitsandbytes relies on strict low-level dynamic linking that fails whenever host runtime libraries, ABI symbols, or operating system paths diverge from its pre-built binaries.

Running the python -m bitsandbytes diagnostic tool is essential because it immediately clarifies whether the issue is an unlocated libcudart.so, an incompatible host libstdc++ version, or a Windows environment trying to load Linux-specific paths. When these mismatches occur, targeted source compilation via CMake on Linux or operating within WSL2 / utilizing Windows-compatible builds provides the most reliable way to align the library’s custom kernels with your active CUDA environment.

Frequently Asked Questions

Why does bitsandbytes throw a CUDA Setup failed error even when PyTorch detects a GPU?

This error occurs because the underlying C++/CUDA extension fails to load, locate, or execute its GPU kernels. Common root causes include dynamic CUDA library discovery issues, CUDA version and C++ ABI symbol mismatches, or an unsupported GPU compute capability.

How can I diagnose CUDA and runtime issues with bitsandbytes?

You can run the command ‘python -m bitsandbytes’ in your terminal. This diagnostic tool inspects environment paths, checks for required shared libraries like libcudart.so, reports detected CUDA and compute capability versions, and identifies binary loading failures.

What compute capability is required to run bitsandbytes?

Bitsandbytes requires a minimum compute capability of 6.0+ (Pascal or newer) for basic 8-bit optimizers and 4-bit/8-bit quantization routines. For accelerated LLM.int8() mixed-precision matrix multiplication, a compute capability of 7.5+ (Turing or newer) is required.

Why does bitsandbytes encounter initialization errors on Windows?

On native Windows environments, bitsandbytes often defaults to searching for Linux-specific file paths and .so shared libraries rather than Windows-compatible dynamic libraries. Running inside WSL2 or installing compatible Windows wheels provides a practical workaround to avoid these path and dynamic linking issues.

Keep reading: LLM Inference & Serving

• QuaRot: Enabling End-to-End 4-Bit LLM Inference via Randomized Rotations — Learn how QuaRot achieves end-to-end 4-bit LLM inference across weights, activations, and KV cache by eliminating outliers with randomized Hadamard…
• Troubleshooting vLLM Startup: Resolving KV Cache Capacity Exceeded Errors — Fix vLLM ValueError: model’s max seq len is larger than KV cache capacity. Learn how adjusting max_model_len, gpu_memory_utilization, and FP8 solves…
• Multi-Head Latent Attention: Scaling LLM Serving Throughput via Low-Rank KV Compression — Discover Multi-Head Latent Attention (MLA) and how low-rank KV cache compression boosts LLM inference throughput while slashing memory footprint.
• Troubleshooting ‘ValueError: Some modules are dispatched on the CPU or the disk’ in Transformers — Fix ‘ValueError: Some modules are dispatched on the CPU or the disk’ in Hugging Face Transformers. Learn why quantized models fail to fit in GPU VRAM.

All guides in this series: LLM Inference & Serving Guides: vLLM, Ollama & Hugging Face