Fixing Silent PyTorch DataLoader Crashes in Docker Containers

Understanding the Cause and Symptoms of DataLoader Shared Memory Crashes

PyTorch supports multi-process data loading through torch.utils.data.DataLoader when the num_workers parameter is set to a value greater than zero. Under this configuration, parallel worker processes fetch and collate data simultaneously. To efficiently pass CPU tensor data between worker processes and the main process without expensive data copying, PyTorch relies on Inter-Process Communication (IPC) backed by POSIX shared memory, which is mounted at /dev/shm.

By default, Docker allocates a restricted size of only 64 MB to /dev/shm for Docker containers. When deep learning workloads involve large batch sizes, heavy tensor payloads, or multiple worker processes, the active data buffers can quickly exceed this 64 MB shared memory allocation. In tmpfs-backed shared memory, mapping operations such as mmap or ftruncate may initially succeed, but when worker processes touch and access mapped pages without sufficient backing space, the operating system raises a SIGBUS (Bus Error) signal. This terminates worker processes with error notices such as DataLoader worker (pid X) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory… or DataLoader worker exited unexpectedly. This failure is distinct from a SIGKILL signal (killed by signal: Killed), which indicates that an out-of-memory killer terminated a process because actual container RAM — a separate resource from /dev/shm — was exhausted.

Resolving Shared Memory Exhaustion

To prevent these crashes, you can configure the container’s shared memory capacity or IPC settings at launch:

1. Increase Container Shared Memory Size (Recommended)
   Explicitly set /dev/shm when starting the container using --shm-size:

docker run --shm-size=8g my_pytorch_image

Note that 8g is only an illustrative example; you should size this value according to your number of workers, batch size, and prefetch settings. Prefer this method in most production workflows because it preserves container isolation while providing dedicated shared memory.

In docker-compose.yml, specify shm_size for the corresponding service:

services:
  training:
    image: my_pytorch_image
    shm_size: '8gb'

1. Use the Host IPC Namespace
   Alternatively, share the host machine’s IPC namespace directly with the container:

docker run --ipc=host my_pytorch_image

Prefer --ipc=host in trusted environments where you want to grant the container access to host IPC resources without manually sizing /dev/shm, keeping in mind that sharing the host IPC namespace weakens container isolation.

Sources: docs.pytorch.org/docs/stable/data.html, docs.pytorch.org/docs/stable/notes/multiprocessing.html, Shared Memory in Docker with the shm-size Flag – datawookie, Docker –shm-size: Set /dev/shm and Fix Crashes, PyTorch: increasing –shm-size to allow multiprocessing data loaders · Issue #937 · aws/sagemaker-python-sdk ·

Closing thoughts

Ultimately, these crashes highlight a mismatch between PyTorch’s multi-process IPC design and Docker’s restrictive default container settings. The default 64 MB shared memory cap can be problematic when using multiple workers, as signals like SIGBUS or unexpected worker exit messages can lead developers to investigate their dataset code rather than the container configuration. By adjusting container shared memory via --shm-size or using --ipc=host in trusted setups, teams can ensure reliable multi-process data loading without running out of shared memory.

Frequently Asked Questions

Why does PyTorch DataLoader crash in default Docker containers?

PyTorch uses POSIX shared memory at /dev/shm to pass tensor data between worker processes and the main process when num_workers is greater than zero. Docker restricts /dev/shm to 64 MB by default, which is easily exhausted during heavy data loading workloads, leading to worker crashes.

What default shared memory allocation does Docker provide to containers?

By default, Docker allocates a restricted size of only 64 MB to /dev/shm.

What error signals or messages occur when PyTorch shared memory is exhausted?

Worker processes terminate with a SIGBUS (Bus Error) signal, often accompanied by messages stating that the worker was killed by a bus error and may be out of shared memory, or generic messages such as “DataLoader worker exited unexpectedly”. A SIGKILL signal is a distinct failure mode caused by out-of-memory termination due to actual RAM exhaustion, not /dev/shm size.

How can I fix /dev/shm exhaustion in Docker?

You can override the default shared memory limit by passing --shm-size to docker run (sized according to your number of workers, batch size, and prefetch), setting shm_size in docker-compose.yml, or using --ipc=host to share the host IPC namespace in trusted environments.

Why does PyTorch DataLoader rely on POSIX shared memory?

PyTorch relies on Inter-Process Communication (IPC) backed by POSIX shared memory to pass CPU tensor data between worker processes and the main process. This avoids expensive data copying when running multi-process data loading.

Keep reading: Training & Architectures

• Differential Transformer: Canceling Context Noise with Differential Attention — Learn how the Differential Transformer cancels attention noise via dual softmax subtraction to boost long-context retrieval and reduce model…
• Muon Optimizer: Scaling Matrix Orthogonalization for Efficient Deep Learning Training — Discover how the Muon optimizer leverages Newton-Schulz matrix orthogonalization and spectral norm steepest descent to accelerate deep learning…
• Fixing Quality Degradation When Merging 4-Bit QLoRA Adapters in PEFT — Prevent quality loss when merging 4-bit QLoRA adapters with merge_and_unload in PEFT. Learn the exact 16-bit reloading workflow to avoid rounding…
• Byte Latent Transformer: Replacing Tokenizers with Dynamic Entropy-Based Patching — Discover the Byte Latent Transformer (BLT), a tokenizer-free architecture that replaces subword tokenization with dynamic entropy-based byte patching.

All guides in this series: LLM Training & Architecture Guides: Fine-Tuning, RL & New Models