Tag: Transformers

  • Titans: Learning to Memorize at Test Time with Neural Long-Term Memory

    Titans: Learning to Memorize at Test Time with Neural Long-Term Memory

    TL;DR

    • Titans introduces a dual-memory architecture combining attention for short-term dependencies with a Neural Long-Term Memory module for persistent historical context.
    • The framework resolves trade-offs between quadratic attention and lossy recurrent models through three variants: Memory as Context, Memory as Gate, and Memory as Layer.
    • Titans scales to context windows exceeding 2 million tokens and achieves superior performance across language, genomics, time series, and Needle-in-a-Haystack benchmarks.

    Introduction to Titans and Test-Time Neural Memory

    Titans introduces a dual-memory architectural paradigm that combines attention with a dedicated Neural Long-Term Memory module. In this framework, attention acts as short-term memory, leveraging its accurate token dependency modeling over a limited context window, while the Neural Long-Term Memory module acts as persistent memory by learning to memorize historical context. This Neural Long-Term Memory allows the system to retain and utilize long-past information to assist attention on the current context, while maintaining the advantages of fast parallelizable training and fast inference.

    Built upon the integration of these two complementary modules, the Titans family features three architectural variants designed to incorporate memory effectively: Memory as Context (MAC), Memory as Gate (MAG), and Memory as Layer (MAL). Across evaluations spanning language modeling, common-sense reasoning, genomics, and time series benchmarks, Titans demonstrates greater effectiveness than Transformers and modern linear recurrent models. Furthermore, the architecture scales to context window sizes exceeding 2M tokens while achieving higher accuracy on Needle-in-a-Haystack tasks relative to baseline models.

    Sources: Titans: Learning to Memorize at Test Time

    Architectural Paradigms: Comparing Attention and Recurrent Memory Trade-Offs

    To understand the motivation behind this architecture, it is helpful to examine modern sequence modeling, which has long been characterized by a fundamental tension between attention-based architectures and recurrent models. On one hand, Transformers excel at capturing precise, fine-grained, and direct dependencies between tokens across a context window. However, self-attention incurs a quadratic computational complexity of O(N^2) with respect to sequence length, resulting in extreme memory footprints and computational bottlenecks that severely restrict the size of usable context windows.

    In contrast, Recurrent Neural Networks (RNNs) and linear recurrent models offer linear computational complexity of O(N) and fast inference by compressing historical context into a fixed-size hidden state. However, this fixed-capacity compression introduces significant information loss and degrades associative recall over extensive contexts.

    Titans reconciles these conflicting paradigms by structuring memory hierarchically. Rather than relying entirely on quadratic self-attention or static hidden-vector compression, the architecture assigns short-term, high-resolution local dependencies to an attention mechanism while offloading past historical context to a dynamic Neural Long-Term Memory module. This design retains the fast inference and linear compute scaling of recurrent models without sacrificing the associative fidelity required for long-context sequence modeling.

    Sources: arxiv.org, youtube.com, medium.com, openreview.net, huggingface.co

    Closing thoughts

    Ultimately, by reframing sequence modeling as a hierarchical dual-memory system rather than a single compromised mechanism, Titans convincingly resolves the structural tension between quadratic attention bottlenecks and lossy recurrent compression. In our view, assigning persistent historical context to an active neural long-term memory while reserving standard attention for high-resolution local dependencies is the key architectural insight that allows the model to scale past two million tokens without degrading retrieval fidelity.

    Furthermore, the demonstrated gains across domains as varied as genomics, time series, and language benchmarks suggest that modular memory integration—whether via context, gating, or dedicated layers—is fundamentally more effective than forcing a uniform mechanism to handle both immediate and distant dependencies. In doing so, Titans proves that sequence models can achieve fast inference and linear scaling without sacrificing the associative accuracy essential for vast context windows.

    Frequently Asked Questions

    What is the dual-memory architecture used in Titans?

    Titans combines an attention mechanism that serves as short-term memory with a dedicated Neural Long-Term Memory module for persistent memory. This design allows the model to retain past information to support attention on the current context while maintaining fast parallelizable training and fast inference.

    What are the three architectural variants of Titans?

    The Titans family includes three architectural variants: Memory as Context (MAC), Memory as Gate (MAG), and Memory as Layer (MAL).

    How does Titans resolve the trade-offs between Transformers and recurrent models?

    Titans organizes memory hierarchically by assigning short-term, high-resolution local dependencies to attention while offloading past historical context to a Neural Long-Term Memory module. This enables linear compute scaling and fast inference without the information loss typical of static recurrent compression.

    What context window size can Titans achieve?

    Titans scales to context window sizes exceeding 2 million tokens while delivering higher accuracy on Needle-in-a-Haystack tasks compared to baseline models.

  • Fixing Cannot Copy Out of Meta Tensor Errors in Hugging Face Transformers

    Fixing Cannot Copy Out of Meta Tensor Errors in Hugging Face Transformers

    TL;DR

    • The ‘Cannot copy out of meta tensor’ error happens when attempting manual operations like .to(device) on meta tensors, which serve as memory-free model placeholders.
    • Avoid calling .to() manually when loading models with device_map=’auto’, as Hugging Face Accelerate automatically handles layer placement and device transfers via hooks.
    • To resolve meta tensor errors in custom pipelines, use load_checkpoint_and_dispatch() when dispatching an Accelerate-managed model, or torch.nn.Module.to_empty() when manually allocating a model before loading a state dict — they solve different scenarios, not interchangeable alternatives.

    Understanding Meta Tensors in PyTorch and Hugging Face

    The error NotImplementedError: Cannot copy out of meta tensor; no data! occurs when PyTorch, Hugging Face Transformers, or Accelerate attempts to copy data, perform operations, or execute standard .to(device) calls on a tensor residing on the meta device. In PyTorch, a tensor on the meta device acts as a lightweight placeholder that stores metadata—such as shape, strides, and data type—without allocating physical memory or actual storage bytes. Both PyTorch and Hugging Face Transformers use this mechanism to construct large model architectures instantly without overwhelming CPU RAM or GPU VRAM.

    When loading large models in Hugging Face Transformers using device_map="auto" or low_cpu_mem_usage=True, Accelerate initially instantiates the model skeleton on the meta device. Setting low_cpu_mem_usage=True instructs from_pretrained to load the model via meta-device initialization, which prevents full model weights from being loaded into system memory twice. Accelerate then calculates optimal parameter placement across available hardware (splitting across GPUs, CPU RAM, or disk) before populating real weight data. Data-dependent operations that need real bytes—such as .to("cuda") or .item()—fail on a meta tensor because no physical memory has been allocated (though the exact exception differs: .to() raises this article’s Cannot copy out of meta tensor; no data! error, while .item() raises a separate Tensor.item() cannot be called on meta tensors error). Metadata-only operations like .clone(), by contrast, succeed on a meta tensor and simply return another meta tensor. If custom logic or user code tries to interact with or move these placeholder parameters before they are fully materialized into real tensors, PyTorch throws the error.

    To avoid and fix meta tensor errors, follow these standard practices and API guidelines:

    • Avoid Manual .to() Calls on device_map="auto" Models: Do not call model.to("cuda") or model.to(device) after instantiating a model with device_map="auto". Accelerate automatically manages layer movement and execution across devices using PyTorch forward hooks. Calling manual device transfer methods forces PyTorch to attempt copying empty meta tensors, triggering the exception.
    • Use Native Checkpoint Dispatching for Custom Models: When constructing custom model pipelines using init_empty_weights() to route layer creation onto the meta device, load and place real weights using load_checkpoint_and_dispatch() — this function loads a checkpoint from disk and replaces the empty meta tensors with real weight tensors before attaching execution hooks. dispatch_model() is a narrower tool: it only attaches the execution hooks that route each layer’s forward pass to its assigned device, and it expects the model to already hold real (non-meta) weights — calling it alone on a model still on the meta device will not populate those weights.
    • Use torch.nn.Module.to_empty() for Uninitialized Allocation: If manual model allocation is required before loading state dicts into memory, replace standard .to(device) calls with torch.nn.Module.to_empty(). This native PyTorch method transfers a model off the meta device onto a target device (such as CPU or CUDA) and allocates uninitialized memory without trying to read non-existent meta tensor values.
    • Verify Distributed & Offloading Settings: In distributed training or inference environments (such as DeepSpeed ZeRO-3 or FSDP), ensure training arguments do not conflict with model loading options. Passing explicit offloading configurations or disabling incompatible CPU RAM efficient loading flags ensures model weights are fully materialized before training loops begin.

    Sources: pytorch.org, huggingface.co, huggingface.co, github.com

    Closing thoughts

    Looking at the underlying cause, the “cannot copy out of meta tensor” error most often traces back to a friction point where legacy PyTorch habits—specifically manual .to(device) transfers—conflict with modern, automated memory management. That said, it is not always user error: Accelerate’s automatic dispatch logic can itself leave some submodules on the meta device for certain model architectures, and a later data-dependent operation (such as the .item() call inside model.generate()) then fails with a related meta-tensor exception — evidence that dispatch-logic bugs, not just manual .to() misuse, are a real source of these errors. Meta tensors are essential lightweight blueprints that enable the loading of massive architectures, but they require developers to trust frameworks like Hugging Face Accelerate to handle parameter placement behind the scenes. Using load_checkpoint_and_dispatch() for Accelerate-managed models, or to_empty() when manually allocating a model before loading a state dict, resolves most instantiation hurdles—though architecture-specific dispatch failures may still require filing an upstream issue. Ultimately, respecting the boundary between placeholder metadata and materialized weights ensures clean, exception-free pipelines without causing system memory exhaustion.

    Frequently Asked Questions

    What causes the “Cannot copy out of meta tensor” error in PyTorch and Hugging Face?

    The error occurs when PyTorch, Hugging Face Transformers, or Accelerate attempts to copy data, perform operations, or execute manual .to(device) calls on a tensor residing on the meta device. Because meta tensors are lightweight placeholders with no physical memory allocated, standard data-copying operations fail.

    How does device_map=”auto” utilize meta tensors when loading models?

    When using device_map="auto" or low_cpu_mem_usage=True, Accelerate initially instantiates the model skeleton on the meta device to prevent loading full model weights into system memory twice. It calculates optimal parameter placement across hardware before populating real weight data.

    Why should manual .to() calls be avoided on models instantiated with device_map=”auto”?

    Accelerate automatically manages layer movement and execution across devices using forward hooks. Calling .to("cuda") or .to(device) manually forces PyTorch to attempt copying empty meta tensors, triggering the exception.

    How can you safely allocate or dispatch custom models created on the meta device?

    For custom models, developers should load and place real weights using load_checkpoint_and_dispatch(), which reads a checkpoint from disk and replaces the empty meta tensors with real weight tensors. dispatch_model() is not a substitute for this — it only attaches execution hooks to a model whose weights are already real, so calling it alone on a model still on the meta device will not load any weights. If manual allocation is required before loading state dicts, torch.nn.Module.to_empty() should be used instead of standard .to(device) calls.