Category: Practical Guides

  • Fixing Quality Degradation When Merging 4-Bit QLoRA Adapters in PEFT

    Fixing Quality Degradation When Merging 4-Bit QLoRA Adapters in PEFT

    TL;DR

    • In QLoRA, the base model weights are stored in 4-bit precision (NormalFloat4 / NF4 via bitsandbytes) while the trained LoRA adapter matrices stay in 16-bit floating point (torch.float16 or torch.bfloat16).
    • Calling merge_and_unload directly on a model that is still loaded in 4-bit forces PEFT to dequantize each weight to 16-bit, add the adapter delta, and then re-quantize the result back into NF4.
    • The dequantize and add steps are effectively lossless; the final re-quantization is not. It rounds every merged weight to the nearest NF4 bucket, across every layer at once, and PEFT warns this “may get different generations due to rounding errors.”
    • The fix is to load the base model in 16-bit, run merge_and_unload there, save, and — only if deployment needs it — quantize the merged checkpoint once afterward.

    Understanding the Pitfall: Why Merging into 4-Bit Weights Hurts Output Quality

    In QLoRA configurations, the base model weights are stored in 4-bit precision formats such as NormalFloat4 (NF4) via bitsandbytes, whereas the trained LoRA adapter matrices A and B are kept in 16-bit floating-point precision (torch.float16 or torch.bfloat16).

    When you call merge_and_unload() on a model that is still loaded in 4-bit, PEFT cannot add a 16-bit delta to a 4-bit weight directly. Instead it dequantizes each base weight back to the compute dtype (bf16/fp16), adds the LoRA update ΔW = (α / r)·BA in floating point, and then re-quantizes the updated weight back into NF4. The dequantize and add steps are well defined and essentially lossless. The problem is the last step: re-quantization snaps every merged weight to the nearest value its NF4 bucket can represent, and that rounding is applied to every weight in every merged layer at once.

    So there is a single mechanism at work here — re-quantization rounding error — not a pile of unrelated failure modes. NF4 and fp16 are not “incompatible” formats; the conversion between them is well defined. What you lose is precision when a 16-bit sum is forced back into NF4’s limited set of representable values.

    PEFT itself flags this rather than forbidding it: merging a LoRA module into a 4-bit linear layer “may get different generations due to rounding errors.” In practice the merged model still runs, but its outputs can shift measurably from the same adapter running unmerged — enough to matter for evaluation or production, and impossible to recover once the merge is saved. Community threads on the PEFT tracker (issue #2321, issue #2105) are users asking why this warning appears and asking for the behavior to be documented, not reports of the model being destroyed. Treat it as a quality regression to design around, not a guaranteed failure.

    Sources: huggingface.co, github.com, github.com

    The Correct Workflow: Reloading in Full Precision Before Merging

    The officially supported path is to merge the adapter into an unquantized 16-bit base model, never into the 4-bit model you trained against.

    1. Load the base model without quantization. Call transformers.AutoModelForCausalLM.from_pretrained with the compute dtype set to torch.bfloat16 or torch.float16 — use dtype= on current Transformers (torch_dtype= still works but is deprecated) — and omit quantization_config / load_in_4bit entirely for this phase. Set device_map="auto", or an explicit CPU/GPU map if memory is tight.
    2. Attach the trained adapter. peft.PeftModel.from_pretrained(base_model, adapter_path) loads the adapter weights onto the unquantized base.
    3. Call merge_and_unload(). This fuses ΔW = (α / r)·BA into the 16-bit base weights and returns a plain Transformers model with no adapter layers left. Useful arguments: safe_merge=True clones each weight matrix one layer at a time to check for NaN before committing that layer (peak overhead is one layer, not a second full model); progressbar=True shows layer-by-layer progress; adapter_names limits the merge to specific adapters.
    4. Save the merged model. model.save_pretrained(output_dir) writes a standalone 16-bit checkpoint that no longer depends on PEFT or bitsandbytes to load.
    5. Re-quantize only if deployment needs it. If you need a 4-bit or 8-bit artifact for inference, quantize this saved 16-bit checkpoint once, after the merge, with a current post-training method (bitsandbytes, torchao, AWQ, or GPTQ). That keeps quantization to a single, measurable step on the final model instead of one buried inside the merge.

    Sources: huggingface.co, huggingface.co

    System and Memory Requirements for 16-Bit Model Fusion

    The merge arithmetic itself is cheap. The real constraint is holding the base model in 16-bit, since the whole point of the correct workflow is that it is no longer quantized during the merge.

    At bf16/fp16 you need roughly two bytes per parameter, spread across VRAM and system RAM combined, plus headroom for the adapter and for the single weight matrix safe_merge clones per layer as it checks for NaN (one layer at a time, not a full second model). That is several times the footprint of the 4-bit model you trained with, which is the trade-off for avoiding re-quantization.

    You do not need a single GPU large enough to hold the whole model. device_map="auto" offloads layers to CPU RAM, and because the merge is a one-off operation, running it partly or entirely on CPU is fine — fitting matters far more than speed. If system RAM is also tight, pass an explicit device map that keeps most layers on CPU. Keep load_in_4bit off for this whole phase; the low-bit representation for deployment is produced later, from the saved checkpoint.

    Sources: huggingface.co

    Comparing Quality: 4-Bit Merge, 16-Bit Merge, and Runtime QLoRA

    Three configurations are worth keeping distinct when you evaluate the result:

    • Naive 4-bit merge — adapter fused into the 4-bit model, with the re-quantization rounding baked into every layer. It runs, but generations can diverge from the trained adapter and the loss is permanent.
    • Proper 16-bit merge — adapter fused into the unquantized base, then saved. Generations track the unmerged adapter up to ordinary floating-point noise, because no re-quantization happens during the merge.
    • Runtime QLoRA inference — base kept in 4-bit, adapter applied on the fly, nothing merged. This is the natural reference point for adapter quality, but it carries adapter overhead on every forward pass and needs bitsandbytes at serving time.

    This is a conceptual comparison, not a numeric benchmark — the right numbers are the ones you measure on your own task. For deployment, do the 16-bit merge first, then quantize the merged checkpoint if you need a low-bit model for inference. Applying quantization once to a clean merged model (bitsandbytes NF4 for a calibration-free load-time path, or AWQ / GPTQ for calibration-based PTQ) gives you one rounding pass you can actually measure, rather than one hidden inside merge_and_unload. Before shipping, validate the deployed model against the runtime-QLoRA setup on your own evaluation set — a single quantization step on the final model is far easier to sign off on than rounding introduced mid-merge.

    Sources: huggingface.co

    Closing thoughts

    Merging a QLoRA adapter straight into 4-bit base weights is not a shortcut worth taking. PEFT will do it, but it has to re-quantize every updated weight back into NF4, and that rounding pass — applied across every merged layer — is what pushes the fused model’s outputs away from what you trained.

    The reliable path is boring: load the base model in 16-bit, attach the adapter, run merge_and_unload there, save, and quantize once at the end if deployment needs it. That keeps quantization to a single measurable step on the final model instead of an invisible one inside the merge. When you do need a low-bit artifact, reach for a current post-training quantization method rather than merging inside an already-quantized model.

    Frequently Asked Questions

    What precisions are used for the base model and adapters in QLoRA?

    In QLoRA, the base model weights are stored in 4-bit precision formats like NormalFloat4 (NF4) via bitsandbytes. The trained LoRA adapter matrices are kept in 16-bit floating-point precision, such as torch.float16 or torch.bfloat16.

    Why does calling merge_and_unload directly on a 4-bit model change generation quality?

    PEFT cannot add a 16-bit adapter delta to a 4-bit weight, so it dequantizes each base weight to bf16/fp16, adds the delta, and re-quantizes the result back into NF4. That final re-quantization rounds every merged weight to the nearest NF4 bucket, across every layer at once, and PEFT warns it “may get different generations due to rounding errors.” The dequantize and add steps are lossless; only the re-quantization loses information.

    What stops the LoRA update from integrating exactly during a 4-bit merge?

    Nothing stops the addition itself — it happens in floating point and is accurate. What you lose is precision when the summed weight is forced back into NF4’s limited set of representable values. Merging into an unquantized 16-bit base skips that step entirely, so the adapter integrates without rounding loss.

  • Resolving Pydantic State Deserialization Failures in LangGraph Checkpointers

    Resolving Pydantic State Deserialization Failures in LangGraph Checkpointers

    TL;DR

    • Pydantic state deserialization problems in LangGraph checkpointers come from four distinct mechanisms with different symptoms: msgpack allowlist restrictions under strict mode, classes that cannot be re-imported (dynamic or local scope), schema drift between old checkpoints and newer models, and non-serializable attributes that actually fail when the checkpoint is written.
    • The usual symptom is silent degradation, not a raised exception: JsonPlusSerializer returns the raw kwargs dictionary (or a validation-skipped model_construct object) when it cannot rehydrate a model, so failures often surface later as an AttributeError or KeyError inside a node rather than at load time.
    • Fixes are cause-specific: register models in the msgpack allowlist for allowlist blocks, move models to module top level for import failures, add field defaults plus a model_validator or reducer for schema drift, and keep runtime objects out of state fields for the save-time case.

    Understanding LangGraph State Serialization with JsonPlusSerializer

    In LangGraph, state persistence across execution supersteps is managed by checkpointer implementations extending BaseCheckpointSaver, such as MemorySaver, SqliteSaver, and PostgresSaver. These checkpointers delegate data encoding and decoding to serialization protocols located in langgraph.checkpoint.serde. The default serialization engine is JsonPlusSerializer (langgraph.checkpoint.serde.jsonplus.JsonPlusSerializer), which encodes state primarily through msgpack, with a JSON path as fallback, while preserving standard Python types, LangChain and LangGraph primitives, and Pydantic BaseModel instances.

    When a checkpoint is loaded, the serializer rebuilds each custom object from a stored (module, class_name, kwargs) triple. Success depends on two independent conditions: whether the module passes the deserialization allowlist, and whether the class can actually be imported and constructed. By default—when LANGGRAPH_STRICT_MSGPACK is unset—the msgpack policy is permissive (allowed_msgpack_modules=True): unregistered modules are still reconstructed and only a warning is logged. Strict mode, enabled by LANGGRAPH_STRICT_MSGPACK=true or by passing an explicit allowlist, blocks modules that are not listed. The important detail is that blocking does not raise—the serializer returns the raw data instead.

    Common Causes of Deserialization Failures

    Pydantic state deserialization issues within JsonPlusSerializer stem from four distinct mechanisms. They produce different symptoms and require different fixes, so they are worth separating:

    • msgpack Allowlist Restrictions (strict mode only): Under strict mode, JsonPlusSerializer checks each custom object’s module path before instantiating it. If a Pydantic model’s module is not on the allowlist, the msgpack Pydantic branch returns the stored kwargs dictionary unchanged—no exception, and no visible signal unless you are watching the warning log. The node then receives a dict where it expected a model. InvalidModuleError is a real type, but it is raised only on the separate JSON constructor path, and even there JsonPlusSerializer‘s reviver catches it, logs a warning, and falls back to the generic LangChain reviver—so it is not the symptom to code against.
    • Classes That Cannot Be Re-Imported: Rebuilding a class during deserialization requires resolving module.ClassName back to an importable symbol. Models defined inside a function body or closure, or parametrized generic models with no concrete top-level subclass, expose no such symbol. The lookup fails and the value deserializes as a raw dictionary—independent of allowlist configuration or strict mode. Adding the class to the allowlist does not help, because the problem is resolution, not permission.
    • Schema Drift Between Checkpoint and Model: When a state model gains a required field without a default, renames a field, or tightens a type, reading an older checkpoint does not surface a Pydantic ValidationError from the serializer. It tries cls(**kwargs), and on failure falls back to cls.model_construct(**kwargs), which skips validation and returns a partially populated model; if that also fails, it returns the raw dict. A ValidationError appears only later, when a reducer or node re-validates the value.
    • Non-Serializable Attributes (a save-time failure): Placing runtime artifacts that cannot be encoded—raw exception objects, thread locks, open client sessions, file handles—inside Pydantic BaseModel fields makes the checkpoint fail when it is written, not when it is read. Despite the “deserialization” label, the fix belongs on the encoding side: keep these objects out of persisted state.

    Resolution Strategies and Best Practices

    Each strategy targets a specific mechanism above rather than all of them at once.

    • Register Models in the msgpack Allowlist (cause 1): When running with strict msgpack mode, pass custom Pydantic models to allowed_msgpack_modules on JsonPlusSerializer—as class objects or (module, qualname) tuples—or call serde.with_msgpack_allowlist([...]) to obtain a new serializer with a merged allowlist. allowed_msgpack_modules and allowed_json_modules gate two different decode paths; msgpack carries the large majority of state, so it is the one that usually matters. Compiling a StateGraph with strict mode enabled already derives an allowlist from the state, input, output, and context schemas, so—when the chosen checkpointer supports with_allowlist(...)—top-level state models are covered without manual registration:
    from langgraph.checkpoint.serde.jsonplus import JsonPlusSerializer
    from langgraph.checkpoint.sqlite import SqliteSaver
    from my_app.schemas import CustomStateModel
    
    serde = JsonPlusSerializer(
        allowed_msgpack_modules=[CustomStateModel]  # or [("my_app.schemas", "CustomStateModel")]
    )
    checkpointer = SqliteSaver(conn, serde=serde)
    
    • Declare State Models at Top-Level Module Scope (cause 2): Define every Pydantic state schema, sub-model, and dataclass at the top level of an importable module rather than inside a nested function or closure. For generic models in Pydantic v2, subclass BaseModel together with typing.Generic[T] and declare an explicit concrete subclass at module scope, so the stored metadata maps to an importable symbol.
    • Design for Schema Resilience (cause 3): Give every new field a default via Field(default=...) or Field(default_factory=...) so old checkpoints still satisfy the current model. Then use @model_validator(mode="before") or a graph channel reducer to coerce any dictionary fallback payload back into the typed BaseModel before nodes observe it.
    • Keep Runtime Objects Out of State (cause 4): Persist identifiers or serializable configuration instead of live handles, and rebuild the runtime object inside the node. For genuinely custom encoding, implement SerializerProtocol (dumps_typed(obj) -> tuple[str, bytes] and loads_typed(data) -> Any) and pass it as the checkpointer’s serde.

    Sources: github.com, langchain.com, langchain.com, langchain.com, langchain.com

    Closing thoughts

    These four failure modes do not share a single root cause, and it is a mistake to treat them all as the price of security hardening. Only the msgpack allowlist check is a security control. Import-resolution failures are a packaging and code-organization problem; schema drift is a data-contract problem between a stored checkpoint and a newer model; non-serializable attributes are an encoding constraint that shows up before anything is ever read back. What the four have in common is only the outcome: JsonPlusSerializer prefers to hand back a raw dictionary or a validation-skipped object rather than raise, so a broken assumption travels silently into your graph.

    The practical consequence is that resilient graphs treat state models as version-stable data contracts and verify what actually comes out of the checkpointer. Define models at module top level so they can be re-imported, give every new field a default so old checkpoints still load, and add a model_validator or channel reducer that turns any dict fallback back into a typed model before a node touches it. Reserve allowlist configuration for the one case it addresses—strict msgpack mode—and enable strict mode deliberately rather than by accident.

    Frequently Asked Questions

    What are the primary causes of Pydantic state deserialization failures in LangGraph?

    Four distinct mechanisms: (1) under strict msgpack mode, a model whose module is not on the allowlist is returned as a raw dict; (2) classes defined in a function body, closure, or as an unsubclassed generic cannot be re-imported and also come back as dicts; (3) schema drift between an old checkpoint and a newer model, where the serializer falls back to model_construct or a raw dict rather than raising; and (4) non-serializable attributes such as locks, sessions, or exception objects in state fields, which fail when the checkpoint is written.

    What does a deserialization failure actually look like at runtime?

    Usually not an exception. JsonPlusSerializer returns the stored kwargs dictionary, or a model_construct object with validation skipped, when it cannot rehydrate a model. The failure typically surfaces later as an AttributeError or KeyError inside a node that expected a typed model. InvalidModuleError exists but is confined to the JSON constructor path and is caught internally; a ValidationError appears only if a reducer or node re-validates the value.

    How do you register custom Pydantic models for the msgpack allowlist?

    Pass them to allowed_msgpack_modules on JsonPlusSerializer, as class objects or (module, qualname) tuples, or call serde.with_msgpack_allowlist([...]) to get a new serializer with a merged allowlist. When you compile a StateGraph with strict mode enabled, LangGraph already derives an allowlist from the state, input, output, and context schemas, so top-level state models are covered without manual registration—provided the chosen checkpointer supports with_allowlist(...).

    Why must state models be defined at top-level module scope?

    Deserialization resolves a stored module.ClassName back to an importable symbol. Models defined inside nested functions, or generics without a concrete top-level subclass, have no such symbol, so they deserialize as raw dictionaries no matter how the allowlist is configured.

    How do you keep old checkpoints loading after a schema change?

    Add defaults with Field(default=...) or Field(default_factory=...) for every new field, and add a @model_validator(mode="before") or a channel reducer that coerces a plain-dict fallback back into the typed model before nodes run.

  • Resolving SQLite Database Lock Errors in ChromaDB During Concurrent Writes

    Resolving SQLite Database Lock Errors in ChromaDB During Concurrent Writes

    TL;DR

    • SQLite enforces a global write lock in Embedded ChromaDB, which causes SQLITE_BUSY database lock errors during concurrent write operations and read-to-write lock upgrades.
    • Standard write contention can be mitigated by keeping insert batch sizes between 50 and 250 records and raising SQLite’s own PRAGMA busy_timeout to at least 5 seconds — this is a general SQLite setting, not a parameter ChromaDB itself exposes.
    • Transitioning to ChromaDB’s Client-Server mode using HttpClient or AsyncHttpClient resolves embedded SQLite locking limits by offloading concurrency control to a standalone server process.

    Understanding Embedded ChromaDB and SQLite Persistence Engine

    Embedded ChromaDB handles local database persistence using PersistentClient. This client automatically saves database files to the local machine and reloads them on startup if previous data exists.

    The storage location on disk is configured through the path parameter, which determines where database files are written and loaded from. If no path is explicitly provided, the client defaults to using .chroma.

    The persistence client exposes programmatic methods for system management:

    • heartbeat(): Returns a nanosecond heartbeat value, useful for confirming the client is still connected.
    • reset(): Empties and completely resets the database. Executing this method is destructive and cannot be reversed.

    Sources: docs.trychroma.com — Persistent Client

    Root Causes of SQLite Locking Issues in Concurrent Write Workloads

    Underneath this local persistence mechanism, SQLite manages concurrent write operations by enforcing a global write lock that permits only one writer at a time. When a write transaction begins, it holds this global lock for its entire duration, blocking all other write attempts until the transaction completes.

    The SQLITE_BUSY error (“database is locked”) occurs whenever a transaction cannot acquire this global write lock. If another transaction holds the lock, even a simple insert operation fails immediately with this error.

    A distinct edge case occurs when an active read transaction attempts to upgrade to a write transaction. If another database connection has already modified the database or is in the process of modifying it, this upgrade attempt fails immediately with SQLITE_BUSY. Unlike general write lock contention, this specific read-to-write lock upgrade failure is not helped by setting a busy timeout.

    To address standard write lock contention, SQLite itself — independent of ChromaDB, which does not expose this as a documented client parameter — provides a general pragma, PRAGMA busy_timeout. This sets the duration that transactions will wait to acquire the write lock before returning “database is locked” instead of failing immediately. In production systems, a PRAGMA busy_timeout setting of 5 seconds or more is recommended.

    Sources: tenthousandmeters.com — SQLite concurrent writes and "database is locked" errors

    Mitigating Write Contention Through Batch Size and Understanding SQLite Storage Behavior

    Beyond PRAGMA busy_timeout, how ingestion parameters are configured also shapes how often write locks get contended in the first place. ChromaDB documentation recommends keeping insert batch sizes on the smaller side, specifically between 50 and 250 records. Choosing batch sizes in this range yields lower and more consistent latency while making writes less likely to encounter timeout errors. Although overall throughput plateaus and remains fairly flat across batch sizes between 100 and 500, smaller batches are preferred to prevent latency spikes and timeouts.

    For insert write concurrency, ChromaDB records writes to a log and flushes them every N operations. Because of this logging and flushing mechanism, mean latency does not fluctuate as the number of concurrent writers increases.

    Separately from locking behavior, ChromaDB’s on-disk footprint is worth understanding on its own: the database saves metadata and documents via SQLite, and disk usage is highly variable, depending entirely on whether full documents and metadata are being retained. For example, a sample collection containing approximately 40,000 documents (averaging 1,000 words each) and roughly 600,000 metadata entries requires about 1.7GB of storage. SQLite handles database disk paging effectively and supports database sizes scaling into the terabyte range.

    Sources: docs.trychroma.com — Performance guide (single node)

    Architectural Alternatives: Embedded SQLite vs Client-Server Mode for Multi-Threaded Writes

    When local tuning strategies and parameter adjustments prove insufficient for multi-threaded or multi-process write workloads, ChromaDB supports Client-Server ChromaDB where applications interact with a standalone server process instead of managing local storage. Synchronous access is established using HttpClient, whereas non-blocking access is provided via AsyncHttpClient. The two client implementations maintain identical method signatures and behaviors, but AsyncHttpClient makes all methods that would otherwise block run asynchronously.

    Configuring the standalone server instance is handled through environment variables:

    • CHROMA_PERSIST_PATH: Dictates the directory used for persisted data. This defaults to ./chroma in the frontend configuration and is commonly set to /data in container deployments.
    • CHROMA_SQLITE_FILENAME: Specifies the SQLite database filename created under the persist path, defaulting to chroma.sqlite3.
    • CHROMA_LISTEN_ADDRESS: Sets the bind address for the frontend server, defaulting to 0.0.0.0.
    • CHROMA_PORT: Defines the HTTP listening port for the frontend server, defaulting to 8000.

    Adopting Client-Server ChromaDB provides an architectural alternative to running Embedded ChromaDB directly inside client processes. In one third-party case, developers maintaining the MemPalace project reported encountering chromadb.errors.InternalError: Error in compaction: Failed to apply logs to the hnsw segment writer during concurrent multi-process operation. Their assessment was that Embedded ChromaDB’s SQLite database and HNSW segment writers are inherently not thread- or process-safe for concurrent writers. To address this within their application, they implemented a workaround by switching from the embedded PersistentClient to HttpClient, offloading concurrency and execution control entirely to the server process.

    It’s worth being precise about what this MemPalace failure actually is: an HNSW compaction error is a different failure mode from the SQLITE_BUSY lock errors described earlier in this article. It isn’t SQLite’s write-lock contention — it originates from concurrent access to the HNSW vector-index segment writer instead. PRAGMA busy_timeout has no bearing on it. The two problems just happen to share the same fix here (moving to Client-Server mode), not the same underlying mechanism.

    Sources: docs.trychroma.com — Client-Server mode (HttpClient / AsyncHttpClient), docs.trychroma.com — Server environment variables, MemPalace/mempalace issue #832

    Closing thoughts

    Ultimately, while Embedded ChromaDB provides a convenient local persistence mechanism, attempting to push it through heavy concurrent write workloads reveals clear architectural limitations inherent to embedded SQLite. Keeping batch sizes between 50 and 250 records, and raising SQLite’s own PRAGMA busy_timeout (a general SQLite setting, not something ChromaDB exposes as a client parameter), can alleviate standard SQLite write contention, but neither helps with unhandled read-to-write lock upgrades, and neither has any bearing on HNSW compaction failures — a separate concurrency issue in the HNSW segment writer, illustrated so far only by one third party’s reported case, rather than a SQLite locking problem. In my judgment, pushing heavy concurrent write workloads through Embedded ChromaDB carries stability risk on both fronts — SQLite lock contention and (per that one reported case) HNSW segment-writer safety alike. Transitioning to ChromaDB’s Client-Server mode using HttpClient or AsyncHttpClient remains the most definitive solution: it doesn’t make the underlying SQLite database disappear (the server itself still persists to chroma.sqlite3), but it stops multiple application processes from contending directly over the same local SQLite file, and in the MemPalace case, moving to HttpClient was also reported to resolve the HNSW segment-writer failure.

    Frequently Asked Questions

    What causes the SQLITE_BUSY database locked error in ChromaDB?

    The SQLITE_BUSY error occurs when a transaction cannot acquire SQLite’s global write lock because another transaction is holding it. It also happens when an active read transaction attempts to upgrade to a write transaction while another connection is modifying the database.

    What batch size is recommended for ChromaDB insert operations?

    ChromaDB documentation recommends keeping insert batch sizes between 50 and 250 records. This range yields lower, more consistent latency and makes writes less likely to hit timeout errors.

    How does PRAGMA busy_timeout help handle SQLite write lock contention?

    PRAGMA busy_timeout sets the duration that transactions will wait to acquire the write lock before failing with a database locked error. A setting of 5 seconds or more is recommended in production systems.

    How does switching to Client-Server mode solve embedded SQLite locking limitations?

    Transitioning to Client-Server mode offloads concurrency and execution control to a dedicated server process using HttpClient or AsyncHttpClient. This architecture bypasses the embedded SQLite locking constraints and thread-safety limitations.

  • Fixing Cannot Copy Out of Meta Tensor Errors in Hugging Face Transformers

    Fixing Cannot Copy Out of Meta Tensor Errors in Hugging Face Transformers

    TL;DR

    • The ‘Cannot copy out of meta tensor’ error happens when attempting manual operations like .to(device) on meta tensors, which serve as memory-free model placeholders.
    • Avoid calling .to() manually when loading models with device_map=’auto’, as Hugging Face Accelerate automatically handles layer placement and device transfers via hooks.
    • To resolve meta tensor errors in custom pipelines, use load_checkpoint_and_dispatch() when dispatching an Accelerate-managed model, or torch.nn.Module.to_empty() when manually allocating a model before loading a state dict — they solve different scenarios, not interchangeable alternatives.

    Understanding Meta Tensors in PyTorch and Hugging Face

    The error NotImplementedError: Cannot copy out of meta tensor; no data! occurs when PyTorch, Hugging Face Transformers, or Accelerate attempts to copy data, perform operations, or execute standard .to(device) calls on a tensor residing on the meta device. In PyTorch, a tensor on the meta device acts as a lightweight placeholder that stores metadata—such as shape, strides, and data type—without allocating physical memory or actual storage bytes. Both PyTorch and Hugging Face Transformers use this mechanism to construct large model architectures instantly without overwhelming CPU RAM or GPU VRAM.

    When loading large models in Hugging Face Transformers using device_map="auto" or low_cpu_mem_usage=True, Accelerate initially instantiates the model skeleton on the meta device. Setting low_cpu_mem_usage=True instructs from_pretrained to load the model via meta-device initialization, which prevents full model weights from being loaded into system memory twice. Accelerate then calculates optimal parameter placement across available hardware (splitting across GPUs, CPU RAM, or disk) before populating real weight data. Data-dependent operations that need real bytes—such as .to("cuda") or .item()—fail on a meta tensor because no physical memory has been allocated (though the exact exception differs: .to() raises this article’s Cannot copy out of meta tensor; no data! error, while .item() raises a separate Tensor.item() cannot be called on meta tensors error). Metadata-only operations like .clone(), by contrast, succeed on a meta tensor and simply return another meta tensor. If custom logic or user code tries to interact with or move these placeholder parameters before they are fully materialized into real tensors, PyTorch throws the error.

    To avoid and fix meta tensor errors, follow these standard practices and API guidelines:

    • Avoid Manual .to() Calls on device_map="auto" Models: Do not call model.to("cuda") or model.to(device) after instantiating a model with device_map="auto". Accelerate automatically manages layer movement and execution across devices using PyTorch forward hooks. Calling manual device transfer methods forces PyTorch to attempt copying empty meta tensors, triggering the exception.
    • Use Native Checkpoint Dispatching for Custom Models: When constructing custom model pipelines using init_empty_weights() to route layer creation onto the meta device, load and place real weights using load_checkpoint_and_dispatch() — this function loads a checkpoint from disk and replaces the empty meta tensors with real weight tensors before attaching execution hooks. dispatch_model() is a narrower tool: it only attaches the execution hooks that route each layer’s forward pass to its assigned device, and it expects the model to already hold real (non-meta) weights — calling it alone on a model still on the meta device will not populate those weights.
    • Use torch.nn.Module.to_empty() for Uninitialized Allocation: If manual model allocation is required before loading state dicts into memory, replace standard .to(device) calls with torch.nn.Module.to_empty(). This native PyTorch method transfers a model off the meta device onto a target device (such as CPU or CUDA) and allocates uninitialized memory without trying to read non-existent meta tensor values.
    • Verify Distributed & Offloading Settings: In distributed training or inference environments (such as DeepSpeed ZeRO-3 or FSDP), ensure training arguments do not conflict with model loading options. Passing explicit offloading configurations or disabling incompatible CPU RAM efficient loading flags ensures model weights are fully materialized before training loops begin.

    Sources: pytorch.org, huggingface.co, huggingface.co, github.com

    Closing thoughts

    Looking at the underlying cause, the “cannot copy out of meta tensor” error most often traces back to a friction point where legacy PyTorch habits—specifically manual .to(device) transfers—conflict with modern, automated memory management. That said, it is not always user error: Accelerate’s automatic dispatch logic can itself leave some submodules on the meta device for certain model architectures, and a later data-dependent operation (such as the .item() call inside model.generate()) then fails with a related meta-tensor exception — evidence that dispatch-logic bugs, not just manual .to() misuse, are a real source of these errors. Meta tensors are essential lightweight blueprints that enable the loading of massive architectures, but they require developers to trust frameworks like Hugging Face Accelerate to handle parameter placement behind the scenes. Using load_checkpoint_and_dispatch() for Accelerate-managed models, or to_empty() when manually allocating a model before loading a state dict, resolves most instantiation hurdles—though architecture-specific dispatch failures may still require filing an upstream issue. Ultimately, respecting the boundary between placeholder metadata and materialized weights ensures clean, exception-free pipelines without causing system memory exhaustion.

    Frequently Asked Questions

    What causes the “Cannot copy out of meta tensor” error in PyTorch and Hugging Face?

    The error occurs when PyTorch, Hugging Face Transformers, or Accelerate attempts to copy data, perform operations, or execute manual .to(device) calls on a tensor residing on the meta device. Because meta tensors are lightweight placeholders with no physical memory allocated, standard data-copying operations fail.

    How does device_map=”auto” utilize meta tensors when loading models?

    When using device_map="auto" or low_cpu_mem_usage=True, Accelerate initially instantiates the model skeleton on the meta device to prevent loading full model weights into system memory twice. It calculates optimal parameter placement across hardware before populating real weight data.

    Why should manual .to() calls be avoided on models instantiated with device_map=”auto”?

    Accelerate automatically manages layer movement and execution across devices using forward hooks. Calling .to("cuda") or .to(device) manually forces PyTorch to attempt copying empty meta tensors, triggering the exception.

    How can you safely allocate or dispatch custom models created on the meta device?

    For custom models, developers should load and place real weights using load_checkpoint_and_dispatch(), which reads a checkpoint from disk and replaces the empty meta tensors with real weight tensors. dispatch_model() is not a substitute for this — it only attaches execution hooks to a model whose weights are already real, so calling it alone on a model still on the meta device will not load any weights. If manual allocation is required before loading state dicts, torch.nn.Module.to_empty() should be used instead of standard .to(device) calls.

  • Fixing High CPU Spikes and Freezes During Large Codebase Indexing in Cursor

    Fixing High CPU Spikes and Freezes During Large Codebase Indexing in Cursor

    TL;DR

    • High CPU spikes during Cursor codebase indexing are usually not an embedding-generation cost — they mostly come from background rg (ripgrep) processes scanning large binary/artifact-heavy workspaces with wide-scope flags.
    • Developers can shrink that scan scope by excluding heavy files with .cursorignore, .cursorindexingignore, and a global ignore list, disabling symlink traversal, and keeping the indexed file count low — all of which narrow what the background scan has to cover.
    • Separately, adding watcher exclusions, capping TypeScript server memory, and disabling redundant linting extensions address other CPU-consuming processes that often run alongside indexing, not the scan itself.

    Understanding Why Cursor Indexing Triggers High CPU Spikes

    Cursor builds a searchable semantic index of a project’s codebase by generating Vector Embeddings for code files. This Codebase Indexing process powers AI features such as Semantic Search, @Codebase context retrieval, and agent reasoning across entire repositories. (Cursor Tab is a separate feature that draws on recent edits, surrounding code, and linter errors rather than the codebase index — though .cursorignore still blocks Tab suggestions on excluded files.) However, when opening large codebases, monorepos, or projects containing heavy build artifacts, third-party packages, binary models, or deeply nested dependencies, indexing can cause temporary CPU spikes, increased memory usage, or responsiveness freezes.

    The spikes are not primarily an embedding-generation cost. According to a Cursor forum thread where Cursor support diagnosed a user’s report of CPU usage staying above 350% across multiple processes, the runaway cost comes from background rg (ripgrep) processes that scan the workspace for rules files and codebase indexing. On workspaces with many large binary files and artifacts, flags like --hidden, --follow, and --no-ignore-parent dramatically expand the scan scope, and it is this filesystem scan — not the downstream embedding step — that pins the CPU. (The same thread also documents a separate, unresolved bug where tracked files appear to vanish after a crash during heavy indexing over Remote SSH; support notes ripgrep itself is read-only and cannot delete files, so the cause there remains an open question distinct from the CPU spikes discussed here.)

    To control indexing behavior and avoid background performance degradation, Cursor exposes several configuration mechanisms:

    • .cursorignore: Placed in the project root directory, this file uses standard .gitignore pattern syntax to explicitly exclude files and directories from all AI context operations and indexing. Its scope includes blocking files from Semantic Search indexing, Cursor Tab, inline editing, Agent tools, and @ mentions.
    • .cursorindexingignore: A specialized configuration file used exclusively to skip specified files or folders during background Codebase Indexing. Unlike .cursorignore, developers can still manually reference items listed in .cursorindexingignore using @ mentions when necessary.
    • .gitignore and Default Ignore Integration: Cursor automatically respects standard .gitignore rules across subdirectories, along with a built-in default ignore list (e.g., lockfiles, build artifacts, and binary or media files) for Codebase Indexing.
    • A global ignore list: Cursor’s settings expose a way to apply exclusion patterns (e.g., **/.env, **/dist/**) across all open workspaces without committing workspace-specific ignore files — look under the Indexing/Ignore Files section of Cursor Settings, since the exact menu path has moved between versions.
    • Indexing & Docs Settings Panel: Accessible via Cursor Settings > Features > Codebase Indexing (or Indexing & Docs), this panel displays workspace indexing status, permits inspection of included files through “View included files”, provides toggles for automatic indexing on new repositories, and offers controls to clear cache or trigger manual re-indexing.

    To prevent performance issues and mitigate CPU spikes and freezes in large environments, community guides and support threads recommend several practices:

    • Exclude Heavy and Non-Code Artifacts: Place large dependency folders (node_modules/, vendor/), generated outputs (dist/, build/), machine learning checkpoints (.pth, .onnx, .npz), and log files into .cursorignore — this is what keeps the rg scan scope small in the first place.
    • Keep the Indexed File Count in the 500–2,000 Range: eastondev.com’s guide frames this range as its primary recommendation for query speed rather than CPU load directly, and pairs it with at least 16GB of RAM (32GB for comfortable headroom) — but a smaller indexed working set is also, by definition, less for the background rg scan above to cover.
    • Disable Symlink Traversal: Set search.followSymlinks to false in settings to prevent background file search tools from recursively traversing symlink loops in complex workspaces.
    • Configure File Watcher Exclusions: Add heavy generated directories to files.watcherExclude in workspace settings (.vscode/settings.json) to minimize OS file-system watching overhead.
    • Cap TypeScript Server Memory and Disable Redundant Linters (a separate CPU source, not the indexing rg scan above): smartremotegigs.com recommends setting typescript.tsserver.maxTsServerMemory to a fixed value (e.g., 2048) and disabling real-time linting extensions (ESLint, SonarLint, Prettier), since these run as their own processes alongside indexing and can compound the same symptoms — CPU spikes and freezes — through an entirely different mechanism.
    • Scoped Monorepo Strategies: For large monorepos, structure project rules (.cursor/rules/*.mdc) or set up localized .cursorignore rules to restrict the focus of AI scanning to relevant sub-projects.
    • Last Resort — Clear the Index Cache: If spikes persist after the above, delete Cursor’s local index cache (~/Library/Application Support/Cursor/Index/ on Mac, %APPDATA%\Cursor\Index\ on Windows) and let it reindex from scratch.

    Sources: towardsdatascience.com, eastondev.com, cursor.com, smartremotegigs.com

    Closing thoughts

    When examining these performance dynamics and configuration controls, severe CPU spikes during indexing are rarely a flaw in Cursor itself, but rather the cost of letting background rg scans run unconstrained across massive, artifact-heavy codebases. Developers do not need to sacrifice deep AI features to maintain a fluid workspace; the solution lies in thoughtfully defining scanning boundaries using targeted options like .cursorindexingignore and a global ignore list, and in keeping the indexed file count low. By pairing these ignore configurations with precise editor tweaks — disabling symlink traversal, turning off file watchers for generated directories, and separately taming TypeScript server memory or redundant linting extensions — you eliminate performance bottlenecks without losing context retrieval. Ultimately, taking a proactive approach to workspace boundaries turns background indexing from a resource-draining nuisance into a quiet, highly responsive feature.

    Frequently Asked Questions

    Why does codebase indexing in Cursor trigger high CPU spikes?

    Cursor generates vector embeddings for code files to build a searchable semantic index, but the CPU spikes themselves mostly trace back to an earlier step: background rg (ripgrep) processes scanning the workspace for rules files and codebase indexing. On workspaces with many large binary files, build artifacts, or deeply nested dependencies, wide-scope scan flags can push CPU usage past 350% before embedding generation even begins.

    What is the difference between .cursorignore and .cursorindexingignore?

    .cursorignore excludes files from all AI context operations and indexing, blocking them from @ mentions, inline editing, and Agent tools. Conversely, .cursorindexingignore only skips files during background codebase indexing, allowing developers to still manually reference them using @ mentions.

    How can developers exclude files globally across all workspaces in Cursor?

    Cursor’s settings expose a global ignore list under the Indexing/Ignore Files section (the exact menu path has moved across versions) that applies universal exclusion patterns across all open workspaces without requiring workspace-specific ignore files.

    What settings adjustments can minimize performance bottlenecks during indexing?

    Developers can set search.followSymlinks to false to prevent recursive symlink loops and add heavy generated directories to files.watcherExclude in workspace settings to reduce OS file-system watching overhead.

  • Fixing Silent PyTorch DataLoader Crashes in Docker Containers

    Fixing Silent PyTorch DataLoader Crashes in Docker Containers

    TL;DR

    • PyTorch DataLoader uses POSIX shared memory at /dev/shm to efficiently pass CPU tensor data between parallel worker processes.
    • Docker containers allocate a default limit of only 64 MB to /dev/shm, causing workers to crash with a SIGBUS signal once that space is exhausted — a distinct failure from the SIGKILL signal issued when a container’s actual RAM runs out.
    • Preventing these silent DataLoader crashes requires configuring container-level shared memory to match PyTorch’s multi-process requirements.

    Understanding the Cause and Symptoms of DataLoader Shared Memory Crashes

    PyTorch supports multi-process data loading through torch.utils.data.DataLoader when the num_workers parameter is set to a value greater than zero. Under this configuration, parallel worker processes fetch and collate data simultaneously. To efficiently pass CPU tensor data between worker processes and the main process without expensive data copying, PyTorch relies on Inter-Process Communication (IPC) backed by POSIX shared memory, which is mounted at /dev/shm.

    By default, Docker allocates a restricted size of only 64 MB to /dev/shm for Docker containers. When deep learning workloads involve large batch sizes, heavy tensor payloads, or multiple worker processes, the active data buffers can quickly exceed this 64 MB shared memory allocation. Once POSIX shared memory is exhausted, the operating system can no longer satisfy memory mapping (mmap) requests issued by the data loading processes, and worker processes terminate abruptly with a SIGBUS (Bus Error) signal — the specific symptom of /dev/shm exhaustion. This is a distinct failure mode from a SIGKILL signal, which instead indicates that the Linux OOM killer terminated a worker because the container’s actual RAM allocation — a separate resource from /dev/shm — was exhausted. Both failure modes surface as silent process exits or the generic runtime error DataLoader worker exited unexpectedly, which is why the two causes are frequently conflated during debugging.

    Sources: pytorch.org, pytorch.org, datawookie.dev, last9.io, github.com

    Closing thoughts

    Ultimately, these unexpected crashes highlight a fundamental mismatch between PyTorch’s high-performance IPC design and Docker’s restrictive default container settings. In my view, the default 64 MB shared memory cap creates an insidious trap, as misleading signals like SIGBUS or generic exit codes frequently prompt developers to hunt for non-existent bugs in their data code rather than addressing the underlying environment. Because modern workloads easily exceed this tiny allocation when using multiple workers, encountering these silent crashes is almost inevitable under unconfigured container setups. Ensuring reliable multi-process data loading ultimately requires recognizing that PyTorch’s parallel architecture cannot be divorced from proper container-level shared memory configuration.

    Frequently Asked Questions

    Why does PyTorch DataLoader crash silently in default Docker containers?

    PyTorch uses POSIX shared memory at /dev/shm to pass tensor data between worker processes and the main process when num_workers is greater than zero. Docker restricts /dev/shm to 64 MB by default, which is easily exhausted during heavy data loading workloads, leading to worker crashes.

    What default shared memory allocation does Docker provide to containers?

    By default, Docker allocates a restricted size of only 64 MB to /dev/shm.

    What error signals or messages occur when PyTorch shared memory is exhausted?

    Worker processes typically terminate with a SIGBUS (Bus Error) signal, the specific symptom of /dev/shm exhaustion. A SIGKILL signal is a separate failure mode caused by the Linux OOM killer terminating a worker due to actual RAM exhaustion, not /dev/shm size. Both surface as silent process exits or the generic runtime error “DataLoader worker exited unexpectedly”.

    Why does PyTorch DataLoader rely on POSIX shared memory?

    PyTorch relies on Inter-Process Communication (IPC) backed by POSIX shared memory to pass CPU tensor data between worker processes and the main process. This avoids expensive data copying when running multi-process data loading.

  • Debugging Silent Context Truncation in Ollama’s OpenAI-Compatible API

    Debugging Silent Context Truncation in Ollama’s OpenAI-Compatible API

    TL;DR

    • Ollama’s OpenAI-compatible API silently trims prompt inputs that exceed the context window instead of returning an explicit error.
    • This truncation occurs because standard OpenAI API parameters like max_tokens do not configure Ollama’s input context size.
    • Developers can detect truncation using OLLAMA_DEBUG=1 and resolve it by setting PARAMETER num_ctx in a custom Modelfile or using the OLLAMA_CONTEXT_LENGTH environment variable.

    Understanding Silent Context Truncation in Ollama

    In LLM serving frameworks like Ollama, context window size represents the maximum number of tokens (comprising both the input prompt and output response) that a model can retain in memory at a given time. When applications send requests to Ollama’s OpenAI-compatible API endpoint v1/chat/completions, input sequences that exceed the model’s assigned context window trigger silent context truncation. Instead of throwing an explicit error, Ollama trims earlier messages or system prompts to fit within the allocated context window, which can cause subtle response quality degradation or lost instructions.

    This behavior is primarily driven by API schema discrepancies and default runtime boundaries:

    • API Schema Discrepancy: While native Ollama endpoints (/api/generate or /api/chat) accept context window configuration directly via the num_ctx parameter, the standard OpenAI-compatible API schema does not natively support num_ctx. Standard parameters like max_tokens control output generation limits rather than input context allocation. Consequently, requests routed through v1/chat/completions rely on the model’s defined context window or global server settings unless overridden in a custom model.
    • Default Context Limits: To prevent out-of-memory (OOM) errors, Ollama enforces default context window limits—traditionally 2048 or 4096 tokens, or variable limits based on available GPU VRAM. Even if a model natively supports a larger context window (such as 32k or 128k tokens), running it without explicit configuration binds it to a smaller runtime default.

    Several diagnostic tools can be used to inspect active context window bounds and detect truncation events:

    • Server Debug Mode: Launching the server with OLLAMA_DEBUG=1 ollama serve enables verbose logging. The server logs will explicitly flag truncation with messages such as truncating input prompt or truncating input messages which exceed context length.
    • Runtime Verification: Running ollama ps in the CLI shows currently loaded models alongside their actively allocated memory bounds under the CONTEXT column.
    • Modelfile Inspection: Running ollama show --modelfile <model_name> inspects a model’s base configuration to verify whether a default PARAMETER num_ctx has been saved.

    To adjust and expand the active context window for OpenAI-compatible API client integrations, context window size can be configured via a custom model Modelfile or a server-wide environment variable.

    Creating a custom model via a Modelfile is the recommended method for OpenAI-compatible API compatibility:

    FROM llama3.2
    PARAMETER num_ctx 16384
    

    After building the model with ollama create my-custom-model, specify "model": "my-custom-model" in v1/chat/completions API calls.

    Alternatively, setting the OLLAMA_CONTEXT_LENGTH environment variable before starting the server establishes a global context window default for loaded models:

    OLLAMA_CONTEXT_LENGTH=32768 ollama serve
    

    For native Ollama endpoints (/api/generate, /api/chat), runtime context window configurations resolve according to the following priority hierarchy — note this num_ctx request parameter is NOT available on the OpenAI-compatible v1/chat/completions endpoint, which can only be configured via the Modelfile or environment variable below:

    Native API Request Parameters (num_ctx, native endpoints only) > Modelfile (PARAMETER num_ctx) > Environment Variable (OLLAMA_CONTEXT_LENGTH) > Default Allocation

    Sources: ollama.com, serverman.co.uk, medium.com, reddit.com, ollama.com

    Closing thoughts

    Ultimately, while Ollama’s OpenAI-compatible API offers seamless integration, its conservative runtime defaults create a subtle trap where longer prompts are quietly trimmed without raising explicit errors. Relying on out-of-the-box settings is risky because standard OpenAI parameters like max_tokens affect output generation rather than expanding Ollama’s input context window. In my view, explicitly configuring context limits via custom Modelfiles or the OLLAMA_CONTEXT_LENGTH environment variable should be treated as a mandatory setup step for any production v1/chat/completions integration. Actively navigating this configuration hierarchy and validating allocated limits through debug tools is essential to prevent invisible prompt degradation while maintaining system stability.

    Frequently Asked Questions

    What is silent context truncation in Ollama?

    Silent context truncation occurs when input sequences exceed a model’s allocated context window size. Instead of throwing an explicit error, Ollama trims earlier messages or system prompts to fit within the available context window.

    Why does silent context truncation happen with Ollama’s OpenAI-compatible API?

    The standard OpenAI-compatible API schema does not natively support Ollama’s num_ctx configuration parameter, and parameters like max_tokens only restrict output length. Consequently, requests default to conservative server or runtime context limits unless overridden.

    How can I detect if context truncation is occurring in Ollama?

    You can start the server with OLLAMA_DEBUG=1 to view verbose logs that explicitly flag truncation events. Additionally, running ollama ps shows loaded context bounds, and ollama show –modelfile displays default model parameters.

    How do I expand the context window size for OpenAI-compatible API calls in Ollama?

    You can define PARAMETER num_ctx in a custom model’s Modelfile or set the OLLAMA_CONTEXT_LENGTH environment variable before starting the Ollama server.

  • vLLM CUDA Out of Memory: Fixing Startup, Runtime, and Persistent OOM Loops

    vLLM CUDA Out of Memory: Fixing Startup, Runtime, and Persistent OOM Loops

    TL;DR

    • “vLLM CUDA OOM” is three different problems: a startup failure (ValueError: To serve at least one request with the model's max seq len ...), a runtime failure under load (torch.OutOfMemoryError: CUDA out of memory. Tried to allocate ...), and a persistent loop where every request 500s after one bad one.
    • Almost every memory knob in vLLM — gpu_memory_utilization, max_model_len, max_num_seqs, kv_cache_dtype, enforce_eager, tensor_parallel_size — is an engine-init argument. You cannot change it on a running server, so most real fixes require a restart.
    • The only truly live lever is client-side: send fewer concurrent requests, smaller max_tokens, shorter prompts. PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True needs one restart to take effect but then greatly reduces the fragmentation form of the loop for good. LoRA adapters can be hot-swapped via /v1/load_lora_adapter.
    • The persistent loop means the engine’s error-handling has given up. On current vLLM (V1 engine) a dead GPU subprocess raises EngineDeadError; on an old pinned vLLM (before ~v0.10, when the V0 engine still existed) it was AsyncEngineDeadError: Background loop has errored already. Neither recovers in place — you restart. The practical goal is “restart without downtime” (load balancer, rolling deploy), not “never restart”.

    Three different “vLLM CUDA OOM” problems

    The phrase “vLLM won’t stop throwing CUDA OOM” covers three failures with different causes and different fixes. Identify which one you have before changing anything.

    1. Startup: not enough memory for the KV cache

    The server refuses to start and prints a ValueError like:

    ValueError: To serve at least one request with the model's max seq len (40960), (5.62 GiB KV cache is needed, which is larger than the available KV cache memory (4.89 GiB). Based on the available memory, the estimated maximum model length is 35600. Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
    

    (The mismatched parenthesis after the sequence length is in vLLM’s own message.) It means: after loading the weights and reserving overhead, the memory left for the KV cache cannot hold even one request at max_model_len tokens. This is a sizing problem, fixed with init arguments — covered below.

    2. Runtime: an allocation failure under load

    The server is up and serving, then a request or a burst of them fails. There are two distinct sub-cases, and only one is graceful:

    • KV-cache block pressure. Too many concurrent sequences need more KV cache blocks than the pool has. On the V1 engine the scheduler preempts the newest sequences and later recomputes them (the default RECOMPUTE mode), so this shows up as higher latency and lower throughput, not errors. Nothing crashes.
    • Allocator OOM outside the pool. A prefill spike, activation buffers, CUDA-graph capture, or a second process on the GPU pushes transient usage past the free margin, and PyTorch raises:
    torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU 0 has a total capacity of 23.99 GiB of which 1.10 GiB is free. ... If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.
    

    Recompute preemption does not protect against this one — it only rations KV cache blocks, not activation or graph memory. An allocator OOM here can take down the EngineCore subprocess.

    3. The persistent loop: every request fails after one bad one

    One large request triggers an OOM, and from then on every request — even trivial ones — returns a 500. Three sub-causes, and they need different responses:

    • A dead engine (most common). The OOM exception escaped into the engine’s core loop and killed it. On current vLLM the GPU subprocess is gone and every request raises EngineDeadError (with EngineCore ... died messages in the logs). On an old pinned vLLM (before the V0 engine was removed around v0.10) the async background task died instead, raising AsyncEngineDeadError: Background loop has errored already. Neither is recoverable in place — the maintainers’ guidance is to restart the process. No allocator setting brings a dead engine back.
    • Allocator fragmentation. The engine is alive but PyTorch is holding reserved-but-unusable memory in fragmented pools; nvidia-smi shows free memory in aggregate while no single block is large enough. This is the case expandable_segments:True addresses.
    • Under-provisioning. The spike that OOMs keeps recurring because the server is sized past its GPU. The fix is the startup sizing covered further down, not recovery.

    Sources: vLLM issue #38516 (KV cache ValueError), vLLM issue #16118 (estimate max-model-len)

    Can you fix it without restarting? Mostly no — here is what is live

    Be clear-eyed about what is actually adjustable while the server runs.

    Init-only — changing these means restarting vllm serve: gpu_memory_utilization, max_model_len, max_num_seqs, max_num_batched_tokens, kv_cache_dtype, enforce_eager, tensor_parallel_size, pipeline_parallel_size, quantization, swap_space. These are all EngineArgs, baked in when the engine starts.

    Live, right now, no restart:

    • Client-side pressure. The only real serving-side lever on a running engine is sending less: lower client concurrency, cap max_tokens per request, shorten prompts. This immediately reduces peak KV cache and activation memory and will pull a server back from the edge.
    • LoRA adapters. If the pressure comes from loading many fine-tunes, start with VLLM_ALLOW_RUNTIME_LORA_UPDATING=True and add or drop adapters through POST /v1/load_lora_adapter and POST /v1/unload_lora_adapter, with no interruption to in-flight traffic. vLLM’s docs warn against enabling this outside a trusted, isolated environment, since it can load arbitrary adapters.

    One restart, then durable:

    • PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True — an environment variable read by PyTorch’s CUDA allocator, not by vLLM, so it is picked up on the next process start. It switches the allocator to growable segments that resist fragmentation, and the PyTorch OOM message itself recommends it. After that one restart it greatly reduces the fragmentation form of the loop without needing further restarts.

    Automatic, nothing to set:

    • KV-cache preemption. The V1 scheduler’s default RECOMPUTE preemption lets the server ration KV cache blocks under load instead of erroring. It does not cover allocator OOMs from activation or graph memory. This is automatic on current vLLM, which ships only the V1 engine.

    Sources: vLLM optimization docs, vLLM conserving memory docs

    Fixing the startup KV-cache OOM

    For the startup ValueError, the goal is to make the KV cache fit. In rough order of preference:

    • Lower --max-model-len to what your workload actually needs. vLLM prints an estimate (“estimated maximum model length is 35600”) — use that as a ceiling. The startup check requires enough KV cache memory to serve one request at the full max_model_len; a context window you never use just raises that bar.
    • Raise --gpu-memory-utilization if the GPU is dedicated to this server. The default is 0.9; 0.920.95 is safe on a card doing nothing else. Do not go to 1.0 — PyTorch’s allocator and prefill spikes need headroom.
    • Compress the KV cache with --kv-cache-dtype fp8 (or fp8_e4m3 / fp8_e5m2). This halves KV cache bytes versus fp16 for a small, usually acceptable quality cost, and often single-handedly resolves the error.
    • Cap concurrency with --max-num-seqs (and --max-num-batched-tokens). Fewer simultaneous sequences means a smaller KV cache reservation.
    • Shard the model with --tensor-parallel-size N (splits weights across N GPUs, freeing room for KV cache on each) or --pipeline-parallel-size N (splits layers).
    • Quantize the weights (AWQ, GPTQ, FP8) so the model itself occupies less, leaving more for the cache.
    vllm serve meta-llama/Llama-3.1-8B-Instruct \
      --max-model-len 16384 \
      --gpu-memory-utilization 0.92 \
      --kv-cache-dtype fp8 \
      --max-num-seqs 64
    

    Sources: vLLM optimization docs, vLLM issue #16118

    Fixing runtime OOM under load

    For torch.OutOfMemoryError while serving, the KV cache pool is sized fine but peak usage overflows the margin around it.

    • Lower --gpu-memory-utilization to 0.850.90. Counter-intuitively, giving vLLM less of the card leaves more raw headroom for the transient allocations — prefill, activations, CUDA graphs — that live outside the reserved pool. On a shared host, 0.80.
    • Add --enforce-eager. CUDA-graph capture holds a few hundred MB of static memory and can spike while capturing. Disabling it (at a modest decode-throughput cost) reclaims that room. If you want graphs but less memory, shrink compilation_config.cudagraph_capture_sizes instead.
    • Cap --max-num-seqs / --max-num-batched-tokens. These bound how many sequences and how much prefill can be in flight at once, the main driver of runtime spikes.
    • Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True before launch so a spike that is recovered does not strand fragmented memory behind it.
    • Check for other processes on the GPU (nvidia-smi) — a second CUDA process that grabs memory after vLLM’s pre-allocation is a classic cause of “it worked yesterday” OOMs.

    Watch the distinction drawn earlier: if you are only seeing latency climb and throughput sag with no errors, that is KV-cache preemption doing its job under load, and the answer is to reduce traffic or resize the deployment rather than to keep tuning the allocator margin.

    Sources: vLLM optimization docs, vLLM OOM root-cause diagnosis

    When every request fails after one bad one

    When every request fails after one bad one, work through this in order:

    1. Check whether the engine is dead. Grep the logs for EngineDeadError and EngineCore ... died, or AsyncEngineDeadError: Background loop has errored already on an old pinned vLLM. If you see either, the engine loop is gone and no in-place setting revives it. Restart the server — and make that restart cheap: put vLLM behind a load balancer or a Kubernetes Deployment with a readiness probe so a rolling restart drains and replaces the instance without dropping traffic. “Without restarting” is the wrong goal here; “restart without downtime” is the achievable one.
    2. If the engine is alive but every request still OOMs, suspect fragmentation. Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and restart once. With expandable segments PyTorch grows and shrinks one pool instead of stranding memory in fragmented blocks, which greatly reduces this form of the loop. After that restart you should not need another for fragmentation.
    3. Make sure the V0 engine is not pinned. Current vLLM only has the V1 engine, whose GPU-subprocess isolation keeps a crash from propagating straight into the request handler. If an old deployment still sets VLLM_USE_V1=0, upgrading off V0 removes the AsyncEngineDeadError failure mode entirely.
    4. If it keeps recurring, you are under-provisioned. Apply the startup-sizing fixes from the earlier section (lower max_model_len, fp8 KV cache, lower max_num_seqs) so the spike never happens, rather than recovering from it again and again.

    Sources: vLLM GPU OOM CrashLoopBackOff runbook

    Preventing it: a starting configuration

    A starting point that avoids most OOM incidents on a single dedicated GPU:

    export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
    
    vllm serve <model> \
      --max-model-len <your real p99 context, not the model max> \
      --max-num-seqs <your real concurrency ceiling> \
      --gpu-memory-utilization 0.90 \
      --kv-cache-dtype fp8
    

    The principle: size max_model_len and max_num_seqs to the workload you actually have, not the model’s maximums; keep gpu_memory_utilization around 0.90 so there is headroom outside the pool; turn on fp8 KV cache unless you have measured a quality regression; and set the allocator env var so a recovered spike leaves less fragmentation behind. Load-test at your real concurrency before production — the KV cache math depends on numbers you only know at runtime.

    Sources: vLLM optimization docs

    Closing thoughts

    The honest version of “solve the vLLM OOM loop without restarting the server” is: prevent the OOM with correct init sizing, set expandable_segments:True once so fragmentation is far less likely to make it persistent, and lean on the V1 engine’s subprocess isolation and recompute preemption so KV-cache pressure costs you latency instead of a wedged server.

    What you genuinely do live is narrow — client-side load reduction now, LoRA hot-swap, and allocator config that applies on the next start. Everything else is an engine-init argument, so the practical target is not “never restart” but “restart without downtime”: a load balancer or a rolling Kubernetes deployment turns a required restart into a non-event. Chasing in-place recovery of a dead engine — EngineDeadError, or AsyncEngineDeadError on an old pinned V0 — is effort better spent on sizing the server so it does not die.

    Frequently Asked Questions

    How do I fix “CUDA out of memory” in vLLM?

    First identify which OOM it is. A startup ValueError: To serve at least one request ... is a KV-cache sizing problem: lower --max-model-len, raise --gpu-memory-utilization, or add --kv-cache-dtype fp8. A torch.OutOfMemoryError while serving is a headroom problem: lower --gpu-memory-utilization to 0.85, add --enforce-eager, cap --max-num-seqs, and set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. All of those except the env var require restarting the server.

    What does “To serve at least one request with the model’s max seq len” mean?

    After loading weights and reserving overhead, the memory left for the KV cache cannot hold a single request at max_model_len tokens. vLLM prints an estimated maximum model length; set --max-model-len at or below it, raise --gpu-memory-utilization, or use --kv-cache-dtype fp8 to make each token’s KV entry smaller.

    Can I change gpu_memory_utilization without restarting vLLM?

    No. It is an engine-init argument, fixed when vllm serve starts. The same is true of max_model_len, max_num_seqs, kv_cache_dtype, enforce_eager, and tensor_parallel_size. Changing any of them means a restart.

    What is a good gpu_memory_utilization value?

    0.90 on a dedicated GPU, 0.920.95 if the card is doing nothing else, 0.800.85 on a shared host or if you get runtime OOMs. Never 1.0 — PyTorch’s allocator and prefill spikes need memory outside vLLM’s reserved pool. The default is 0.9.

    Does --swap-space help with OOM on the vLLM V1 engine?

    Effectively no. V1’s default preemption mode is RECOMPUTE, and V1 dropped host-memory swap for the KV cache, so --swap-space does not offload blocks to CPU the way it did on V0. Rely on recompute preemption and correct sizing instead.

    How do I stop vLLM OOM errors from persisting across every request?

    First check the logs: EngineDeadError (or AsyncEngineDeadError: Background loop has errored already on an old pinned vLLM) means the engine loop is dead and only a restart fixes it. If the engine is alive but every request still OOMs, set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and restart once — that greatly reduces the allocator-fragmentation form of the loop. Current vLLM’s recompute preemption also keeps sustained KV-cache pressure from escalating into a crash.

    Does vLLM restart itself after a CUDA OOM crash?

    No. A dead engine (EngineDeadError, or AsyncEngineDeadError on old pinned vLLM) stays dead until the process is restarted externally. Run vLLM behind a load balancer or a Kubernetes Deployment with a readiness probe so that restart drains and replaces the instance without dropping traffic.

  • Fixing LangGraph GraphRecursionError and LangChain Agent Infinite Loops

    Fixing LangGraph GraphRecursionError and LangChain Agent Infinite Loops

    TL;DR

    • GraphRecursionError: Recursion limit of N reached without hitting a stop condition means a custom LangGraph run executed N super-steps without any node routing to END. N is whatever recursion_limit is set to for that run.
    • The familiar 25 is the LangGraph JS default and was the LangGraph Python default before v1.0.6. Current LangGraph Python defaults much higher (the docs say 1000; the source constant is 10007), so if you are on recent Python and still see a small number, a template, langgraph-cli, or your own config set it.
    • The prebuilt create_react_agent does not raise on this condition. When its step budget runs low it returns an AIMessage reading Sorry, need more steps to process this request. instead, so “the agent quietly gave up” and GraphRecursionError are two faces of the same loop.
    • Diagnose by streaming with stream_mode="updates" (or a LangSmith trace) to see which node repeats, then fix the routing: make the conditional edge depend on state that actually advances, add a path to END, or cap steps gracefully with the RemainingSteps managed value.
    • Legacy LangChain AgentExecutor loops have their own signatures: a silent max_iterations cutoff or an OutputParserException retry storm (Could not parse LLM output: / ... is not a valid tool, try another one.). Same idea, different knobs, covered at the end.

    GraphRecursionError: what “recursion limit reached” actually means

    When a LangGraph graph runs, execution advances in super-steps. A super-step is one round of the Pregel loop: every node with input available runs, and nodes that run in parallel in the same round count as a single super-step. recursion_limit is the ceiling on super-steps for a run. When the loop reaches it without any node handing control to END, LangGraph raises:

    langgraph.errors.GraphRecursionError: Recursion limit of N reached without hitting a stop condition. You can increase the limit by setting the `recursion_limit` config key.
    

    N in that message is your configured limit, not a fixed constant. Where the well-known 25 comes from:

    • LangGraph JS still defaults to 25.
    • LangGraph Python before v1.0.6 defaulted to 25 as well.
    • LangGraph Python from v1.0.6 on raised the default sharply. The applied runtime value is the source constant DEFAULT_RECURSION_LIMIT (currently 10007, overridable with the LANGGRAPH_DEFAULT_RECURSION_LIMIT environment variable); the docs round it down to “1000 steps.” That exact number is an internal detail and has shifted between releases, but it is firmly in the thousands, not 25.

    So a small N on recent LangGraph Python is a signal in itself: a tutorial snippet, langgraph-cli, a project template, or an explicit {"recursion_limit": ...} in your own code set it. Search your codebase for recursion_limit before assuming the framework picked the number.

    Whatever N is, the limit is a circuit breaker, not the bug. Hitting it tells you one of two things is true:

    • the graph legitimately needs more than N super-steps for this input (plausible when N is 25; almost never when N is 1000+), or
    • the graph is cycling: a node, or a small group of nodes, keeps handing control back and forth and the state that should move the run toward a stop condition never changes.

    Agents built with create_react_agent from langgraph.prebuilt are a two-node loop: the model node proposes tool calls, the tool node runs them, control returns to the model. If the model keeps asking for tools and never returns a final answer, the loop still has to be stopped, but the prebuilt agent stops it for you. Its default state carries a remaining_steps value, derived from recursion_limit minus the current step. After each model response, _are_more_steps_needed() checks whether that response still wants tools and how many steps are left; when only one or two remain, the agent returns AIMessage("Sorry, need more steps to process this request.") instead of proceeding. Because remaining_steps tracks recursion_limit, raising the limit never converts that message into a GraphRecursionError — you only get the exception from the prebuilt agent if you pass a custom state schema that omits remaining_steps. When recursion_limit is small (JS, older Python, or an explicit low setting) you hit this quickly and see the message; when it is in the thousands, the loop burns that many model calls first, so you usually notice the cost and latency before the message ever appears.

    Sources: GRAPH_RECURSION_LIMIT, LangGraph Graph API

    Step 1 — Unblock the run (and when raising the limit is legitimate)

    First find out what your limit actually is and what set it. recursion_limit is a run-level config value, not a graph-construction argument, so grep for it in your own code, your templates, and any langgraph-cli config. If you are on recent LangGraph Python and the number in the error is small, that search is where the fix usually is — remove the low override, or set it deliberately.

    Set it explicitly when you invoke or stream:

    # per invocation
    result = graph.invoke(inputs, {"recursion_limit": 50})
    
    # streaming
    for chunk in graph.stream(inputs, {"recursion_limit": 50}):
        ...
    
    # bind it once to a prebuilt agent
    agent = create_react_agent(model, tools).with_config({"recursion_limit": 50})
    

    Sizing it: one tool call costs two super-steps (the model node, then the tool node). Count the tool calls a successful run makes, multiply by two for super-steps, then double again for retries and reflection. An agent that normally makes ten tool calls does ~20 super-steps of useful work, so recursion_limit around 40 leaves headroom without hiding a runaway; a fixed three-step pipeline runs fine with 10. A tight explicit limit like this is worth keeping on LangGraph Python too — the ~1000/10007 default is so high that a real loop wastes thousands of model calls before it trips.

    Do not reach for a huge number to make the error disappear. If you cannot explain why the run needs that many super-steps, a bigger limit only buys a longer, more expensive failure. Diagnose first.

    Sources: GRAPH_RECURSION_LIMIT

    Step 2 — Find the node that is looping

    The fastest way to see a loop is to print what each super-step does. Stream with stream_mode="updates", which yields one {node_name: state_delta} dict per node execution:

    for step in graph.stream(inputs, stream_mode="updates"):
        for node, delta in step.items():
            print(node, "->", delta)
    

    In a healthy run the node names advance and each delta carries new information. In a loop you see the same node (or the same short A -> B -> A sequence) repeating, and the deltas are empty, identical, or oscillating between two values. That node, and the edge that routes back into it, is where the bug is.

    A few more ways to narrow it down:

    • stream_mode="debug" emits task and task_result events with the full input and output of every node, which is useful when the delta alone does not explain the routing decision.
    • A LangSmith trace renders the run as a tree; a cycle shows up as an obviously repeating branch. This is the least effort option if tracing is already enabled.
    • Inside any node you can read config["metadata"]["langgraph_step"] to know which super-step you are on, which is handy for a targeted print or breakpoint once you know roughly where the loop is.

    Sources: LangGraph Graph API

    Step 3 — Fix the actual cause

    Once you know which node repeats, the cause is almost always one of these.

    The router never advances

    A conditional edge decides the next node from a field in state. If the node that is supposed to update that field does not (a missing return key, an overwritten reducer, a typo), the router sees the same value forever and keeps choosing the same branch.

    def route(state: State) -> str:
        # loops forever if `state["status"]` is never set to "done" by any node
        return END if state["status"] == "done" else "worker"
    

    Fix it by making sure the node writes the field the router reads, and by routing on something that measurably progresses (a counter, a shrinking work queue, a done flag that a node actually sets).

    There is no path to END

    Every cyclic graph needs at least one edge, usually a conditional one, that can reach END. If every branch leads back into the cycle, the only exit is the recursion limit. Add the terminating condition explicitly.

    create_react_agent keeps asking for tools

    The symptom is usually the Sorry, need more steps to process this request. message (a higher recursion_limit only makes the run longer and more expensive before it returns that same message), because the model keeps emitting tool calls instead of a final answer. Common reasons: a tool always raises and returns an unhelpful error string, so the model retries it; the tool docstring does not say what the result means or when the task is complete; the system prompt never tells the model to answer directly once it has enough information. Fix the tool’s description and its error/return payload, add an explicit “when you have the answer, respond without calling a tool” instruction, raise recursion_limit only if the task genuinely needs more calls, and deduplicate repeated calls (below).

    Two nodes ping-pong

    Unconditional A -> B and B -> A edges are an infinite loop by construction. One of the two edges has to be conditional and able to leave the cycle.

    Cap steps gracefully with RemainingSteps

    To return a partial answer instead of crashing, add the RemainingSteps managed value to your state. LangGraph populates it with how many super-steps are left before the limit, so a node can bail out early:

    from langgraph.managed import RemainingSteps
    from typing import Annotated, TypedDict
    from operator import add
    
    class State(TypedDict):
        messages: Annotated[list, add]
        remaining_steps: RemainingSteps
    
    def worker(state: State):
        if state["remaining_steps"] <= 2:
            return {"messages": [("assistant", "Stopping early with a partial result.")]}
        ...
    

    Break out from inside a node with Command

    A node (or a tool in LangGraph) can return a Command to both update state and jump straight to a terminal node, which is useful when a node detects a cancellation or an unrecoverable condition mid-run:

    from langgraph.graph import END
    from langgraph.types import Command
    
    def guard(state: State) -> Command:
        if state.get("cancelled"):
            return Command(goto=END, update={"messages": [("assistant", "Cancelled.")]})
        return Command(goto="worker")
    

    Sources: LangGraph Graph API, ReAct Agent doesn't throw GraphRecursionError

    Loop-detection guard: stop repeated tool calls

    Most runaway agents repeat the same action. A small amount of state plus one check catches that before the recursion limit does. Record a signature of each tool call and stop when it repeats consecutively:

    import json
    
    def tool_guard(state: State):
        calls = state.get("recent_calls", [])
        last = state["messages"][-1]
        sig = None
        if getattr(last, "tool_calls", None):
            tc = last.tool_calls[0]
            sig = json.dumps([tc["name"], tc["args"]], sort_keys=True)
    
        if sig and calls[-2:] == [sig, sig]:          # 3rd identical call in a row
            return {"messages": [("assistant",
                                  "Repeated the same tool call three times; stopping to summarize.")],
                    "route": "summarize"}
        return {"recent_calls": (calls + [sig])[-5:] if sig else calls}
    

    Wire it in as a node between the model and the tools, or as pre_model_hook / middleware if you are on a framework version that supports it. The point is the same: detect “no progress” (identical payloads, repeated stderr, the same observation N times) and force a stop, summarize, propose alternatives turn instead of another identical step.

    Sources: LangGraph Graph API

    Legacy LangChain AgentExecutor loops

    The pre-LangGraph agent runtime, AgentExecutor (now shipped in langchain / langchain-classic), has its own loop controls and its own failure signatures.

    Execution caps

    • max_iterations limits intermediate steps and defaults to 15. Setting it to None removes the cap entirely, which is what turns a misbehaving agent into an unbounded one.
    • max_execution_time (default None) is a wall-clock limit in seconds.
    • early_stopping_method (default "force") controls what happens when a cap is hit. "force" returns a fixed response, Agent stopped due to iteration limit or time limit. The API also documents "generate" (one more LLM call to synthesize an answer from the steps gathered so far), but several LangChain versions raise a ValueError about an unsupported early_stopping_method when you actually pass it, so treat "force" as the value you can rely on.
    agent_executor = AgentExecutor(
        agent=agent,
        tools=tools,
        max_iterations=15,
        max_execution_time=60,
        early_stopping_method="force",
        handle_parsing_errors=True,
    )
    

    ReAct parsing deadlocks

    A ReAct agent ends a run by emitting an exact string (historically a Final Answer: line). If the model’s format drifts, the output parser raises OutputParserException: Could not parse LLM output:. With handle_parsing_errors=True the executor feeds the error back and asks the model to reformat; a model that never produces a parseable answer will do this until max_iterations cuts it off. Tightening the format instructions in the prompt is the real fix; handle_parsing_errors is a seatbelt.

    The “not a valid tool” loop

    A recurring report: an agent calls a tool name that is not registered, gets back ... is not a valid tool, try another one., and its next thought decides it still needs that same tool. It calls the invalid tool again, gets the same observation, and repeats until it gives up with “I don’t know”. One documented case hit this while inspecting a SQL database (repeated list_tables_sql_db calls on LangChain 0.0.215 / Python 3.10.11); a related case appeared when an agent was wrapped as a tool and loaded into a second agent, after which every tool call in the outer agent came back invalid. The mechanism is the same as the modern create_react_agent case: a bad observation the model is not able to recover from, repeated because nothing stops it.

    New code should prefer LangGraph or create_react_agent over AgentExecutor, but the diagnosis is identical: find the step that repeats, then remove the reason it repeats.

    Sources: AgentExecutor reference, list_tables_sql_db is not a valid tool, try another one., {tool_name} is not a valid tool, try another one.

    Closing thoughts

    recursion_limit and max_iterations are financial circuit breakers. They keep a broken run from getting expensive, but a run that hits them is telling you the graph has no reliable path to a stop condition for that input.

    The durable fix is a graph whose state moves monotonically toward termination: every cycle has a conditional edge that can reach END, the field that edge checks is written by a node on every pass, and “no progress” is itself a terminal condition rather than something you wait out. Instrument the run with stream_mode="updates" or LangSmith so a loop is visible in seconds, add RemainingSteps so the graph degrades to a partial answer instead of an exception, and add a small duplicate-call guard so the common “same action forever” failure is caught early. Keep an explicit, tight recursion_limit regardless of platform, since the current LangGraph Python default is high enough to let a loop run for thousands of steps. With those in place, an agent that wanders into a bad path pauses and pivots instead of repeating until the limit and crashing.

    Frequently Asked Questions

    How do I fix GraphRecursionError in LangGraph?

    Decide first whether the graph is looping or just long. Stream the run with stream_mode="updates" and look for a node that repeats with no change to state. If it is a real loop, fix the routing: make the conditional edge depend on a state field that a node updates every pass, ensure some branch can reach END, or add a RemainingSteps check that returns a partial result. Only if the graph genuinely needs more steps, raise the cap for that call with graph.invoke(inputs, {"recursion_limit": 50}).

    What does “Recursion limit of N reached without hitting a stop condition” mean?

    Your LangGraph run executed N super-steps (rounds of the node-execution loop) and no node ever routed to END. N is whatever recursion_limit was set to for the run. The message almost always indicates a cycle where the state that should trigger termination never changes.

    What is the default recursion limit in LangGraph?

    It depends on the platform and version. LangGraph JS defaults to 25. LangGraph Python defaulted to 25 through v1.0.5, then raised it from v1.0.6 on — the applied value is the source constant DEFAULT_RECURSION_LIMIT (currently 10007, settable via LANGGRAPH_DEFAULT_RECURSION_LIMIT), which the docs simplify to “1000 steps.” Treat it as “in the thousands”; the exact figure is an internal detail. Parallel nodes in one round count as a single super-step, so the limit is on rounds of the loop, not total node executions.

    I am on recent LangGraph Python and still see “Recursion limit of 25” — why?

    Because 25 is no longer the Python default. Something set it: a tutorial or docs snippet, a langgraph-cli config, a project template, create_react_agent example code, or an explicit {"recursion_limit": 25} in your invoke/stream call. Grep your code and config for recursion_limit.

    Should I just increase recursion_limit?

    Only if you can explain why the run needs more super-steps. Raising it is correct for a legitimately long graph on a low limit. If you do not know why the graph loops, a higher limit just produces a slower, more expensive failure; diagnose the cycle first.

    How do I find which node is causing the loop?

    Run for step in graph.stream(inputs, stream_mode="updates"): print(step) and watch the node names. A loop shows up as the same node, or a short repeating sequence of nodes, emitting empty or identical state deltas. stream_mode="debug" and a LangSmith trace give the same picture with more detail.

    How is recursion_limit different from LangChain’s max_iterations?

    recursion_limit is a LangGraph run-config value that caps super-steps and raises GraphRecursionError when a custom graph exceeds it. max_iterations is an AgentExecutor constructor argument that caps intermediate steps in the legacy agent runtime and, on reaching the cap, returns a stopped-response string rather than raising (default 15).

    How do I return a partial result instead of raising GraphRecursionError?

    Add the RemainingSteps managed value to your state and check it inside your nodes: when remaining_steps is down to 1 or 2, return a summary or route to a dedicated “wrap up” node instead of continuing the loop. That way the graph stops itself before LangGraph’s hard limit does.

    Why does my LangGraph agent return “Sorry, need more steps to process this request.” instead of an error?

    That message comes from the prebuilt create_react_agent. Its state includes a remaining_steps budget, and when it is nearly exhausted the agent returns AIMessage("Sorry, need more steps to process this request.") rather than raising GraphRecursionError. It means the same thing as the exception: the model kept calling tools without producing a final answer. Raise recursion_limit only if the task genuinely needs more calls; otherwise diagnose the tool/model loop with stream_mode="updates".