Author: JH

  • Test-Time Compute Scaling with Process Reward Models: Mechanisms, Strategies, and Trade-offs

    Test-Time Compute Scaling with Process Reward Models: Mechanisms, Strategies, and Trade-offs

    TL;DR

    • Process Reward Models evaluate each intermediate step in a Chain-of-Thought trajectory, catching reasoning errors at their point of origin to improve accuracy.
    • Step-level generative verification leverages explicit reasoning and code verification, allowing compact models like GenPRM to outperform much larger baselines.
    • Offline compute strategies like Sleep-time Compute pre-calculate context before queries arrive, significantly reducing test-time latency and operational costs.

    Understanding Process Reward Models and Step-Level Feedback

    Test-Time Compute Scaling (also known as Inference-Time Scaling) allocates additional computational resources, measured in FLOPs, during the generation process rather than relying solely on pre-training scale to improve response accuracy. Evaluating reasoning trajectories during inference depends heavily on the feedback granularity of the verifier model.

    Outcome Reward Models (ORMs) evaluate only the final output of a completed response. Because they score solely the terminal state, ORMs suffer from credit assignment problems—frequently rewarding a generation trajectory that happened to arrive at the correct final answer despite relying on flawed intermediate logic.

    Process Reward Models (PRMs) address this credit assignment challenge by introducing step-by-step process supervision. Instead of assessing only the end answer, PRMs evaluate each individual intermediate step in a Chain-of-Thought (CoT) trajectory. This step-level feedback provides dense, fine-grained scoring across the response, catching reasoning errors at their point of origin and preventing early mistakes from compounding as the trajectory progresses.

    Landmark research by Lightman et al. (2023) established that training reward models with step-by-step process supervision significantly outperforms outcome-based supervision. This concept underpins modern reasoning paradigms more broadly: models such as OpenAI’s o1 and o3 are widely understood to achieve their gains through reinforcement-learned internal reasoning rather than an external verifier, an approach closely related to the Implicit Test-Time Compute Scaling paradigm described below.

    Recent developments highlight two distinct architectural approaches to leveraging step-level feedback and compute during inference:

    • External Search with PRMs: Using separate, dedicated PRM verifier models to score intermediate steps and steer explicit search algorithms across trajectories.
    • Implicit Test-Time Compute Scaling via Reinforcement Learning (RL): Training policy models via RL to produce longer, self-correcting internal Chain-of-Thought (CoT) tokens directly inside a single inference pass without relying on external verifier loops.

    Sources: arxiv.org, jhu.edu, youtube.com, github.io, medium.com

    Why Step-Level Generative Verification Improves Reasoning

    Building on the External Search paradigm introduced above, step-level generative verification provides a critical advancement over standard PRM verifier designs. Traditional Process Reward Models (PRMs) face structural limitations due to their reliance on scalar value predictions without leveraging the generative capabilities of language models, which restricts their process supervision, generalization capabilities, and ability to leverage Test-Time Compute Scaling. Step-level generative verification overcomes these constraints by requiring the model to perform explicit Chain-of-Thought (CoT) reasoning alongside code verification before rendering a judgment for each individual reasoning step.

    In frameworks such as GenPRM, step-level generative verification utilizes Relative Progress Estimation (RPE) and a rationale synthesis framework that integrates code verification to produce high-quality process supervision labels and explicit rationale data. This structured step-level rationale generation allows process verification to unlock Test-Time Compute Scaling. Consequently, a 1.5B GenPRM model leverages Test-Time Compute Scaling to outperform GPT-4o, while a 7B GenPRM model surpasses Qwen2.5-Math-PRM-72B on ProcessBench using only 23K training instances from the MATH dataset. Beyond basic step evaluation, generating explicit reasoning and code verification for each step enables the verifier to serve as a critic model for policy model refinement, bridging the operational gap between PRMs and critic models.

    Sources: GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning

    Execution-Based Reward Models for Code Generation Search

    While generative verification strengthens step-level scoring, code generation presents a related but distinct challenge for search-based inference. Rather than scoring intermediate reasoning steps directly like a classic PRM, search and verification mechanisms in this domain frequently generate multiple complete candidate solutions and validate each one using execution results from model-generated unit tests as an outcome-based reward signal. However, because models can produce flawed tests with high confidence, these test-based reward signals are often unreliable, degrading the overall quality of candidate verification.

    Scaling the number of unit tests provides a positive correlation with reward signal quality, yielding higher performance benefits on more challenging problems. To balance reward precision with execution efficiency during search, a dynamic scaling mechanism can adaptively adjust the quantity of generated unit tests according to problem difficulty. Utilizing CodeRM-8B—a lightweight unit test generator designed for high-quality scaling—alongside dynamic unit test scaling delivers marked accuracy improvements across multiple benchmarks, including performance gains of 18.43% for Llama3-8B and 3.42% for GPT-4o-mini on HumanEval Plus.

    Sources: Dynamic Scaling of Unit Tests for Code Reward Modeling

    Latency Overhead and Operational Cost Consequences

    Although integrating search and verification mechanisms significantly boosts accuracy, scaling compute during inference imposes severe latency overhead and high operational inference costs during active user interactions. To mitigate these execution-time constraints, offline pre-computation approaches such as Sleep-time Compute can reallocate compute demands by allowing models to “think” offline about context prior to query presentation. By anticipating user queries and pre-computing useful quantities ahead of time, this approach significantly decreases the compute required during the actual test-time phase.

    Empirical evaluations on Stateful GSM-Symbolic and Stateful AIME show that Sleep-time Compute reduces the amount of Test-Time Compute Scaling needed to achieve equivalent accuracy by approximately 5x. Furthermore, scaling the amount of Sleep-time Compute allocated offline yields direct accuracy improvements, driving gains of up to 13% on Stateful GSM-Symbolic and 18% on Stateful AIME.

    To further lower per-query operational costs, Sleep-time Compute can be amortized across multiple requests sharing the same context. In Multi-Query GSM-Symbolic—an extension incorporating multiple related queries per context—amortizing pre-computed quantities across these queries reduces the average compute cost per query by 2.5x. Analysis indicates that the predictability of incoming user queries correlates directly with the effectiveness of Sleep-time Compute in reducing test-time latency and operational overhead. Case study evidence confirms the applicability of this strategy to complex, realistic workloads, including agentic software engineering (SWE) tasks.

    Sources: Sleep-time Compute: Beyond Inference Scaling at Test-time

    Closing thoughts

    Ultimately, the evidence demonstrates that shifting from coarse outcome evaluations to step-level generative verification allows compact models to achieve superior reasoning by systematically catching intermediate errors at their point of origin. However, because scaling search and verification during live inference introduces severe latency and operational cost bottlenecks, relying solely on real-time verifications remains practically constrained. By pairing dynamic step-level feedback with offline strategies like Sleep-time Compute, systems can pre-calculate context and amortize overhead across predictable queries to drastically reduce test-time costs. In my view, the true promise of test-time scaling lies in this precise synergy—balancing fine-grained process supervision with intelligent offline compute allocation to deliver high-accuracy reasoning efficiently.

    Frequently Asked Questions

    What is the main difference between Outcome Reward Models (ORMs) and Process Reward Models (PRMs)?

    Outcome Reward Models evaluate only the final output of a completed response, which can result in rewarding flawed intermediate logic. In contrast, Process Reward Models evaluate each individual intermediate step in a Chain-of-Thought trajectory to catch reasoning errors at their point of origin.

    How does step-level generative verification improve reasoning over traditional PRMs?

    Traditional PRMs rely on scalar value predictions without using the generative capabilities of language models. Step-level generative verification overcomes this limitation by requiring explicit Chain-of-Thought reasoning alongside code verification before judging each step.

    What are two main architectural approaches to using step-level feedback during inference?

    The two main approaches are External Search with PRMs, which uses dedicated verifiers to steer explicit search algorithms, and Implicit Test-Time Compute Scaling via RL, which trains policy models to generate self-correcting internal reasoning tokens in a single pass.

    How does Sleep-time Compute help mitigate test-time latency overhead?

    Sleep-time Compute reallocates compute demands by allowing models to pre-compute context offline before a user presents a query. This reduces the test-time compute required to achieve equivalent accuracy by approximately 5x and lowers per-query costs when amortized across multiple requests.

  • Kolmogorov-Arnold Networks: Rethinking Neural Architecture via Learnable Spline Activations

    Kolmogorov-Arnold Networks: Rethinking Neural Architecture via Learnable Spline Activations

    TL;DR

    • Kolmogorov-Arnold Networks (KANs) serve as an alternative to Multi-Layer Perceptrons by placing learnable B-spline activation functions directly on network edges to model complex, non-monotonic dependencies.
    • KANs significantly enhance model interpretability in applications such as cognitive diagnosis while achieving competitive training efficiency through optimized implementations.
    • Architectural extensions like the Kurkova-Kolmogorov-Arnold Network outperform traditional MLPs and original KANs in function approximation and operator learning tasks.

    Introduction to Kolmogorov-Arnold Networks (KANs)

    Kolmogorov-Arnold Networks (KANs) were introduced in April 2024 by a team of researchers from MIT, Caltech, and Northeastern University (several affiliated with the NSF Institute for Artificial Intelligence and Fundamental Interactions): Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačić, Thomas Y. Hou, and Max Tegmark. Accepted as an oral presentation at ICLR 2025, KANs offer a fundamentally different neural network architecture intended to serve as an alternative to standard Multi-Layer Perceptrons (MLPs).

    Sources: arxiv.org, iclr.cc, arxiv.org, arxiv.org, github.com

    Mathematical Foundation and Structural Differences from MLPs

    At the core of this architectural alternative is the mathematical foundation provided by the Kolmogorov-Arnold Representation Theorem. One notable application of this foundation outside standard neural networks is in Fuzzy Cognitive Maps (FCMs), a graph-based reasoning framework where nodes represent factors and edges represent causal influence between them. Drawing upon this theorem, Kolmogorov-Arnold Fuzzy Cognitive Maps (KA-FCMs) redefine the causal transmission mechanism to overcome the limitations of standard FCMs. The standard FCM formulation relies on static scalar synaptic weights and monotonic activation functions, placing non-linearity at the nodes’ aggregation phase, which fundamentally constrains its ability to model non-monotonic causal dependencies. In contrast, the KA-FCM architecture replaces static scalar weights with learnable univariate B-spline activation functions located directly on the model edges. This shift moves the non-linearity from the node aggregation phase directly to the causal influence phase along the edges, enabling the modeling of arbitrary, non-monotonic causal relationships without increasing graph density or introducing hidden layers.

    Sources: Non-monotonic causal discovery with Kolmogorov-Arnold Fuzzy Cognitive Maps

    Model Interpretability and Efficiency Benefits

    Beyond these core mathematical properties, this shift in architecture provides significant advantages in model interpretability and efficiency. In domain-specific applications such as intelligent education and cognitive diagnosis, Kolmogorov-Arnold Networks (KANs) address the long-standing interpretability limitations of Multi-Layer Perceptrons (MLPs). Neural Cognitive Diagnosis Models rely on embeddings for students, exercises, and knowledge concepts to reveal proficiency for downstream recommendation tasks. While neural Cognitive Diagnosis Models typically outperform traditional models, their reliance on MLPs yields poor interpretability even when constrained by monotonicity assumptions. Replacing MLPs with KANs in cognitive diagnosis architectures—such as in the KAN2CD framework—enhances model interpretability through two structural designs: direct replacement of standard MLPs in existing neural Cognitive Diagnosis Models with KANs, and hierarchical combination of student, exercise, and concept embeddings through several distinct KANs before combining and learning their outputs within a unified KAN to generate final predictions. The learned structures of KANs allow these enhanced neural Cognitive Diagnosis Models to maintain the clear interpretability of traditional Cognitive Diagnosis Models while surpassing existing neural Cognitive Diagnosis Models in interpretability.

    Regarding computational efficiency, original KAN architectures present a challenge due to slow training speeds. However, modifying the implementation of original KANs accelerates training, yielding computational costs that are competitive with existing models. On four real-world datasets, efficient KAN-based models outperform traditional Cognitive Diagnosis Models and maintain a performance lead over existing neural Cognitive Diagnosis Models while retaining high interpretability and competitive training costs.

    Sources: Endowing Interpretability for Neural Cognitive Diagnosis by Efficient Kolmogorov-Arnold Networks

    Architectural Extensions and Learning Dynamics

    Building upon these interpretability and efficiency benefits, researchers have also introduced broader architectural extensions and examined their learning dynamics. Inspired by the Kolmogorov-Arnold Representation Theorem and Kurkova’s principle of using approximate representations, the Kurkova-Kolmogorov-Arnold Network (KKAN) is a two-block architecture that combines robust Multi-Layer Perceptron (MLP) based inner functions with flexible linear combinations of basis functions as outer functions. Proven to be a universal approximator, KKAN demonstrates versatility across scientific machine-learning applications, including function regression, Physics-Informed Machine Learning (PIML), and operator-learning frameworks. In benchmark results, KKANs outperform MLPs and original Kolmogorov-Arnold Networks (KANs) in function approximation and operator learning tasks, while achieving performance comparable to fully optimized MLPs for PIML.

    An analysis using information bottleneck theory provides insight into the geometric complexity and learning dynamics of these models, identifying three universal learning stages across all types of architectures: fitting, transition, and diffusion. Optimal generalization is achieved during the diffusion stage. Additionally, a strong correlation exists between geometric complexity and signal-to-noise ratio (SNR). To dynamically maintain a high SNR, self-scaled residual-based attention weights can be used, ensuring uniform convergence and prolonged learning.

    Sources: KKANs: Kurkova-Kolmogorov-Arnold Networks and Their Learning Dynamics

    Closing thoughts

    Taking these empirical and structural insights together, Kolmogorov-Arnold Networks demonstrate how relocating non-linear activation functions directly onto network edges via learnable B-splines provides a compelling alternative to traditional MLPs that fundamentally enhances how complex, non-monotonic dependencies are modeled. In my view, this paradigm shift successfully resolves the historical trade-off between interpretability and predictive power, as evidenced by clear explainability gains in cognitive diagnosis tasks and superior function approximation in hybrid extensions like KKANs. As optimized implementations continue to resolve early training inefficiencies and leverage dynamic controls like residual-based attention, KAN-based architectures demonstrate that rethinking foundational neural building blocks can yield significant gains in transparency without sacrificing performance or computational viability.

    Frequently Asked Questions

    What are Kolmogorov-Arnold Networks (KANs) and who introduced them?

    Kolmogorov-Arnold Networks (KANs) are an alternative neural network architecture to standard Multi-Layer Perceptrons (MLPs). They were introduced in April 2024 by a team of researchers from MIT, Caltech, and Northeastern University.

    How do KANs differ structurally from standard Multi-Layer Perceptrons?

    KANs replace static scalar weights with learnable univariate B-spline activation functions located directly on the model edges. This moves non-linearity from the node aggregation phase to the edges, enabling the model to represent arbitrary non-monotonic causal relationships.

    How do KANs address interpretability and efficiency challenges?

    KANs improve interpretability in domain-specific applications like cognitive diagnosis, matching the explainability of traditional models while maintaining neural performance. Additionally, modified KAN implementations accelerate training speeds to make computational costs competitive with existing models.

    What is the Kurkova-Kolmogorov-Arnold Network (KKAN)?

    The KKAN is a two-block architectural extension that combines MLP-based inner functions with flexible linear combinations of basis functions as outer functions. Proven to be a universal approximator, it outperforms standard MLPs and original KANs in function approximation and operator learning tasks.

  • Debugging Silent Context Truncation in Ollama’s OpenAI-Compatible API

    Debugging Silent Context Truncation in Ollama’s OpenAI-Compatible API

    TL;DR

    • Ollama’s OpenAI-compatible API silently trims prompt inputs that exceed the context window instead of returning an explicit error.
    • This truncation occurs because standard OpenAI API parameters like max_tokens do not configure Ollama’s input context size.
    • Developers can detect truncation using OLLAMA_DEBUG=1 and resolve it by setting PARAMETER num_ctx in a custom Modelfile or using the OLLAMA_CONTEXT_LENGTH environment variable.

    Understanding Silent Context Truncation in Ollama

    In LLM serving frameworks like Ollama, context window size represents the maximum number of tokens (comprising both the input prompt and output response) that a model can retain in memory at a given time. When applications send requests to Ollama’s OpenAI-compatible API endpoint v1/chat/completions, input sequences that exceed the model’s assigned context window trigger silent context truncation. Instead of throwing an explicit error, Ollama trims earlier messages or system prompts to fit within the allocated context window, which can cause subtle response quality degradation or lost instructions.

    This behavior is primarily driven by API schema discrepancies and default runtime boundaries:

    • API Schema Discrepancy: While native Ollama endpoints (/api/generate or /api/chat) accept context window configuration directly via the num_ctx parameter, the standard OpenAI-compatible API schema does not natively support num_ctx. Standard parameters like max_tokens control output generation limits rather than input context allocation. Consequently, requests routed through v1/chat/completions rely on the model’s defined context window or global server settings unless overridden in a custom model.
    • Default Context Limits: To prevent out-of-memory (OOM) errors, Ollama enforces default context window limits—traditionally 2048 or 4096 tokens, or variable limits based on available GPU VRAM. Even if a model natively supports a larger context window (such as 32k or 128k tokens), running it without explicit configuration binds it to a smaller runtime default.

    Several diagnostic tools can be used to inspect active context window bounds and detect truncation events:

    • Server Debug Mode: Launching the server with OLLAMA_DEBUG=1 ollama serve enables verbose logging. The server logs will explicitly flag truncation with messages such as truncating input prompt or truncating input messages which exceed context length.
    • Runtime Verification: Running ollama ps in the CLI shows currently loaded models alongside their actively allocated memory bounds under the CONTEXT column.
    • Modelfile Inspection: Running ollama show --modelfile <model_name> inspects a model’s base configuration to verify whether a default PARAMETER num_ctx has been saved.

    To adjust and expand the active context window for OpenAI-compatible API client integrations, context window size can be configured via a custom model Modelfile or a server-wide environment variable.

    Creating a custom model via a Modelfile is the recommended method for OpenAI-compatible API compatibility:

    FROM llama3.2
    PARAMETER num_ctx 16384
    

    After building the model with ollama create my-custom-model, specify "model": "my-custom-model" in v1/chat/completions API calls.

    Alternatively, setting the OLLAMA_CONTEXT_LENGTH environment variable before starting the server establishes a global context window default for loaded models:

    OLLAMA_CONTEXT_LENGTH=32768 ollama serve
    

    For native Ollama endpoints (/api/generate, /api/chat), runtime context window configurations resolve according to the following priority hierarchy — note this num_ctx request parameter is NOT available on the OpenAI-compatible v1/chat/completions endpoint, which can only be configured via the Modelfile or environment variable below:

    Native API Request Parameters (num_ctx, native endpoints only) > Modelfile (PARAMETER num_ctx) > Environment Variable (OLLAMA_CONTEXT_LENGTH) > Default Allocation

    Sources: ollama.com, serverman.co.uk, medium.com, reddit.com, ollama.com

    Closing thoughts

    Ultimately, while Ollama’s OpenAI-compatible API offers seamless integration, its conservative runtime defaults create a subtle trap where longer prompts are quietly trimmed without raising explicit errors. Relying on out-of-the-box settings is risky because standard OpenAI parameters like max_tokens affect output generation rather than expanding Ollama’s input context window. In my view, explicitly configuring context limits via custom Modelfiles or the OLLAMA_CONTEXT_LENGTH environment variable should be treated as a mandatory setup step for any production v1/chat/completions integration. Actively navigating this configuration hierarchy and validating allocated limits through debug tools is essential to prevent invisible prompt degradation while maintaining system stability.

    Frequently Asked Questions

    What is silent context truncation in Ollama?

    Silent context truncation occurs when input sequences exceed a model’s allocated context window size. Instead of throwing an explicit error, Ollama trims earlier messages or system prompts to fit within the available context window.

    Why does silent context truncation happen with Ollama’s OpenAI-compatible API?

    The standard OpenAI-compatible API schema does not natively support Ollama’s num_ctx configuration parameter, and parameters like max_tokens only restrict output length. Consequently, requests default to conservative server or runtime context limits unless overridden.

    How can I detect if context truncation is occurring in Ollama?

    You can start the server with OLLAMA_DEBUG=1 to view verbose logs that explicitly flag truncation events. Additionally, running ollama ps shows loaded context bounds, and ollama show –modelfile displays default model parameters.

    How do I expand the context window size for OpenAI-compatible API calls in Ollama?

    You can define PARAMETER num_ctx in a custom model’s Modelfile or set the OLLAMA_CONTEXT_LENGTH environment variable before starting the Ollama server.

  • vLLM CUDA Out of Memory: Fixing Startup, Runtime, and Persistent OOM Loops

    vLLM CUDA Out of Memory: Fixing Startup, Runtime, and Persistent OOM Loops

    TL;DR

    • “vLLM CUDA OOM” is three different problems: a startup failure (ValueError: To serve at least one request with the model's max seq len ...), a runtime failure under load (torch.OutOfMemoryError: CUDA out of memory. Tried to allocate ...), and a persistent loop where every request 500s after one bad one.
    • Almost every memory knob in vLLM — gpu_memory_utilization, max_model_len, max_num_seqs, kv_cache_dtype, enforce_eager, tensor_parallel_size — is an engine-init argument. You cannot change it on a running server, so most real fixes require a restart.
    • The only truly live lever is client-side: send fewer concurrent requests, smaller max_tokens, shorter prompts. PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True needs one restart to take effect but then greatly reduces the fragmentation form of the loop for good. LoRA adapters can be hot-swapped via /v1/load_lora_adapter.
    • The persistent loop means the engine’s error-handling has given up. On current vLLM (V1 engine) a dead GPU subprocess raises EngineDeadError; on an old pinned vLLM (before ~v0.10, when the V0 engine still existed) it was AsyncEngineDeadError: Background loop has errored already. Neither recovers in place — you restart. The practical goal is “restart without downtime” (load balancer, rolling deploy), not “never restart”.

    Three different “vLLM CUDA OOM” problems

    The phrase “vLLM won’t stop throwing CUDA OOM” covers three failures with different causes and different fixes. Identify which one you have before changing anything.

    1. Startup: not enough memory for the KV cache

    The server refuses to start and prints a ValueError like:

    ValueError: To serve at least one request with the model's max seq len (40960), (5.62 GiB KV cache is needed, which is larger than the available KV cache memory (4.89 GiB). Based on the available memory, the estimated maximum model length is 35600. Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
    

    (The mismatched parenthesis after the sequence length is in vLLM’s own message.) It means: after loading the weights and reserving overhead, the memory left for the KV cache cannot hold even one request at max_model_len tokens. This is a sizing problem, fixed with init arguments — covered below.

    2. Runtime: an allocation failure under load

    The server is up and serving, then a request or a burst of them fails. There are two distinct sub-cases, and only one is graceful:

    • KV-cache block pressure. Too many concurrent sequences need more KV cache blocks than the pool has. On the V1 engine the scheduler preempts the newest sequences and later recomputes them (the default RECOMPUTE mode), so this shows up as higher latency and lower throughput, not errors. Nothing crashes.
    • Allocator OOM outside the pool. A prefill spike, activation buffers, CUDA-graph capture, or a second process on the GPU pushes transient usage past the free margin, and PyTorch raises:
    torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU 0 has a total capacity of 23.99 GiB of which 1.10 GiB is free. ... If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.
    

    Recompute preemption does not protect against this one — it only rations KV cache blocks, not activation or graph memory. An allocator OOM here can take down the EngineCore subprocess.

    3. The persistent loop: every request fails after one bad one

    One large request triggers an OOM, and from then on every request — even trivial ones — returns a 500. Three sub-causes, and they need different responses:

    • A dead engine (most common). The OOM exception escaped into the engine’s core loop and killed it. On current vLLM the GPU subprocess is gone and every request raises EngineDeadError (with EngineCore ... died messages in the logs). On an old pinned vLLM (before the V0 engine was removed around v0.10) the async background task died instead, raising AsyncEngineDeadError: Background loop has errored already. Neither is recoverable in place — the maintainers’ guidance is to restart the process. No allocator setting brings a dead engine back.
    • Allocator fragmentation. The engine is alive but PyTorch is holding reserved-but-unusable memory in fragmented pools; nvidia-smi shows free memory in aggregate while no single block is large enough. This is the case expandable_segments:True addresses.
    • Under-provisioning. The spike that OOMs keeps recurring because the server is sized past its GPU. The fix is the startup sizing covered further down, not recovery.

    Sources: vLLM issue #38516 (KV cache ValueError), vLLM issue #16118 (estimate max-model-len)

    Can you fix it without restarting? Mostly no — here is what is live

    Be clear-eyed about what is actually adjustable while the server runs.

    Init-only — changing these means restarting vllm serve: gpu_memory_utilization, max_model_len, max_num_seqs, max_num_batched_tokens, kv_cache_dtype, enforce_eager, tensor_parallel_size, pipeline_parallel_size, quantization, swap_space. These are all EngineArgs, baked in when the engine starts.

    Live, right now, no restart:

    • Client-side pressure. The only real serving-side lever on a running engine is sending less: lower client concurrency, cap max_tokens per request, shorten prompts. This immediately reduces peak KV cache and activation memory and will pull a server back from the edge.
    • LoRA adapters. If the pressure comes from loading many fine-tunes, start with VLLM_ALLOW_RUNTIME_LORA_UPDATING=True and add or drop adapters through POST /v1/load_lora_adapter and POST /v1/unload_lora_adapter, with no interruption to in-flight traffic. vLLM’s docs warn against enabling this outside a trusted, isolated environment, since it can load arbitrary adapters.

    One restart, then durable:

    • PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True — an environment variable read by PyTorch’s CUDA allocator, not by vLLM, so it is picked up on the next process start. It switches the allocator to growable segments that resist fragmentation, and the PyTorch OOM message itself recommends it. After that one restart it greatly reduces the fragmentation form of the loop without needing further restarts.

    Automatic, nothing to set:

    • KV-cache preemption. The V1 scheduler’s default RECOMPUTE preemption lets the server ration KV cache blocks under load instead of erroring. It does not cover allocator OOMs from activation or graph memory. This is automatic on current vLLM, which ships only the V1 engine.

    Sources: vLLM optimization docs, vLLM conserving memory docs

    Fixing the startup KV-cache OOM

    For the startup ValueError, the goal is to make the KV cache fit. In rough order of preference:

    • Lower --max-model-len to what your workload actually needs. vLLM prints an estimate (“estimated maximum model length is 35600”) — use that as a ceiling. The startup check requires enough KV cache memory to serve one request at the full max_model_len; a context window you never use just raises that bar.
    • Raise --gpu-memory-utilization if the GPU is dedicated to this server. The default is 0.9; 0.920.95 is safe on a card doing nothing else. Do not go to 1.0 — PyTorch’s allocator and prefill spikes need headroom.
    • Compress the KV cache with --kv-cache-dtype fp8 (or fp8_e4m3 / fp8_e5m2). This halves KV cache bytes versus fp16 for a small, usually acceptable quality cost, and often single-handedly resolves the error.
    • Cap concurrency with --max-num-seqs (and --max-num-batched-tokens). Fewer simultaneous sequences means a smaller KV cache reservation.
    • Shard the model with --tensor-parallel-size N (splits weights across N GPUs, freeing room for KV cache on each) or --pipeline-parallel-size N (splits layers).
    • Quantize the weights (AWQ, GPTQ, FP8) so the model itself occupies less, leaving more for the cache.
    vllm serve meta-llama/Llama-3.1-8B-Instruct \
      --max-model-len 16384 \
      --gpu-memory-utilization 0.92 \
      --kv-cache-dtype fp8 \
      --max-num-seqs 64
    

    Sources: vLLM optimization docs, vLLM issue #16118

    Fixing runtime OOM under load

    For torch.OutOfMemoryError while serving, the KV cache pool is sized fine but peak usage overflows the margin around it.

    • Lower --gpu-memory-utilization to 0.850.90. Counter-intuitively, giving vLLM less of the card leaves more raw headroom for the transient allocations — prefill, activations, CUDA graphs — that live outside the reserved pool. On a shared host, 0.80.
    • Add --enforce-eager. CUDA-graph capture holds a few hundred MB of static memory and can spike while capturing. Disabling it (at a modest decode-throughput cost) reclaims that room. If you want graphs but less memory, shrink compilation_config.cudagraph_capture_sizes instead.
    • Cap --max-num-seqs / --max-num-batched-tokens. These bound how many sequences and how much prefill can be in flight at once, the main driver of runtime spikes.
    • Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True before launch so a spike that is recovered does not strand fragmented memory behind it.
    • Check for other processes on the GPU (nvidia-smi) — a second CUDA process that grabs memory after vLLM’s pre-allocation is a classic cause of “it worked yesterday” OOMs.

    Watch the distinction drawn earlier: if you are only seeing latency climb and throughput sag with no errors, that is KV-cache preemption doing its job under load, and the answer is to reduce traffic or resize the deployment rather than to keep tuning the allocator margin.

    Sources: vLLM optimization docs, vLLM OOM root-cause diagnosis

    When every request fails after one bad one

    When every request fails after one bad one, work through this in order:

    1. Check whether the engine is dead. Grep the logs for EngineDeadError and EngineCore ... died, or AsyncEngineDeadError: Background loop has errored already on an old pinned vLLM. If you see either, the engine loop is gone and no in-place setting revives it. Restart the server — and make that restart cheap: put vLLM behind a load balancer or a Kubernetes Deployment with a readiness probe so a rolling restart drains and replaces the instance without dropping traffic. “Without restarting” is the wrong goal here; “restart without downtime” is the achievable one.
    2. If the engine is alive but every request still OOMs, suspect fragmentation. Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and restart once. With expandable segments PyTorch grows and shrinks one pool instead of stranding memory in fragmented blocks, which greatly reduces this form of the loop. After that restart you should not need another for fragmentation.
    3. Make sure the V0 engine is not pinned. Current vLLM only has the V1 engine, whose GPU-subprocess isolation keeps a crash from propagating straight into the request handler. If an old deployment still sets VLLM_USE_V1=0, upgrading off V0 removes the AsyncEngineDeadError failure mode entirely.
    4. If it keeps recurring, you are under-provisioned. Apply the startup-sizing fixes from the earlier section (lower max_model_len, fp8 KV cache, lower max_num_seqs) so the spike never happens, rather than recovering from it again and again.

    Sources: vLLM GPU OOM CrashLoopBackOff runbook

    Preventing it: a starting configuration

    A starting point that avoids most OOM incidents on a single dedicated GPU:

    export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
    
    vllm serve <model> \
      --max-model-len <your real p99 context, not the model max> \
      --max-num-seqs <your real concurrency ceiling> \
      --gpu-memory-utilization 0.90 \
      --kv-cache-dtype fp8
    

    The principle: size max_model_len and max_num_seqs to the workload you actually have, not the model’s maximums; keep gpu_memory_utilization around 0.90 so there is headroom outside the pool; turn on fp8 KV cache unless you have measured a quality regression; and set the allocator env var so a recovered spike leaves less fragmentation behind. Load-test at your real concurrency before production — the KV cache math depends on numbers you only know at runtime.

    Sources: vLLM optimization docs

    Closing thoughts

    The honest version of “solve the vLLM OOM loop without restarting the server” is: prevent the OOM with correct init sizing, set expandable_segments:True once so fragmentation is far less likely to make it persistent, and lean on the V1 engine’s subprocess isolation and recompute preemption so KV-cache pressure costs you latency instead of a wedged server.

    What you genuinely do live is narrow — client-side load reduction now, LoRA hot-swap, and allocator config that applies on the next start. Everything else is an engine-init argument, so the practical target is not “never restart” but “restart without downtime”: a load balancer or a rolling Kubernetes deployment turns a required restart into a non-event. Chasing in-place recovery of a dead engine — EngineDeadError, or AsyncEngineDeadError on an old pinned V0 — is effort better spent on sizing the server so it does not die.

    Frequently Asked Questions

    How do I fix “CUDA out of memory” in vLLM?

    First identify which OOM it is. A startup ValueError: To serve at least one request ... is a KV-cache sizing problem: lower --max-model-len, raise --gpu-memory-utilization, or add --kv-cache-dtype fp8. A torch.OutOfMemoryError while serving is a headroom problem: lower --gpu-memory-utilization to 0.85, add --enforce-eager, cap --max-num-seqs, and set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. All of those except the env var require restarting the server.

    What does “To serve at least one request with the model’s max seq len” mean?

    After loading weights and reserving overhead, the memory left for the KV cache cannot hold a single request at max_model_len tokens. vLLM prints an estimated maximum model length; set --max-model-len at or below it, raise --gpu-memory-utilization, or use --kv-cache-dtype fp8 to make each token’s KV entry smaller.

    Can I change gpu_memory_utilization without restarting vLLM?

    No. It is an engine-init argument, fixed when vllm serve starts. The same is true of max_model_len, max_num_seqs, kv_cache_dtype, enforce_eager, and tensor_parallel_size. Changing any of them means a restart.

    What is a good gpu_memory_utilization value?

    0.90 on a dedicated GPU, 0.920.95 if the card is doing nothing else, 0.800.85 on a shared host or if you get runtime OOMs. Never 1.0 — PyTorch’s allocator and prefill spikes need memory outside vLLM’s reserved pool. The default is 0.9.

    Does --swap-space help with OOM on the vLLM V1 engine?

    Effectively no. V1’s default preemption mode is RECOMPUTE, and V1 dropped host-memory swap for the KV cache, so --swap-space does not offload blocks to CPU the way it did on V0. Rely on recompute preemption and correct sizing instead.

    How do I stop vLLM OOM errors from persisting across every request?

    First check the logs: EngineDeadError (or AsyncEngineDeadError: Background loop has errored already on an old pinned vLLM) means the engine loop is dead and only a restart fixes it. If the engine is alive but every request still OOMs, set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and restart once — that greatly reduces the allocator-fragmentation form of the loop. Current vLLM’s recompute preemption also keeps sustained KV-cache pressure from escalating into a crash.

    Does vLLM restart itself after a CUDA OOM crash?

    No. A dead engine (EngineDeadError, or AsyncEngineDeadError on old pinned vLLM) stays dead until the process is restarted externally. Run vLLM behind a load balancer or a Kubernetes Deployment with a readiness probe so that restart drains and replaces the instance without dropping traffic.

  • Fixing LangGraph GraphRecursionError and LangChain Agent Infinite Loops

    Fixing LangGraph GraphRecursionError and LangChain Agent Infinite Loops

    TL;DR

    • GraphRecursionError: Recursion limit of N reached without hitting a stop condition means a custom LangGraph run executed N super-steps without any node routing to END. N is whatever recursion_limit is set to for that run.
    • The familiar 25 is the LangGraph JS default and was the LangGraph Python default before v1.0.6. Current LangGraph Python defaults much higher (the docs say 1000; the source constant is 10007), so if you are on recent Python and still see a small number, a template, langgraph-cli, or your own config set it.
    • The prebuilt create_react_agent does not raise on this condition. When its step budget runs low it returns an AIMessage reading Sorry, need more steps to process this request. instead, so “the agent quietly gave up” and GraphRecursionError are two faces of the same loop.
    • Diagnose by streaming with stream_mode="updates" (or a LangSmith trace) to see which node repeats, then fix the routing: make the conditional edge depend on state that actually advances, add a path to END, or cap steps gracefully with the RemainingSteps managed value.
    • Legacy LangChain AgentExecutor loops have their own signatures: a silent max_iterations cutoff or an OutputParserException retry storm (Could not parse LLM output: / ... is not a valid tool, try another one.). Same idea, different knobs, covered at the end.

    GraphRecursionError: what “recursion limit reached” actually means

    When a LangGraph graph runs, execution advances in super-steps. A super-step is one round of the Pregel loop: every node with input available runs, and nodes that run in parallel in the same round count as a single super-step. recursion_limit is the ceiling on super-steps for a run. When the loop reaches it without any node handing control to END, LangGraph raises:

    langgraph.errors.GraphRecursionError: Recursion limit of N reached without hitting a stop condition. You can increase the limit by setting the `recursion_limit` config key.
    

    N in that message is your configured limit, not a fixed constant. Where the well-known 25 comes from:

    • LangGraph JS still defaults to 25.
    • LangGraph Python before v1.0.6 defaulted to 25 as well.
    • LangGraph Python from v1.0.6 on raised the default sharply. The applied runtime value is the source constant DEFAULT_RECURSION_LIMIT (currently 10007, overridable with the LANGGRAPH_DEFAULT_RECURSION_LIMIT environment variable); the docs round it down to “1000 steps.” That exact number is an internal detail and has shifted between releases, but it is firmly in the thousands, not 25.

    So a small N on recent LangGraph Python is a signal in itself: a tutorial snippet, langgraph-cli, a project template, or an explicit {"recursion_limit": ...} in your own code set it. Search your codebase for recursion_limit before assuming the framework picked the number.

    Whatever N is, the limit is a circuit breaker, not the bug. Hitting it tells you one of two things is true:

    • the graph legitimately needs more than N super-steps for this input (plausible when N is 25; almost never when N is 1000+), or
    • the graph is cycling: a node, or a small group of nodes, keeps handing control back and forth and the state that should move the run toward a stop condition never changes.

    Agents built with create_react_agent from langgraph.prebuilt are a two-node loop: the model node proposes tool calls, the tool node runs them, control returns to the model. If the model keeps asking for tools and never returns a final answer, the loop still has to be stopped, but the prebuilt agent stops it for you. Its default state carries a remaining_steps value, derived from recursion_limit minus the current step. After each model response, _are_more_steps_needed() checks whether that response still wants tools and how many steps are left; when only one or two remain, the agent returns AIMessage("Sorry, need more steps to process this request.") instead of proceeding. Because remaining_steps tracks recursion_limit, raising the limit never converts that message into a GraphRecursionError — you only get the exception from the prebuilt agent if you pass a custom state schema that omits remaining_steps. When recursion_limit is small (JS, older Python, or an explicit low setting) you hit this quickly and see the message; when it is in the thousands, the loop burns that many model calls first, so you usually notice the cost and latency before the message ever appears.

    Sources: GRAPH_RECURSION_LIMIT, LangGraph Graph API

    Step 1 — Unblock the run (and when raising the limit is legitimate)

    First find out what your limit actually is and what set it. recursion_limit is a run-level config value, not a graph-construction argument, so grep for it in your own code, your templates, and any langgraph-cli config. If you are on recent LangGraph Python and the number in the error is small, that search is where the fix usually is — remove the low override, or set it deliberately.

    Set it explicitly when you invoke or stream:

    # per invocation
    result = graph.invoke(inputs, {"recursion_limit": 50})
    
    # streaming
    for chunk in graph.stream(inputs, {"recursion_limit": 50}):
        ...
    
    # bind it once to a prebuilt agent
    agent = create_react_agent(model, tools).with_config({"recursion_limit": 50})
    

    Sizing it: one tool call costs two super-steps (the model node, then the tool node). Count the tool calls a successful run makes, multiply by two for super-steps, then double again for retries and reflection. An agent that normally makes ten tool calls does ~20 super-steps of useful work, so recursion_limit around 40 leaves headroom without hiding a runaway; a fixed three-step pipeline runs fine with 10. A tight explicit limit like this is worth keeping on LangGraph Python too — the ~1000/10007 default is so high that a real loop wastes thousands of model calls before it trips.

    Do not reach for a huge number to make the error disappear. If you cannot explain why the run needs that many super-steps, a bigger limit only buys a longer, more expensive failure. Diagnose first.

    Sources: GRAPH_RECURSION_LIMIT

    Step 2 — Find the node that is looping

    The fastest way to see a loop is to print what each super-step does. Stream with stream_mode="updates", which yields one {node_name: state_delta} dict per node execution:

    for step in graph.stream(inputs, stream_mode="updates"):
        for node, delta in step.items():
            print(node, "->", delta)
    

    In a healthy run the node names advance and each delta carries new information. In a loop you see the same node (or the same short A -> B -> A sequence) repeating, and the deltas are empty, identical, or oscillating between two values. That node, and the edge that routes back into it, is where the bug is.

    A few more ways to narrow it down:

    • stream_mode="debug" emits task and task_result events with the full input and output of every node, which is useful when the delta alone does not explain the routing decision.
    • A LangSmith trace renders the run as a tree; a cycle shows up as an obviously repeating branch. This is the least effort option if tracing is already enabled.
    • Inside any node you can read config["metadata"]["langgraph_step"] to know which super-step you are on, which is handy for a targeted print or breakpoint once you know roughly where the loop is.

    Sources: LangGraph Graph API

    Step 3 — Fix the actual cause

    Once you know which node repeats, the cause is almost always one of these.

    The router never advances

    A conditional edge decides the next node from a field in state. If the node that is supposed to update that field does not (a missing return key, an overwritten reducer, a typo), the router sees the same value forever and keeps choosing the same branch.

    def route(state: State) -> str:
        # loops forever if `state["status"]` is never set to "done" by any node
        return END if state["status"] == "done" else "worker"
    

    Fix it by making sure the node writes the field the router reads, and by routing on something that measurably progresses (a counter, a shrinking work queue, a done flag that a node actually sets).

    There is no path to END

    Every cyclic graph needs at least one edge, usually a conditional one, that can reach END. If every branch leads back into the cycle, the only exit is the recursion limit. Add the terminating condition explicitly.

    create_react_agent keeps asking for tools

    The symptom is usually the Sorry, need more steps to process this request. message (a higher recursion_limit only makes the run longer and more expensive before it returns that same message), because the model keeps emitting tool calls instead of a final answer. Common reasons: a tool always raises and returns an unhelpful error string, so the model retries it; the tool docstring does not say what the result means or when the task is complete; the system prompt never tells the model to answer directly once it has enough information. Fix the tool’s description and its error/return payload, add an explicit “when you have the answer, respond without calling a tool” instruction, raise recursion_limit only if the task genuinely needs more calls, and deduplicate repeated calls (below).

    Two nodes ping-pong

    Unconditional A -> B and B -> A edges are an infinite loop by construction. One of the two edges has to be conditional and able to leave the cycle.

    Cap steps gracefully with RemainingSteps

    To return a partial answer instead of crashing, add the RemainingSteps managed value to your state. LangGraph populates it with how many super-steps are left before the limit, so a node can bail out early:

    from langgraph.managed import RemainingSteps
    from typing import Annotated, TypedDict
    from operator import add
    
    class State(TypedDict):
        messages: Annotated[list, add]
        remaining_steps: RemainingSteps
    
    def worker(state: State):
        if state["remaining_steps"] <= 2:
            return {"messages": [("assistant", "Stopping early with a partial result.")]}
        ...
    

    Break out from inside a node with Command

    A node (or a tool in LangGraph) can return a Command to both update state and jump straight to a terminal node, which is useful when a node detects a cancellation or an unrecoverable condition mid-run:

    from langgraph.graph import END
    from langgraph.types import Command
    
    def guard(state: State) -> Command:
        if state.get("cancelled"):
            return Command(goto=END, update={"messages": [("assistant", "Cancelled.")]})
        return Command(goto="worker")
    

    Sources: LangGraph Graph API, ReAct Agent doesn't throw GraphRecursionError

    Loop-detection guard: stop repeated tool calls

    Most runaway agents repeat the same action. A small amount of state plus one check catches that before the recursion limit does. Record a signature of each tool call and stop when it repeats consecutively:

    import json
    
    def tool_guard(state: State):
        calls = state.get("recent_calls", [])
        last = state["messages"][-1]
        sig = None
        if getattr(last, "tool_calls", None):
            tc = last.tool_calls[0]
            sig = json.dumps([tc["name"], tc["args"]], sort_keys=True)
    
        if sig and calls[-2:] == [sig, sig]:          # 3rd identical call in a row
            return {"messages": [("assistant",
                                  "Repeated the same tool call three times; stopping to summarize.")],
                    "route": "summarize"}
        return {"recent_calls": (calls + [sig])[-5:] if sig else calls}
    

    Wire it in as a node between the model and the tools, or as pre_model_hook / middleware if you are on a framework version that supports it. The point is the same: detect “no progress” (identical payloads, repeated stderr, the same observation N times) and force a stop, summarize, propose alternatives turn instead of another identical step.

    Sources: LangGraph Graph API

    Legacy LangChain AgentExecutor loops

    The pre-LangGraph agent runtime, AgentExecutor (now shipped in langchain / langchain-classic), has its own loop controls and its own failure signatures.

    Execution caps

    • max_iterations limits intermediate steps and defaults to 15. Setting it to None removes the cap entirely, which is what turns a misbehaving agent into an unbounded one.
    • max_execution_time (default None) is a wall-clock limit in seconds.
    • early_stopping_method (default "force") controls what happens when a cap is hit. "force" returns a fixed response, Agent stopped due to iteration limit or time limit. The API also documents "generate" (one more LLM call to synthesize an answer from the steps gathered so far), but several LangChain versions raise a ValueError about an unsupported early_stopping_method when you actually pass it, so treat "force" as the value you can rely on.
    agent_executor = AgentExecutor(
        agent=agent,
        tools=tools,
        max_iterations=15,
        max_execution_time=60,
        early_stopping_method="force",
        handle_parsing_errors=True,
    )
    

    ReAct parsing deadlocks

    A ReAct agent ends a run by emitting an exact string (historically a Final Answer: line). If the model’s format drifts, the output parser raises OutputParserException: Could not parse LLM output:. With handle_parsing_errors=True the executor feeds the error back and asks the model to reformat; a model that never produces a parseable answer will do this until max_iterations cuts it off. Tightening the format instructions in the prompt is the real fix; handle_parsing_errors is a seatbelt.

    The “not a valid tool” loop

    A recurring report: an agent calls a tool name that is not registered, gets back ... is not a valid tool, try another one., and its next thought decides it still needs that same tool. It calls the invalid tool again, gets the same observation, and repeats until it gives up with “I don’t know”. One documented case hit this while inspecting a SQL database (repeated list_tables_sql_db calls on LangChain 0.0.215 / Python 3.10.11); a related case appeared when an agent was wrapped as a tool and loaded into a second agent, after which every tool call in the outer agent came back invalid. The mechanism is the same as the modern create_react_agent case: a bad observation the model is not able to recover from, repeated because nothing stops it.

    New code should prefer LangGraph or create_react_agent over AgentExecutor, but the diagnosis is identical: find the step that repeats, then remove the reason it repeats.

    Sources: AgentExecutor reference, list_tables_sql_db is not a valid tool, try another one., {tool_name} is not a valid tool, try another one.

    Closing thoughts

    recursion_limit and max_iterations are financial circuit breakers. They keep a broken run from getting expensive, but a run that hits them is telling you the graph has no reliable path to a stop condition for that input.

    The durable fix is a graph whose state moves monotonically toward termination: every cycle has a conditional edge that can reach END, the field that edge checks is written by a node on every pass, and “no progress” is itself a terminal condition rather than something you wait out. Instrument the run with stream_mode="updates" or LangSmith so a loop is visible in seconds, add RemainingSteps so the graph degrades to a partial answer instead of an exception, and add a small duplicate-call guard so the common “same action forever” failure is caught early. Keep an explicit, tight recursion_limit regardless of platform, since the current LangGraph Python default is high enough to let a loop run for thousands of steps. With those in place, an agent that wanders into a bad path pauses and pivots instead of repeating until the limit and crashing.

    Frequently Asked Questions

    How do I fix GraphRecursionError in LangGraph?

    Decide first whether the graph is looping or just long. Stream the run with stream_mode="updates" and look for a node that repeats with no change to state. If it is a real loop, fix the routing: make the conditional edge depend on a state field that a node updates every pass, ensure some branch can reach END, or add a RemainingSteps check that returns a partial result. Only if the graph genuinely needs more steps, raise the cap for that call with graph.invoke(inputs, {"recursion_limit": 50}).

    What does “Recursion limit of N reached without hitting a stop condition” mean?

    Your LangGraph run executed N super-steps (rounds of the node-execution loop) and no node ever routed to END. N is whatever recursion_limit was set to for the run. The message almost always indicates a cycle where the state that should trigger termination never changes.

    What is the default recursion limit in LangGraph?

    It depends on the platform and version. LangGraph JS defaults to 25. LangGraph Python defaulted to 25 through v1.0.5, then raised it from v1.0.6 on — the applied value is the source constant DEFAULT_RECURSION_LIMIT (currently 10007, settable via LANGGRAPH_DEFAULT_RECURSION_LIMIT), which the docs simplify to “1000 steps.” Treat it as “in the thousands”; the exact figure is an internal detail. Parallel nodes in one round count as a single super-step, so the limit is on rounds of the loop, not total node executions.

    I am on recent LangGraph Python and still see “Recursion limit of 25” — why?

    Because 25 is no longer the Python default. Something set it: a tutorial or docs snippet, a langgraph-cli config, a project template, create_react_agent example code, or an explicit {"recursion_limit": 25} in your invoke/stream call. Grep your code and config for recursion_limit.

    Should I just increase recursion_limit?

    Only if you can explain why the run needs more super-steps. Raising it is correct for a legitimately long graph on a low limit. If you do not know why the graph loops, a higher limit just produces a slower, more expensive failure; diagnose the cycle first.

    How do I find which node is causing the loop?

    Run for step in graph.stream(inputs, stream_mode="updates"): print(step) and watch the node names. A loop shows up as the same node, or a short repeating sequence of nodes, emitting empty or identical state deltas. stream_mode="debug" and a LangSmith trace give the same picture with more detail.

    How is recursion_limit different from LangChain’s max_iterations?

    recursion_limit is a LangGraph run-config value that caps super-steps and raises GraphRecursionError when a custom graph exceeds it. max_iterations is an AgentExecutor constructor argument that caps intermediate steps in the legacy agent runtime and, on reaching the cap, returns a stopped-response string rather than raising (default 15).

    How do I return a partial result instead of raising GraphRecursionError?

    Add the RemainingSteps managed value to your state and check it inside your nodes: when remaining_steps is down to 1 or 2, return a summary or route to a dedicated “wrap up” node instead of continuing the loop. That way the graph stops itself before LangGraph’s hard limit does.

    Why does my LangGraph agent return “Sorry, need more steps to process this request.” instead of an error?

    That message comes from the prebuilt create_react_agent. Its state includes a remaining_steps budget, and when it is nearly exhausted the agent returns AIMessage("Sorry, need more steps to process this request.") rather than raising GraphRecursionError. It means the same thing as the exception: the model kept calling tools without producing a final answer. Raise recursion_limit only if the task genuinely needs more calls; otherwise diagnose the tool/model loop with stream_mode="updates".

  • Demystifying Mixture of Experts Routing: Core Mechanisms, Optimization, and Practical Challenges

    Demystifying Mixture of Experts Routing: Core Mechanisms, Optimization, and Practical Challenges

    TL;DR

    • Mixture of Experts architectures scale model capacity while keeping active computational costs low by using a gating network to route tokens to specific expert subnetworks.
    • Empirical analysis demonstrates that expert activation patterns are highly task-conditioned, with prompts from the same category producing highly similar routing signatures.
    • To resolve single-GPU memory bottlenecks, deployment strategies utilize post-training expert pruning and predictive caching systems like ExpertFlow to reduce memory usage and improve throughput.

    An Introduction to Mixture of Experts (MoE) Routing

    MoE routing represents a core architectural component in modern large language models (LLMs), serving as the foundational mechanism for conditional computation. This approach allows a model to scale its overall parameter capacity to hundreds of billions or even trillions of parameters. Crucially, it achieves this scaling while keeping the active computational cost—the number of parameters actually activated to process a given input—equivalent to that of a much smaller model.

    In a standard Transformer architecture, every input token in a sequence is processed by the exact same Feed-Forward Network (FFN) at each layer. A SparseMoE model restructures this setup by replacing that single FFN with multiple parallel subnetworks known as “experts.” To coordinate this architecture, a gating network is positioned before the expert layer. The gating network is a lightweight, trainable neural network that functions as a traffic director. For each incoming token, it determines which specific experts are most mathematically suited to handle the processing and directs the token to them accordingly.

    The standard mathematical execution of MoE routing operates through three primary steps:

    1. Compute Logits: For an incoming token representation x, the gating network computes a dot product with a learnable weight matrix W_g to produce affinity scores (logits) for each expert:
      Logits = x * W_g

    2. Top-k Routing: Rather than sending the token to all experts, which would be computationally expensive, the gating network selects only the top-k experts that yielded the highest affinity scores. In standard setups, k is typically set to 1 or 2.

    3. Softmax Normalization and Weighted Combine: The logits of the selected experts are normalized using a softmax function. Once these chosen experts process the token, their respective outputs are multiplied by these normalized gating network probabilities and summed together to produce the final representation of the token.

    Sources: huggingface.co, research.google, maartengrootendorst.com, thelmbook.com, tistory.com

    Routing Paradigms: Top-k vs. Soft Routing Mechanisms

    Building on these standard routing mechanics, alternative paradigms have emerged to optimize performance and stability. When evaluating SoftMoE and SparseMoE classifier heads under comparable model capacity, both routing paradigms achieve slightly higher validation accuracy than a dense baseline on the CIFAR10 image classification dataset. To avoid routing collapse, both variants prevent load imbalance through auxiliary loss.

    The primary distinctions between these paradigms emerge in their generalization and loss surface curvature. Hessian-based sharpness metrics at convergence—specifically the largest eigenvalue and the trace of the loss Hessian evaluated on both training and test data—show that SoftMoE exhibits higher sharpness. Meanwhile, SparseMoE and dense models lie in a similar curvature regime, despite all of these models achieving comparable generalization performance. Furthermore, loss surface perturbation analyses show qualitative differences in non-local behavior under finite parameter perturbations between dense and Mixture of Experts models.

    In terms of operational performance, there is a pronounced gap between theoretical and realized efficiency in SparseMoE models. Naively implemented MoE routing fails to yield empirical inference speedups on modern hardware at this scale, highlighting the practical challenges of deploying sparse routing mechanisms.

    Sources: Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization

    Analyzing Real-World Routing: Task-Conditioned Signatures

    To address these practical deployment challenges and gain deeper insights into how experts actually behave, researchers have introduced routing signatures as a tool to analyze whether expert selection in Mixture of Experts architectures exhibits task-conditioned structure. A routing signature is a vector representation that summarizes expert activation patterns across layers for a given prompt.

    Empirical analysis using the OLMoE-1B-7B-0125-Instruct model as a testbed demonstrates that prompts from the same task category induce highly similar routing signatures, whereas prompts from different categories have significantly lower similarity. Specifically, within-category routing similarity reaches 0.8435 +/- 0.0879, which substantially exceeds the across-category similarity of 0.6225 +/- 0.1687, representing a Cohen’s d of 1.44. The distinctiveness of these signatures is strong enough that a simple logistic regression classifier trained solely on routing signatures can achieve 92.5% +/- 6.1% cross-validated accuracy on four-way task classification.

    To confirm that this separation is a genuine reflection of task-specific routing rather than an artifact of architectural constraints, the analysis incorporated permutation and load imbalance baselines. These baselines demonstrate that the observed task-based clustering cannot be explained by sparsity or load imbalance constraints alone. Furthermore, examining layer-wise signal strength and low-dimensional projections of routing signatures reveals that this task-conditioned structure becomes increasingly apparent in the deeper layers of the model. These findings indicate that routing in sparse transformers operates as a measurable, task-sensitive component of conditional computation rather than serving merely as a load imbalance prevention mechanism. To facilitate further routing telemetry and analysis, researchers have released a lightweight toolkit named MOE-XRAY.

    Sources: Task-Conditioned Routing Signatures in Sparse Mixture-of-Experts Transformers

    Addressing Memory Bottlenecks: Expert Pruning and Predictive Caching

    Beyond analyzing routing signatures, another major hurdle in utilizing these architectures is managing their immense parameter sizes. To mitigate this heavy memory footprint, particularly in memory-constrained environments like single-GPU deployments, researchers have developed post-training sparsification techniques and predictive offloading systems.

    One primary direction to optimize deployment efficiency is pruning and skipping inactive or redundant components. Recent research has introduced post-training approaches that provide plug-and-play, expert-level sparsification through task-agnostic and task-specific expert pruning and expert skipping. Unlike historical weight pruning methods that rely on specialized hardware, these techniques prune or skip entire experts to simultaneously reduce overall model sizes and increase inference speeds while maintaining satisfactory performance across a wide range of tasks.

    Another approach to overcoming memory bottlenecks on a single GPU is dynamic offloading, where inactive experts are stored in CPU memory and loaded on demand. Because static caches disregard input-dependent routing and training separate predictor models is often inaccurate or computationally expensive, the ExpertFlow inference system coordinates three distinct components to manage routing dependencies efficiently:

    • Transformer-based routing path predictor: This component estimates expert usage across all Mixture of Experts layers in a single forward pass.
    • Token scheduler: This groups tokens with similar predicted routes to maximize expert utilization.
    • Predictive expert cache: This system loads only the necessary experts into memory while dynamically correcting any routing mispredictions at runtime.

    By coordinating these three components, ExpertFlow enables highly efficient expert loading and execution. On a single GPU, this predictive caching and scheduling pipeline reduces GPU memory usage by up to 93.72% and improves inference throughput by up to 10x compared to strong offloading baselines.

    Sources: ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling, Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models

    Closing thoughts

    Ultimately, as these optimization techniques demonstrate, the viability of Mixture of Experts architectures hinges on closing the pronounced gap between their theoretical computational elegance and the physical realities of modern hardware deployment. While naive routing struggles with empirical inference speeds, the discovery of highly structured, task-conditioned routing signatures in deeper layers demonstrates that these gating networks are executing genuine, meaningful computation rather than arbitrary load balancing. Consequently, by leveraging this predictable behavior through targeted expert pruning and dynamic predictive caching systems like ExpertFlow, developers can successfully resolve severe single-GPU memory bottlenecks. This suggests that the true maturity of MoE technology will be defined not merely by expanding model sizes, but by co-designing hardware-aware routing algorithms with robust diagnostic telemetry.

    Frequently Asked Questions

    What is a routing signature in Mixture of Experts (MoE) architectures?

    A routing signature is a vector representation that summarizes expert activation patterns across layers for a given prompt. It is used by researchers to analyze whether expert selection in MoE models exhibits task-conditioned structures.

    How does the ExpertFlow system optimize single-GPU memory and inference performance?

    ExpertFlow coordinates a Transformer-based routing path predictor, a token scheduler, and a predictive expert cache to manage routing dependencies. This pipeline reduces GPU memory usage by up to 93.72% and improves inference throughput by up to 10x compared to strong offloading baselines.

    What are the three steps involved in standard MoE routing?

    First, the gating network computes affinity scores (logits) for each expert using a dot product. Second, top-k routing selects only the experts with the highest scores, typically setting k to 1 or 2. Finally, the chosen experts’ outputs are normalized using a softmax function and summed together to produce the final representation.