Category: Practical Guides

Troubleshooting guides for LLM tooling such as LangChain, vLLM, Ollama and Hugging Face, from real error messages, citing official docs and GitHub issues.

  • Troubleshooting CrewAI ValidationError: Please Provide an OpenAI API Key

    Troubleshooting CrewAI ValidationError: Please Provide an OpenAI API Key

    TL;DR

    Enabling memory=True on a Crew triggers a Pydantic ValidationError when an OpenAI API key is absent because CrewAI’s embedder parameter defaults to {“provider”: “openai”}.

    Issue #2943 was reported on crewAI 0.121.0 and has since been closed as completed, with a linked commit titled “Fix ValidationError when using memory=True without OpenAI API key”. The sources do not say whether newer releases still reproduce it, but the OpenAI default for memory embeddings is documented across versions, so a missing key or custom embedder is the first thing to check.

    Other OpenAI-related failures, such as knowledge sources or agent and manager LLMs, are separate problems with separate fixes. Setting a Crew embedder does not resolve those.

    For the memory error, either set the OPENAI_API_KEY environment variable or pass an explicit custom embedder configuration (such as Ollama).

    The Symptom: Pydantic ValidationError on Crew Initialization with memory=True

    In issue #2943 submitted to the CrewAI repository, a developer reported an error when adding the parameter memory=True to a Crew. While the setup executed without issue with cache=True, enabling memory=True immediately triggered a validation error.

    The issue occurred during Crew instantiation:

    synergistic_crew = Crew(
        agents=[holiday_agent, macro_agent, news_summarizer_agent, analysis_forecast_agent, user_query_agent],
        tasks=[holiday_task, macro_task, news_summarizer_task, analysis_forecast_task, user_query_task],
        process=Process.sequential,
        verbose=True,
        cache=True,
        memory=True
    )
    

    Adding memory=True produced the following Pydantic validation failure:

    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    pydantic_core._pydantic_core.ValidationError: 1 validation error for Crew
      Value error, Please provide an OpenAI API key. You can get one at https://platform.openai.com/account/api-keys [type=value_error, input_value={'agents': [Agent(role=Ho...': True, 'memory': True}, input_type=dict]
        For further information visit https://errors.pydantic.dev/2.11/v/value_error
    

    The issue report listed the environment as Ubuntu 20.04, Python 3.10, Venv, crewAI version 0.121.0, and crewAI Tools version 0.45.0. Although the reporter noted using “pydantic version V1”, the traceback references pydantic_core and links to https://errors.pydantic.dev/2.11/v/value_error, indicating that Pydantic v2.11 was actively running in the runtime environment.

    The issue was later closed as completed on 2025-06-04, with a linked commit titled “Fix ValidationError when using memory=True without OpenAI API key”. The thread does not state which release contains the fix, so check your installed version and its changelog before assuming you will see the exact same error today.

    Sources: error-pydantic_core._pydantic_core.ValidationError: 1 validation error for Crew

    Why the Error Triggers: Pydantic Validators and OpenAI Default Fallbacks

    CrewAI relies on Pydantic to validate configuration parameters and dependencies when instantiating objects like Crew, Agent, and Task. When memory=True is set on a Crew, CrewAI initializes its memory system, which requires an embedding model to convert text into vector representations. According to the Crews documentation (v1.15.28), the embedder attribute on Crew defaults to {“provider”: “openai”}.

    In issue #2943 on 0.121.0, enabling memory without an alternative embedder and without an OpenAI key produced a validation error whose message asks for an OpenAI API key. The sources do not show exactly where in the Crew validators the check runs, but the traceback shows it surfacing as a Pydantic error during Crew construction:

    ValidationError: 1 validation error for Crew Value error, Please provide an OpenAI API key
    

    Versions matter here. The issue was filed on crewAI 0.121.0, the Crew attribute descriptions in this article come from the v1.15.28 docs, and the unified Memory class comes from the “edge” docs. The Crews docs describe memory in terms of short-term, long-term, and entity memory, while the edge docs describe a single unified Memory class. In both, the documented default embedder is OpenAI. Where a behavior comes from only one of these sources, the text below says so.

    Other parts of CrewAI also involve OpenAI by default or require explicit configuration. They fail at different points and have different fixes:

    • Memory Embeddings: Directly tied to Crew initialization when memory=True, triggering the reported validation error if embedder remains at its OpenAI default.
    • Agent LLMs: The “Connect to any LLM” documentation covers configuring a non-OpenAI model per agent. The fix is setting the llm parameter on the Agent, not the Crew embedder.
    • Hierarchical Process Coordination: When setting process=Process.hierarchical, the Crews documentation specifies that manager_llm or manager_agent is required. Supply one of them. The embedder setting does not affect this.
    • Knowledge Sources: Issue #2164 reports a separate “Invalid Knowledge Configuration: Please provide an OpenAI API key” error for agent knowledge. It is a different error message from the memory one, so check that issue for its own fix.

    Sources: error-pydantic_core._pydantic_core.ValidationError: 1 validation error for Crew · Issue #2943 · crewAIInc/crew, [BUG] Agent knowledge – Invalid Knowledge Configuration: Please provide an OpenAI API key. · Issue #2164 · cre, Crews – CrewAI, Memory – CrewAI, Connect to any LLM – CrewAI

    Inspecting Crew Attributes and Execution Flow Defaults

    The Crew attributes below come from the v1.15.28 Crews documentation. In issue #2943, the same crew ran with cache=True but failed once memory=True was added:

    • embedder: Configuration dictionary for the embedding provider used by the crew. It is primarily utilized by memory and defaults to {“provider”: “openai”}.
    • memory: Toggles memory systems (short-term, long-term, entity memory in earlier documentation; unified Memory in newer releases). It defaults to False.
    • cache: Toggles caching, defaulting to True.
    • process: Execution flow strategy, defaulting to Process.sequential. When set to Process.hierarchical, manager_llm or manager_agent is required.
    • verbose: Controls logging verbosity, defaulting to False.
    • max_rpm: Maximum requests per minute, defaulting to None.

    The embedder attribute is described as primarily used by memory and defaults to {“provider”: “openai”}. The reporter’s crew had cache=True throughout and the error appeared only after memory=True was added, which points at the memory setting. The sources do not explain why cache=True was unaffected, so treat any explanation of that as an inference.

    Sources: Crews – CrewAI

    The Underlying Culprit: CrewAI Unified Memory and Embedding Requirements

    In newer releases of CrewAI (as outlined in the edge documentation), the framework consolidates memory handling into a unified Memory class. When memory=True is passed, the crew creates a default Memory() instance and passes through the crew’s embedder setting.

    In this unified system, memory recall ranks results using composite scoring:

    composite = semantic_weight * similarity + recency_weight * decay + importance_weight * importance
    

    where similarity is computed from vector distances:

    similarity = 1 / (1 + distance)
    

    According to the official documentation, without an explicit embedder configuration, Memory uses OpenAI text-embedding-3-large embeddings to power these vector calculations. When memories are saved, an LLM also infers scope, categories, and importance scores.

    To resolve the ValidationError, developers have two options depending on their architecture:

    Option 1: Providing the OpenAI API Key
    If you plan to use OpenAI models or embeddings, supply the key as an environment variable before initializing your crew (ideally via a .env file or your shell environment):

    import os
    
    os.environ["OPENAI_API_KEY"] = "your-openai-api-key"
    

    Option 2: Configuring a Custom Embedder (Non-OpenAI Setup)
    If you run local or alternative models (such as Ollama) and do not want to use OpenAI, configure a custom embedder. The Crews docs list embedder as a Crew attribute, so the example below passes it to Crew. The edge Memory docs show the same kind of embedder configuration on a Memory(embedder=…) instance, with the Ollama model key named “model_name”. Check which form your installed version accepts:

    from crewai import Crew, Process
    
    crew = Crew(
        agents=[research_agent, writer_agent],
        tasks=[research_task, write_task],
        process=Process.sequential,
        memory=True,
        embedder={
            "provider": "ollama",
            "config": {
                "model_name": "mxbai-embed-large",
                "url": "http://localhost:11434/api/embeddings"
            }
        }
    )
    

    Explicitly declaring the embedder replaces the {“provider”: “openai”} default. This addresses the memory embedding error only. It does not change the llm setting on agents or manager_llm.

    Sources: Memory – CrewAI

    Closing thoughts

    Convenience flags like memory=True simplify multi-agent workflows, but they often carry implicit dependencies on default cloud providers. In CrewAI, enabling memory activates embedding operations that default to OpenAI unless overridden.

    When building workflows with alternative providers or local models, actively specifying configuration parameters—such as setting a custom embedder or defining model providers explicitly—prevents unexpected Pydantic validation errors during crew initialization.

    Frequently Asked Questions

    Why does setting memory=True trigger a validation error requesting an OpenAI API key?

    Enabling memory=True activates CrewAI’s memory system, which requires an embedding model for vector storage and retrieval. Because the embedder parameter defaults to {“provider”: “openai”}, Crew initialization raises a Pydantic ValidationError if OPENAI_API_KEY is not found in the environment.

    What default embedding model does CrewAI use for its memory system?

    The embedder parameter defaults to {“provider”: “openai”}. In CrewAI’s unified Memory documentation, the default embedding model is specified as OpenAI text-embedding-3-large.

    How can you resolve this error without an OpenAI API key?

    You can provide a custom embedder dictionary to the Crew, such as using provider ollama with an embedding model like mxbai-embed-large, or set the OPENAI_API_KEY environment variable. Check which embedder form your installed version accepts.

    Why did cache=True work in issue #2943 while memory=True failed?

    In the issue, the reporter’s crew with cache=True ran, and the error appeared only after memory=True was added. The sources do not give a reason for the difference. The Crews docs describe embedder as primarily used by memory, which is consistent with the error following memory=True.


    Keep reading: LLM Agents & Tooling

    All guides in this series: LLM Agent Guides: LangChain, LangGraph & AI Tooling

  • Troubleshooting vLLM Startup: Resolving KV Cache Capacity Exceeded Errors

    Troubleshooting vLLM Startup: Resolving KV Cache Capacity Exceeded Errors

    TL;DR

    • vLLM aborts engine initialization with a ValueError when the allocated KV cache capacity cannot accommodate the model’s maximum sequence length for at least one request.

    • The issue is commonly resolved by decreasing max_model_len, increasing gpu_memory_utilization, or applying tensor parallelism and quantization.

    • In tight-margin boundary cases, switching to the FlashInfer attention backend—either explicitly or implicitly via FP8 KV cache—has been observed to leave less available KV cache memory, triggering this error under otherwise identical settings.

    Diagnosing the KV Cache Exceeded ValueError during Engine Initialization

    During engine initialization and cache profiling, vLLM manages key-value activations in fixed-size blocks via PagedAttention. Before processing inference requests, the engine evaluates available memory and performs a sanity check: the total number of tokens that can fit into the allocated KV cache blocks must be at least as large as the configured maximum sequence length. If the KV cache capacity is smaller than that limit, the engine aborts initialization with a traceback:

    ValueError: The model's max seq len (<X>) is larger than the maximum number of tokens that can be stored in KV cache (<Y>). Try increasing gpu_memory_utilization or decreasing max_model_len when initializing the engine.
    

    This error prevents runtime Out-of-Memory (OOM) faults by ensuring the engine has enough memory to process at least a single full-context request. The available space for cache blocks is roughly calculated as:

    KV Cache Memory ≈ (Total GPU Memory * gpu_memory_utilization) - (Weights + Activations + CUDA Graphs)
    

    Two primary factors trigger this error during startup:

    • Unset or high default context window: If max_model_len (CLI flag --max-model-len) is omitted, vLLM falls back to the native context limit in the model’s Hugging Face config.json (such as max_position_embeddings, which can reach 32,768, 65,536, or 131,072+ tokens). On GPU memory-constrained hardware, the engine cannot allocate enough blocks to cover such large sequences.
    • Model weight and runtime overhead: Serving models in unquantized FP16 or BF16 formats or enabling CUDA graph capture consumes the majority of the GPU memory budget, leaving insufficient headroom for the remaining KV cache pool.

    To resolve the error during engine initialization, adjust the memory configuration via the following flags:

    vllm serve <model_name> --max-model-len 4096 --gpu-memory-utilization 0.95
    

    Setting --max-model-len to an integer lower than the reported capacity directly caps the required sequence ceiling. Raising gpu_memory_utilization (CLI flag --gpu-memory-utilization) above its default of 0.9 (for example, to 0.95) expands the engine’s reserved GPU memory quota, provided sufficient headroom remains to avoid runtime OOM spikes.

    Additional memory can be freed by distributing weights and attention heads across multiple GPUs using tensor parallelism via --tensor-parallel-size, applying model weight quantization (such as FP8, AWQ, or GPTQ), or specifying --kv-cache-dtype as fp8 (or fp8_e4m3 / fp8_e5m2 on supported architectures) to cut the per-token KV footprint by approximately 50% compared to 16-bit floats. However, operators should be aware that enabling FP8 KV cache can implicitly select the FlashInfer backend on certain GPUs, which in tight memory margin scenarios may reduce the available KV cache pool and trigger this initialization check.

    Sources: [Usage]: What's the relationship between KV cache and MAX_SEQUENCE_LENGTH. · Issue #10517 · vllm-project/vllm , worker – vLLM, docs.vllm.ai/en/latest/design/hybrid_kv_cache_manager, vLLM KV Cache OOM: The Flags That Actually Fix It, Conserving Memory – vLLM

    Resolving Hidden Memory Overhead from FLASHINFER and FP8 KV Cache Configurations

    Beyond standard sequence length and utilization tuning, a ValueError during KV cache initialization can also arise in tight-margin boundary conditions when switching the attention backend via --attention-backend, even when GPU memory, --max-model-len (max_model_len), and --gpu-memory-utilization (gpu_memory_utilization) settings remain identical. In an issue reported on the vLLM repository, serving RedHatAI/Qwen3.8-27B-INT4 (a hybrid Mamba/GDN architecture) on two 16 GB RTX 5060 Ti GPUs (PyTorch 2.13.0+cu130, flashinfer-python 0.6.18, vLLM 0.28.0 and 0.28.1rc1.dev446+g798544433) succeeded under the default FlashAttention (FLASH_ATTN) backend, creating a pool of 105,594 tokens on the development build (108,589 tokens on 0.28.0).

    However, explicitly passing --attention-backend FLASHINFER caused engine initialization to fail. Note that the reproduction command included specialized flags tailored for this hybrid model setup, such as --mamba-cache-mode align, rather than standard generic parameters:

    vllm serve RedHatAI/Qwen3.8-27B-INT4 --tensor-parallel-size 2 \
      --gpu-memory-utilization 0.90 --max-model-len 100352 --max-num-seqs 32 \
      --mamba-cache-mode align --max-num-batched-tokens 3135 \
      --attention-backend FLASHINFER
    

    This configuration failed during the memory check with the following error:

    ValueError: To serve at least one request with the model's max seq len (100352),
    3.21 GiB KV cache is needed, which is larger than the available KV cache memory
    (3.06 GiB). Try increasing `gpu_memory_utilization` ... or decreasing `max_model_len` ...
    

    On vLLM 0.28.0, the same message reported 3.15 GiB available memory instead of 3.06 GiB. Users can also encounter this issue without explicitly setting --attention-backend FLASHINFER, as setting --kv-cache-dtype fp8 implicitly selects FlashInfer on some GPUs. While the exact root cause was not definitively settled in the issue report, the reporter suggested that the FlashInfer workspace memory might not be reserved prior to calculating the KV cache pool size. Because the original error message only advised adjusting gpu_memory_utilization or max_model_len, it never informed users that the attention backend was the variable that changed.

    To address this ambiguity, a pull request has been proposed (still open) to identify the active attention backend and include that information directly within the existing ValueError.

    Sources: [Bug]: –attention-backend FLASHINFER fails the startup KV-cache check at a max-model-len that FLASH_ATTN serves; the message never names the backend, kv cache Value Erorr msg improvement Attention Backend

    Closing thoughts

    Ultimately, while trimming --max-model-len or nudging gpu_memory_utilization upward provides a quick mechanical fix, this error illustrates that KV cache headroom is governed by far more than just model weights and user-defined sequence caps. As the observed FlashInfer boundary case demonstrates, backend selection—whether configured explicitly or selected implicitly through options like FP8 KV caching—can alter the memory available for cache blocks and break razor-thin initialization budgets. Rather than reflexively sacrificing context length when this check fails, operators on memory-constrained GPUs should consider backend selection as an active variable in their memory budget. In this context, the proposed pull request to expose the active attention backend directly within the ValueError traceback represents a welcome diagnostic improvement, clarifying what variable actually impacted the available memory pool.

    Frequently Asked Questions

    Why does vLLM throw a ValueError regarding KV cache capacity during engine initialization?

    The error occurs during memory profiling if the total number of tokens that can fit into the allocated KV cache blocks is smaller than the model’s configured maximum sequence length. This sanity check aborts startup to prevent runtime Out-of-Memory faults, ensuring the engine has enough memory to handle at least a single full-context request.

    What are the primary ways to fix the KV cache ValueError?

    You can resolve the error by decreasing the maximum sequence ceiling with --max-model-len or increasing --gpu-memory-utilization (for example, to 0.95) to allocate more GPU memory. Additional remedies include enabling tensor parallelism via --tensor-parallel-size, applying model weight quantization, or setting --kv-cache-dtype to fp8 to reduce the per-token KV footprint (bearing in mind that FP8 KV cache may implicitly select FlashInfer).

    How can changing the attention backend trigger this KV cache error?

    In narrow memory margin setups, switching the attention backend to FLASHINFER has been observed to leave less available KV cache memory compared to FLASH_ATTN (suspected to stem from how backend workspace memory is reserved relative to KV cache pool sizing). This difference can cause initialization to fail even when GPU memory, --max-model-len, and --gpu-memory-utilization remain completely unchanged.

    Can the FLASHINFER backend be activated without passing --attention-backend FLASHINFER?

    Yes, setting --kv-cache-dtype fp8 implicitly selects the FlashInfer backend on some GPUs, which can alter available KV cache headroom in tight boundary cases and trigger the initialization error.


    Keep reading: LLM Inference & Serving

    All guides in this series: LLM Inference & Serving Guides: vLLM, Ollama & Hugging Face

  • Troubleshooting ChromaDB InvalidDimensionException: Resolving the 1536 vs 384 Dimension Mismatch

    Troubleshooting ChromaDB InvalidDimensionException: Resolving the 1536 vs 384 Dimension Mismatch

    TL;DR

    • ChromaDB raises an InvalidDimensionException when a collection defaults to a 384-dimensional embedder at creation but receives 1536-dimensional query vectors from models like OpenAI.
    • ChromaDB collections enforce a strict, immutable coordinate space that cannot be modified in place once initialized.
    • To prevent dimension mismatches, developers must explicitly configure the desired embedding function when creating the collection.

    Diagnosing the InvalidDimensionException When Using OpenAI Embeddings

    This exception typically manifests when a collection is populated using ChromaDB’s default embedding pipeline and subsequently queried with an external OpenAI embedding model. In a report involving ChromaDB 0.4.24 and Python 3.11.8, a user encountered this failure during retrieval:

    InvalidDimensionException: Embedding dimension 1536 does not match collection dimensionality 384
    

    The underlying issue stems from a mismatch between the embedding function used at document ingestion and the embedding function used at query time. The following pattern illustrates how this mismatch occurs in code:

    persistent_client = chromadb.PersistentClient()
    collection = persistent_client.create_collection("emp6")
    
    # Adding documents without specifying an embedding function defaults to 384 dimensions
    for doc in docs:
        collection.add(
            ids=[str(uuid.uuid1())], metadatas=doc.metadata, documents=doc.page_content
        )
    
    # Querying via LangChain Chroma initialized with OpenAI embeddings (1536 dimensions)
    langchain_chroma = Chroma(
        client=persistent_client,
        collection_name="emp6",
        embedding_function=OpenAIEmbeddings(),
    )
    docs1 = langchain_chroma.similarity_search("What is venture")
    

    This snippet demonstrates creating a collection and adding documents without providing an explicit embedding_function, followed by executing a similarity search through LangChain’s Chroma wrapper configured with OpenAIEmbeddings().

    When calling create_collection without an explicit embedding_function parameter, ChromaDB applies its default embedding function, producing vector embeddings with 384 dimensions when documents are added via collection.add(...). When querying that same collection through an interface configured with OpenAIEmbeddings()—such as LangChain’s similarity_search—the incoming query text is converted into a 1536-dimensional vector. Because the existing collection was already indexed at 384 dimensions, ChromaDB rejects the 1536-dimensional query vector with chromadb.errors.InvalidDimensionException.

    Sources: [Bug]: InvalidDimensionException: Embedding dimension 1536 does not match collection dimensionality 384

    Why ChromaDB Defaults to 384 Dimensions and Enforces Fixed Index Dimensionality

    To understand why this mismatch occurs, it helps to look at how ChromaDB handles vector spaces by default. ChromaDB sets a default dimensionality of 384 because its built-in embedder, DefaultEmbeddingFunction, uses the Sentence Transformers model all-MiniLM-L6-v2, which generates 384-dimensional embeddings. When a collection is initialized using create_collection or get_or_create_collection without specifying the embedding_function parameter, ChromaDB applies this default embedder and sets the collection dimensionality to 384.

    ChromaDB enforces strict dimensionality consistency across all records. By default, the underlying vector index is an HNSW graph index constructed for a single, fixed coordinate space, meaning a collection cannot store or query mixed-dimension vectors.

    Furthermore, collection dimensionality cannot be modified in place once initialized. If an operation—including add, upsert, update, or query—attempts to pass embeddings that do not match the collection’s expected dimensionality, ChromaDB rejects the operation and raises chromadb.errors.InvalidDimensionException.

    Sources: Embedding Functions – Chroma Docs, Adding Data to Chroma Collections – Chroma Docs, Configure Collections – Chroma Docs, chromadb.errors.InvalidDimensionException: Dimensionality of (512) does not match index dimensionality (768) ·, Sentence Transformer – Chroma Docs

    Configuring ChromaDB Collections with OpenAI's 1536-Dimension Embedding Function

    To prevent this conflict and ensure your collection accepts 1536-dimensional vectors, you must explicitly supply an embedding function during collection creation. In Python, the chromadb package includes built-in wrappers for providers, allowing you to import OpenAIEmbeddingFunction directly from chromadb.utils.embedding_functions. You must also set your OPENAI_API_KEY environment variable.

    When calling create_collection, link the embedding function to the collection using the embedding_function parameter:

    # Set your OPENAI_API_KEY environment variable
    from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction
    
    collection = client.create_collection(
        name="my_collection",
        embedding_function=OpenAIEmbeddingFunction(
            model_name="text-embedding-3-small"
        ),
    )
    
    # ChromaDB will use OpenAIEmbeddingFunction to embed your documents
    collection.add(
        ids=["id1", "id2"],
        documents=["doc1", "doc2"],
    )
    

    This configuration ensures ChromaDB automatically invokes OpenAIEmbeddingFunction to generate vector representations whenever you call add, update, upsert, or query on the collection using raw text documents.

    For TypeScript applications, install the provider package using a package manager (such as npm install @chroma-core/openai, pnpm add @chroma-core/openai, bun add @chroma-core/openai, or yarn add @chroma-core/openai). You can then pass the initialized instance to createCollection:

    // Set your OPENAI_API_KEY environment variable
    import { OpenAIEmbeddingFunction } from "@chroma-core/openai";
    
    collection = await client.createCollection({
      name: "my_collection",
      embedding_function: new OpenAIEmbeddingFunction({
        modelName: "text-embedding-3-small",
      }),
    });
    
    await collection.add({
      ids: ["id1", "id2"],
      documents: ["doc1", "doc2"],
    });
    

    This TypeScript setup links OpenAIEmbeddingFunction to the collection so that document insertions and queries are embedded using the specified model rather than the default.

    In environments using the Rust client, embedding functions are not attached to the collection; instead, embeddings must be generated separately with your provider SDK and passed directly into calls such as add and query.

    Sources: Embedding Functions – Chroma Docs

    Deleting, Recreating, and Re-indexing Data in ChromaDB

    Once an appropriately configured collection is in place—or when recreating one from scratch—you can safely begin populating it. When inserting records into a collection, ChromaDB raises chromadb.errors.InvalidDimensionException if the supplied embeddings do not match the dimensionality of embeddings already in the collection. Additionally, attempting to re-add a record using an ID that already exists in the collection will be ignored without raising an error; overwriting existing data instead requires updating the data.

    To index new records into a collection, use .add. Each record requires a unique string ID, and you must supply either documents, embeddings, or both. Metadata is optional and can contain strings, integers, floats, booleans, or arrays of those types (empty arrays are not permitted, and all elements in an array must share the same type).

    If you pass only documents, ChromaDB automatically computes embeddings using the collection’s configured embedding_function:

    collection.add(
        ids=["id1", "id2", "id3"],
        documents=["lorem ipsum...", "doc2", "doc3"],
        metadatas=[{"chapter": 3, "verse": 16}, {"chapter": 3, "verse": 5}, {"chapter": 29, "verse": 11}],
    )
    

    This call generates embeddings for each document via the configured collection embedding_function before inserting them.

    If embeddings are precomputed, you can provide them alongside documents. ChromaDB stores the provided embeddings as-is without re-embedding:

    collection.add(
        ids=["id1", "id2", "id3"],
        embeddings=[[1.1, 2.3, 3.2], [4.5, 6.9, 4.4], [1.1, 2.3, 3.2]],
        documents=["doc1", "doc2", "doc3"],
        metadatas=[{"chapter": 3, "verse": 16}, {"chapter": 3, "verse": 5}, {"chapter": 29, "verse": 11}],
    )
    

    This method is used when you already have vector representations and want to avoid recalculating embeddings during ingestion. If your documents are stored externally (such as large images or videos), you can also pass only ids, embeddings, and metadatas, using the IDs to link the stored vectors back to external files.

    Sources: Adding Data to Chroma Collections – Chroma Docs

    Closing thoughts

    Ultimately, this error highlights the hidden hazard of ChromaDB’s default behavior: while falling back to a 384-dimensional model simplifies initial prototyping, it makes multi-tool pipelines—especially those bridging LangChain and OpenAI—deceptively fragile. Because Chroma’s underlying HNSW index enforces a strict, immutable coordinate space that cannot be modified in place, treating the embedding function as an optional setting is a risky architectural shortcut rather than a true convenience. Developers should view embedding function configuration as a mandatory schema definition from day one, explicitly establishing the expected dimensionality at collection creation to avoid the costly necessity of tearing down and re-indexing stored data.

    Frequently Asked Questions

    What causes the error ‘Embedding dimension 1536 does not match collection dimensionality 384’ in ChromaDB?

    This error occurs when a collection is created without an explicit embedding function, defaulting to 384 dimensions, and is subsequently queried with an external model like OpenAI that generates 1536-dimensional embeddings.

    Why does ChromaDB default to 384 dimensions?

    ChromaDB sets a default dimensionality of 384 because its built-in DefaultEmbeddingFunction relies on the Sentence Transformers model all-MiniLM-L6-v2, which outputs 384-dimensional vectors.

    Can you modify the dimensionality of an existing ChromaDB collection?

    No, collection dimensionality cannot be modified in place once initialized. ChromaDB’s underlying HNSW index enforces a fixed coordinate space, and any operation supplying mismatched vector dimensions will trigger an InvalidDimensionException.

    How can you configure a collection in Python to accept 1536-dimensional OpenAI embeddings?

    You can import OpenAIEmbeddingFunction from chromadb.utils.embedding_functions and pass it to the embedding_function parameter when calling create_collection, while ensuring the OPENAI_API_KEY environment variable is set.


    Keep reading: LLM Agents & Tooling

    All guides in this series: LLM Agent Guides: LangChain, LangGraph & AI Tooling

  • Troubleshooting LangGraph InvalidUpdateError: Resolving Concurrent State Updates with Annotated Reducers

    Troubleshooting LangGraph InvalidUpdateError: Resolving Concurrent State Updates with Annotated Reducers

    TL;DR

    • LangGraph raises InvalidUpdateError when multiple concurrent nodes write to the same state key in a single step without an aggregation rule.
    • To intentionally combine concurrent updates, define the state key using typing.Annotated and a reducer function such as operator.add.
    • Unintentional collisions can be prevented by assigning dedicated state keys to parallel branches or routing them into downstream fan-in nodes.

    Diagnosing InvalidUpdateError and the INVALID_CONCURRENT_GRAPH_UPDATE Error

    In LangGraph, conflicting writes during graph execution surface as the API class langgraph.errors.InvalidUpdateError. In LangGraph’s documentation, this issue is tracked under the standard error code INVALID_CONCURRENT_GRAPH_UPDATE.

    The runtime throws this error when multiple parallel nodes (such as Fan-out branches executing concurrently in the same super-step) return an update dictionary targeting the exact same state key. Because LangGraph cannot determine how to resolve the conflicting incoming values for a channel designed to handle only a single write per step, it halts execution to prevent silent data loss or non-deterministic overwrites.

    The exception produces the following message and troubleshooting reference:

    langgraph.errors.InvalidUpdateError: At key '<key>': Can receive only one value per step. Use an Annotated key to handle multiple values.
    For troubleshooting, visit: https://python.langchain.com/docs/troubleshooting/errors/INVALID_CONCURRENT_GRAPH_UPDATE
    

    (Note: Runtime exceptions may print the legacy troubleshooting URL python.langchain.com/..., which corresponds to the current documentation hosted under docs.langchain.com/....)

    This traceback indicates that two or more concurrently executing nodes attempted to write to <key> within the exact same super-step without an aggregation rule.

    When diagnosing this error, confirm whether the concurrent writes to the same key are intentional or the result of graph design:

    • Unintentional Key Collisions: If parallel branches produce distinct results that should not be combined into the same field, alter the graph structure to use dedicated state keys (such as branch_a_output and branch_b_output). Alternatively, route edges from the parallel nodes into a downstream convergence or fan-in node that consumes both results and applies the unified update in a subsequent step.
    • Intentional Multi-Branch Updates: If multiple parallel branches must write to the same collection or field simultaneously, the target key must be declared using typing.Annotated with an appropriate Reducer function to dictate how the concurrent values should be combined.

    Sources: INVALID_CONCURRENT_GRAPH_UPDATE – Docs by LangChain, InvalidUpdateError, Errors – Docs by LangChain, GitHub Issue Title: InvalidUpdateError when using parallel subgraphs with shared state keys in LangGraph · Iss, How to modify specific item in a State – LangGraph – LangChain Forum, docs(langgraph): fixes fan-out example

    The Pregel Execution Model: Why Concurrent Updates Halt the Tick Loop

    To understand why this failure occurs under the hood, it helps to examine how the Pregel execution model processes state within a super-step. Specifically, an INVALID_CONCURRENT_GRAPH_UPDATE error arises when a StateGraph receives concurrent updates to a state property that does not support them, such as during a Fan-out or other parallel execution where multiple nodes return values for the same key within a single step. While a single node returning a value overwrites that key in the state, multiple nodes returning values for the same key create uncertainty about how to update the internal state, causing the graph to raise an error.

    For example, executing a graph via graph.stream() under these conditions can result in a langgraph.errors.InvalidUpdateError:

    langgraph.errors.InvalidUpdateError: At key 'jokes': Can receive only one value per step. Use an Annotated key to handle multiple values.
    For troubleshooting, visit: https://python.langchain.com/docs/troubleshooting/errors/INVALID_CONCURRENT_GRAPH_UPDATE
    

    To prevent or resolve this error when executing nodes in parallel, relevant state keys can be defined with a Reducer function using typing.Annotated (such as Annotated[list, operator.add]), which specifies the logic for combining multiple values.

    Sources: INVALID_CONCURRENT_GRAPH_UPDATE – Docs by LangChain, Fix to graph-api docs Send API example

    Default Overwrite Channels vs. Reducer Channels

    This distinction highlights the core operational difference between default overwrite channels and reducer-enabled channels. By default, fields defined without an annotation in a TypedDict, dataclass, or Pydantic model operate as single-value channels backed internally by LastValue. When such a channel is configured without a Reducer function, receiving multiple update values in a single step causes LangGraph to raise an InvalidUpdateError:

    langgraph.errors.InvalidUpdateError: At key '<key>': Can receive only one value per step. Use an Annotated key to handle multiple values.
    

    To handle multiple values, fields can be configured with Reducer functions using typing.Annotated (such as Annotated[int, operator.add]).

    The presence or absence of a Reducer function also affects how bugs surface during execution. In one reported open issue (#9005), a developer observed a framework bug where overriding an inherited field in a dataclass or Pydantic subclass caused Command(update=...) to apply the update twice in a single step. When the overridden field (count) was configured without a Reducer function, this duplicate application triggered:

    langgraph.errors.InvalidUpdateError: At key 'count': Can receive only one value per step. Use an Annotated key to handle multiple values.
    

    However, when the field configured a Reducer function such as Annotated[int, operator.add], no error was raised, and the framework bug caused the reducer to apply the update twice (yielding a count of 5 instead of 3 from an initial value of 1 and an update of 2). Here, the reducer is not the source of the issue, but its normal aggregation behavior masks the framework’s double-invocation bug.

    Sources: `Command(update=…)` applies an overridden inherited state field twice, Receiving multiple inputs on a single node not working in a vanilla state

    Configuring Annotated State Keys with operator.add and Custom Reducers

    Building on this reducer mechanism, state schemas in LangGraph can use typing.Annotated within a TypedDict to specify a standard Reducer function such as operator.add:

    from typing import Annotated, Sequence
    import operator
    from typing_extensions import TypedDict
    
    class AgentState(TypedDict):
        messages: Annotated[Sequence[dict], operator.add]
    
    class State(TypedDict):
        my_key: Annotated[str, operator.add]
    

    When multiple parallel upstream nodes (such as up and side in a topology like graph.add_edge(["up", "side"], "down")) execute concurrently in the same super-step and both write updates to the same state key, a reducer is required on that key to prevent update conflicts across those parallel branches before control reaches the downstream down node.

    When list merging requires avoiding duplicate entries, custom reducer functions can be defined. For example, in a discussion around handling subgraph outputs, a community workaround demonstrated a custom reducer:

    def reduce_results(left: list[str], right: list[str]) -> list[str]:
        """This approach might not be the best, but it can solve repeat problems."""
        left_table = {result: i for i, result in enumerate(left)}
        merged = left.copy()
        for right_result in right:
            if (existing_idx := left_table.get(right_result)) is not None:
                merged[existing_idx] = right_result
            else:
                merged.append(right_result)
        return merged
    
    class ParentState(TypedDict):
        question: str
        results: Annotated[list[str], reduce_results]
    

    In that same discussion, a contributor suggested separating input and output schemas when handling subgraphs: using the parent graph’s state for the input allows full access to parent state values, while the output schema contains only the required fields, typed as lists with an Annotated Reducer function.

    Sources: Sink node issue, if multiple subgraphs are used in parallel, two node flow to one node

    Edge Cases: Parallel Subgraphs, Conditional Self-Loops, and Shared State Keys

    Beyond standard fan-out configurations, several distinct edge cases and open bug reports illustrate scenarios where InvalidUpdateError surfaces unexpectedly. These scenarios stem from different factors and should not be conflated:

    In an open bug report (issue #6446), executing a compiled subgraph in parallel with another node in the parent graph resulted in an unexpected InvalidUpdateError. In the reporter’s minimal reproducer, a parent graph branched from an initial node to both a standard node (nodeb) and a compiled subgraph:

    def create_graph():
        builder = StateGraph(State)
        builder.add_node("nodea", nodea)
        builder.add_node("subgraph", subgraph())
        builder.add_node("nodeb", nodeb)
    
        builder.add_edge(START, "nodea")
        builder.add_edge("nodea", "nodeb")
        builder.add_edge("nodea", "subgraph")
        builder.add_edge("nodeb", END)
        builder.add_edge("subgraph", END)
        return builder.compile()
    

    When nodeb and subgraph executed concurrently, LangGraph raised:

    langgraph.errors.InvalidUpdateError: At key 'b': Can receive only one value per step. Use an Annotated key to handle multiple values.
    

    The reporter noted that nodeb only updated key b while the subgraph internal nodes only updated keys s1 and s2. However, the root cause has not been officially confirmed by maintainers, and the report remains an open bug rather than an established graph behavior.

    In another scenario (issue #2336), a user encountered an update collision when combining parallel branches with a conditional self-loop (not a RetryPolicy). An initial node branched into parallel nodes b and c, where node c evaluated a conditional edge that routed back to itself on a retry condition before proceeding downstream. Executing this flow produced InvalidUpdateError targeting key count_b. Because the issue was closed without an official explanation or confirmed fix, it is only presumed that a self-loop iteration coincided with another branch in the same super-step to cause multiple writes to count_b.

    Sources: GitHub Issue Title: InvalidUpdateError when using parallel subgraphs with shared state keys in LangGraph, langgraph.errors.InvalidUpdateError: Can receive only one value per step. Use an Annotated key to handle multiple values. error when running retry in graph parallel processing

    Closing thoughts

    In standard graph workflows, InvalidUpdateError serves as an essential safeguard within LangGraph’s Pregel model, preventing non-deterministic data overwrites when multiple concurrent nodes write to the same unannotated channel. When parallel writes to a shared key are intentional, configuring the field with an Annotated reducer provides a deterministic aggregation rule. For unintentional collisions, architectural adjustments such as dedicated state keys or downstream fan-in nodes cleanly separate updates across steps.

    However, reducers should not be treated as a blanket remedy. In specific bug paths such as subclass field inheritance with Command(update=...) (#9005), adding a reducer can mask duplicate updates that would otherwise trigger a visible error. Furthermore, edge cases like parallel subgraphs (#6446) and conditional self-loops (#2336) represent distinct issues with separate, unconfirmed causes rather than straightforward topology mistakes. Distinguishing between intended concurrent updates, graph architectural patterns, and unresolved framework bugs is crucial when troubleshooting update conflicts in LangGraph.

    Frequently Asked Questions

    Why does LangGraph raise an InvalidUpdateError?

    LangGraph raises this error when multiple parallel nodes attempt to update the exact same state key within the same super-step without an aggregation rule. It halts execution to prevent non-deterministic overwrites and silent data loss.

    How can you resolve InvalidUpdateError when multiple branches intentionally update the same key?

    You can declare the target state key using typing.Annotated along with a Reducer function, such as operator.add or a custom reducer. This tells LangGraph how to combine the conflicting concurrent values instead of attempting an overwrite.

    What should you do if parallel nodes are causing unintentional key collisions?

    You can alter the graph structure to give each branch its own dedicated state key, such as branch_a_output and branch_b_output. Alternatively, you can route the parallel nodes to a downstream fan-in node that aggregates the values in a subsequent step.


    Keep reading: LLM Agents & Tooling

    All guides in this series: LLM Agent Guides: LangChain, LangGraph & AI Tooling

  • Troubleshooting ‘ValueError: Some modules are dispatched on the CPU or the disk’ in Transformers

    Troubleshooting ‘ValueError: Some modules are dispatched on the CPU or the disk’ in Transformers

    TL;DR

    • The dispatch error acts as a safety check when quantized models cannot fit entirely on GPUs because low-precision kernels cannot natively execute on the CPU or disk.
    • Past memory calculation bugs in Accelerate integrations—such as under-sized leaf module buffers and assuming uniform layer sizes in no-split modules—previously caused CPU offloading even on systems with ample GPU VRAM.
    • These issues (tracked in issue #46823) were resolved in Transformers through PR #47203 and PR #47211, superseding earlier proposals such as PR #47061.

    Symptom and Trigger: When the Quantization Dispatch Error Occurs

    In Hugging Face Transformers (working alongside Accelerate and bitsandbytes), the dispatch error occurs during model initialization via from_pretrained() when attempting to load a quantized model—such as with load_in_8bit or load_in_4bit—under automatic or custom device placement using device_map (such as device_map="auto").

    When this condition is met, Hugging Face Transformers raises the following exception:

    ValueError: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model.
    

    This error serves as a safety check triggered by the following sequence of constraints and runtime checks:

    • Kernel Device Constraints: The underlying quantization engine (bitsandbytes) executes its low-precision matrix operations using specialized GPU kernels.
    • VRAM Exhaustion and Placement Calculation: When the dispatch planner calculates module placement across available hardware, it may determine that available GPU VRAM—after accounting for CUDA kernel context overhead—is insufficient to host every quantized layer.
    • Automatic Fallback: Upon exhausting GPU VRAM, the dispatch planner attempts CPU offloading ("cpu") or disk offloading ("disk") for the overflow layers.
    • Execution Safety Check: Because quantized modules cannot execute their low-precision kernels natively on CPU or disk without explicit offload configurations, Hugging Face Transformers halts execution with ValueError: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model to prevent immediate failure during forward passes or inference.

    The error directly indicates that the planned quantization configuration cannot fit entirely within the targeted GPU VRAM under the current device_map strategy.

    Sources: huggingface error2 #error – Qiita, PR #31098 Enable BNB multi-backend support – SemanticDiff, Bitsandbytes · Hugging Face, Bitsandbytes · Hugging Face, Loading big models into memory · Hugging Face

    Accelerate Memory Calculation Bugs: Why Ample VRAM Models Still Spill to CPU

    While this dispatch error is designed to catch genuine out-of-memory conditions, models loaded with load_in_4bit and device_map="auto" historically dispatched modules to CPU despite having an explicit and ample max_memory configuration. Under these circumstances, ValueError: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model was raised from quantizer_bnb_4bit.validate_environment (via _get_device_map in integrations/accelerate.py).

    This behavior, tracked in issue #46823, stemmed from how device memory caps and layer sizes were calculated in integrations/accelerate.py:

    • Leaf module buffer under-sizing: get_balanced_memory calculates device caps with a buffer that previously only measured _no_split_modules. However, childless leaf modules (such as embed_tokens and lm_head, which are not reduced by quantization) act as un-splittable atoms. When a leaf module was larger than the balanced cap, device placement skipped earlier GPUs and collapsed onto the remaining capacity, spilling excess modules to CPU offloading even when total GPU VRAM was six times the model size.
    • Variable layer sizes in no-split modules: The block size for _no_split_modules was historically computed by sampling only the first matching layer. This caused under-packing when layers of the same class varied significantly in size. For instance, loading MiniMaxAI/MiniMax-M3 (a 428B sparse MoE VL model) with load_in_4bit, device_map="auto", and max_memory={i: "165GiB" for i in range(8)} on an 8x B200 setup triggered the dispatch ValueError because auto-placement under-packed the variable layers.

    While an initial proposal (PR #47061) attempted to size buffers via get_max_layer_size, maintainers determined it was not the correct fix and closed it in favor of targeted solutions that have since been merged into transformers main:

    • PR #47203 (merged July 9, 2026) addressed the variable layer sizing bug in _no_split_modules.
    • PR #47211 (merged September 18, 2026) corrected the leaf-module buffer calculation by adopting an updated sizing formula: buffer = int(1.25 * max(buffer, mean_leaves, max(leave_modules_sizes.values(), default=0))).

    Sources: Fix device_map="auto" under-packing that spills quantized models to cpu (#46823), 4-bit load_in_4bit + device_map='auto' + explicit max_memory still dispatches modules to CPU on a model that fits -> ValueError

    Closing thoughts

    While Transformers frames this error as a straightforward physical memory shortage, the underlying cause in certain multi-GPU quantized workflows was rooted in placement calculations. The handling of variable-sized MoE layers and unquantized leaf modules previously caused device_map="auto" to bypass available GPUs and spill modules to CPU. These issues have since been officially resolved in Transformers through PR #47203 and PR #47211. Developers working on older versions who encounter this error on ample hardware should update their transformers installation or inspect per-module allocations and explicit device maps.

    Frequently Asked Questions

    What causes the ‘ValueError: Some modules are dispatched on the CPU or the disk’ error in Hugging Face Transformers?

    This error occurs during model initialization with quantization settings like load_in_8bit or load_in_4bit when available GPU VRAM is insufficient to hold all layers. Because quantized modules require specialized GPU kernels and cannot execute natively on CPU or disk, Transformers halts execution as a safety check when modules are offloaded.

    Why did this dispatch error occur on systems with ample GPU VRAM?

    Historically, memory calculation bugs in Accelerate integrations caused auto-placement to under-pack GPUs. If variable-sized layers or un-splittable leaf modules exceeded the calculated balanced memory buffer, earlier GPUs were skipped, causing remaining layers to collapse onto the last GPU or spill to CPU.

    Which types of modules commonly triggered unintended CPU offloading during quantization?

    Childless leaf modules not reduced by quantization (such as embed_tokens and lm_head) and variable-sized layers in no-split modules (such as those in sparse MoE architectures) commonly caused under-packing when buffer reservations were under-calculated.

    How were these quantization memory calculation issues fixed in Transformers?

    An early proposed PR (#47061) was superseded by two merged fixes: PR #47203 resolved the variable layer size estimation in no-split modules, and PR #47211 updated the buffer calculation in get_balanced_memory using leaf module sizes (buffer = int(1.25 * max(buffer, mean_leaves, max(leave_modules_sizes.values(), default=0)))).


    Keep reading: LLM Inference & Serving

    All guides in this series: LLM Inference & Serving Guides: vLLM, Ollama & Hugging Face

  • Troubleshooting LangChain OutputParserException: Both Final Answer and Parse-able Action Produced

    Troubleshooting LangChain OutputParserException: Both Final Answer and Parse-able Action Produced

    TL;DR

    • The OutputParserException fires when a ReAct-style agent’s output matches both the action pattern (Action:/Action Input:) and the final-answer prefix (Final Answer:) in the same turn, so ReActSingleInputOutputParser cannot tell whether to dispatch a tool call or end the run.
    • handle_parsing_errors=True catches the exception and, by default, feeds back a fixed placeholder message (“Invalid or incomplete response”) as the next observation rather than the actual error text — pass a custom string or callable instead of True to send back something more specific
    • The most reliable fix is architectural: move off initialize_agent‘s legacy text-parsing ReAct agent onto a native tool-calling agent, where the turn boundary is enforced by the provider’s API instead of a regex over free text.

    Error Symptom: OutputParserException in ReAct Agent Execution

    In a GitHub issue report, a zero-shot ReAct agent built with initialize_agent and AgentType.ZERO_SHOT_REACT_DESCRIPTION raised langchain_core.exceptions.OutputParserException: Parsing LLM output produced both a final answer and a parse-able action while running against a self-hosted Meta-Llama-3-8B-Instruct model through a Hugging Face pipeline, with a Wikipedia search tool and a DuckDuckGo search tool attached.

    The reporter’s code, quoted as posted in the issue:

    runAgent = initialize_agent(
        llm=hf,
        tools=tools,
        verbose=True,
        agent_type=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    )
    userInput = "Bitcoin"
    output = runAgent.run(f'Write an academic abstract about {userInput}')
    

    This is reproduced verbatim from the report — hf and tools are defined earlier in the reporter’s own script and not shown here. Note that initialize_agent‘s formal parameter for selecting an agent type is agent, not agent_type. However, when both agent and agent_path are None, initialize_agent defaults to AgentType.ZERO_SHOT_REACT_DESCRIPTION. Thus, passing agent_type is ignored as an extra keyword argument, but the resulting instance is still a zero-shot ReAct agent and reproduces the exact same error.

    When the model over-generates — for example continuing past a tool call to also emit Final Answer: in the same response, or fabricating its own Observation: lines and looping through several invented steps — ReActSingleInputOutputParser.parse() finds both an action match and the final-answer prefix in the same text and raises the exception instead of guessing which one the model meant.

    After first hitting this, the reporter tried passing handle_parsing_errors=True. Rather than resolving cleanly, the run continued generating further hallucinated Thought/Action/Action Input/Observation cycles on its own without reaching a stop state — handle_parsing_errors=True catches the exception and passes the error back into the scratchpad, but it does not prevent the model from continuing to over-generate. A later comment on the same issue suggested adding early_stopping_method="force":

    runAgent = initialize_agent(
        llm=hf,
        tools=tools,
        verbose=True,
        early_stopping_method="force",
        handle_parsing_errors=True,
        agent_type=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    )
    

    "force" is already AgentExecutor‘s default value for early_stopping_method. The early_stopping_method parameter is independent of handle_parsing_errors and controls how the agent terminates once max_iterations or max_execution_time is reached (e.g., when repeated parsing retries exhaust the iteration limit), not at the moment the parsing conflict happens.

    Sources: OutputParserException: Parsing LLM output produced both a final answer and a parse-able action

    Root Cause: Turn Ambiguity in ReActSingleInputOutputParser Protocols

    LangChain’s text-based ReAct agent protocol relies on a strict turn-by-turn structure in which tool execution and agent termination are meant to be mutually exclusive:

    • Action step: the model emits Thought:, Action: <tool_name>, and Action Input: <input>, and the runtime is meant to stop there, dispatch an AgentAction, run the tool, and inject the result as the next Observation:.
    • Termination step: the model emits Thought: and Final Answer: <response>, and the runtime returns an AgentFinish to end the loop.

    ReActSingleInputOutputParser (and the legacy MRKL parser) decide which of these happened by testing the model’s raw text for two separate patterns: an action match (Action: ... / Action Input: ...) and the final-answer prefix defined by the FINAL_ANSWER_ACTION constant ("Final Answer:"). The exception fires whenever both patterns match in the same output, which happens in at least two distinct ways:

    1. The model hallucinates the rest of the trajectory. After an Action/Action Input pair, the model keeps generating instead of stopping — inventing its own Observation: text for a tool it never actually called, then continuing through further Thought/Action cycles until it eventually appends a Final Answer: as well.
    2. The model appends a final answer directly, with no fabricated observation in between. The model emits Action:/Action Input: and then, in the same continuous response, immediately adds Thought: I now know the final answer and Final Answer: — without ever generating an Observation: line at all.

    The distinction matters for choosing a fix: anything that stops generation at the Observation: boundary only intercepts case 1, because case 2 never produces that string in the first place. In both cases, because the runtime cannot tell from the text alone whether to dispatch an AgentAction or return an AgentFinish, ReActSingleInputOutputParser aborts and raises langchain_core.exceptions.OutputParserException using the string constant FINAL_ANSWER_AND_PARSABLE_ACTION_ERROR_MESSAGE.

    Sources: langchain.schema.output_parser.OutputParserException · Issue #8226 · langchain-ai/langchain · GitHub, vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQGQ6Yvdmo…, エラー Parsing LLM output produced both a final answer and a parse-able action #LangChain – Qiita, FINAL_ANSWER_AND_PARSABLE_ACTION_ERROR_MESSAGE, Observation: Invalid or incomplete response using HF TGI · Issue #9381 · langchain-ai/langchain · GitHub

    How to Fix It: From Quick Patch to Eliminating the Error Class

    There are three points at which you can intervene, from patching around the crash to removing the ambiguity altogether.

    1. Catch the exception with handle_parsing_errors. Passing handle_parsing_errors=True to initialize_agent/AgentExecutor stops OutputParserException from crashing the run by catching the exception and returning the error text (or a fallback like “Invalid or incomplete response”) back as an observation so the model can retry. Passing a custom string (e.g. handle_parsing_errors="<custom message>") or a callable lets you control the instructional prompt or dynamically sanitize the output. On its own this only manages the failure at runtime — it does not prevent the model from over-generating again on a later turn. If the model continually repeats parsing errors until hitting max_iterations or max_execution_time, early_stopping_method determines how the agent handles termination at that limit.

    2. Verify and configure stop sequences for the Observation: boundary. Zero-shot ReAct agents in LangChain generally pass stop sequences such as ["\nObservation:", "\n\tObservation:"] to LLM invocations to halt generation immediately after an action is emitted. However, local model wrappers (such as Hugging Face pipeline wrappers) or certain model setups may fail to apply or honor these stop sequences properly. Verifying that stop arguments are actually respected by your LLM wrapper prevents case 1 — the model hallucinating its own observations and looping through further steps. Note that stop sequences do not prevent case 2, where the model appends Final Answer: directly after Action Input: without ever generating an Observation: token.

    3. Move to a native tool-calling agent. ReActSingleInputOutputParser and other text-based parsers exist only because the ReAct protocol encodes actions and answers as plain text that has to be pattern-matched after the fact. Where the underlying model supports tool calling, building the agent with create_tool_calling_agent (or an equivalent structured-output agent) instead of initialize_agent‘s ZERO_SHOT_REACT_DESCRIPTION moves argument parsing and turn boundaries into the API itself, so there is no free-text Action:/Final Answer: pattern left to collide. This isn’t guaranteed for every model, though — a self-hosted model served through a Hugging Face pipeline, like the Meta-Llama-3-8B-Instruct example above, may not expose tool calling the way a hosted provider API does. It is also worth knowing that initialize_agent/AgentExecutor is LangChain’s older agent interface; current LangChain documentation points toward create_agent/LangGraph-based agents as the actively maintained path going forward.

    Sources: langchain.schema.output_parser.OutputParserException · Issue #8226 · langchain-ai/langchain · GitHub, FINAL_ANSWER_AND_PARSABLE_ACTION_ERROR_MESSAGE

    Closing thoughts

    This parsing error illustrates the limitations of relying on regex-based text parsing for agent control flow when underlying LLMs generate past expected turn boundaries. While parameters like handle_parsing_errors handle parser exceptions at runtime by passing feedback back to the scratchpad, and early_stopping_method dictates how the run concludes when iteration limits are reached, neither eliminates the root formatting conflicts on its own.

    For more robust agent pipelines, using native tool-calling agents where supported offers a cleaner architecture by moving tool dispatching from free-form prompt text to structured API interactions.

    Frequently Asked Questions

    Why does the OutputParserException regarding a final answer and parse-able action occur in LangChain?

    This exception occurs when an LLM produces both an action pattern (like Action: and Action Input:) and a final answer prefix (Final Answer:) in a single output turn. Because the ReAct parser checks for both conditions and finds both to be true, it cannot determine whether to invoke a tool via an AgentAction or end execution with an AgentFinish, causing it to abort and raise the error.

    What issue can arise when only handle_parsing_errors=True is passed to initialize_agent?

    Passing handle_parsing_errors=True catches the OutputParserException and returns the error message (or a fallback like “Invalid or incomplete response”) back into the scratchpad as an observation. However, it does not prevent an over-generating model from continuing to hallucinate Thought/Action/Action Input/Observation cycles on subsequent turns instead of cleanly resolving.

    What is the difference between handle_parsing_errors and early_stopping_method?

    handle_parsing_errors manages runtime parser failures when an OutputParserException is raised by feeding error feedback or a custom observation back to the model. early_stopping_method is an independent setting that controls how the execution terminates once max_iterations or max_execution_time has been reached.

    What is the underlying root cause of turn ambiguity in ReActSingleInputOutputParser?

    The root cause is the LLM over-generating text across turn boundaries rather than yielding control after emitting an action or a final answer. When the model outputs both action patterns and termination keywords in the same turn, the parser’s regex and string checks match both mutually exclusive states.


    Keep reading: LLM Agents & Tooling

    All guides in this series: LLM Agent Guides: LangChain, LangGraph & AI Tooling

  • Troubleshooting bitsandbytes: Resolving ‘RuntimeError: CUDA Setup failed despite GPU being available’

    Troubleshooting bitsandbytes: Resolving ‘RuntimeError: CUDA Setup failed despite GPU being available’

    TL;DR

    • The CUDA Setup failed error occurs when bitsandbytes cannot dynamically locate runtime libraries, encounters ABI mismatches, or lacks sufficient GPU compute capability, all of which can be inspected using python -m bitsandbytes.
    • Linux initialization failures caused by dynamic linking issues or missing libstdc++ symbols can be resolved by configuring dynamic library paths or compiling bitsandbytes from source against the local CUDA environment using CMake.
    • Native Windows environments frequently fail due to the library searching for Linux-specific paths and shared objects; running inside WSL2 or installing compatible Windows wheels provides a practical workaround.

    Understanding Why bitsandbytes Fails to Initialize CUDA

    The RuntimeError: CUDA Setup failed despite GPU being available error occurs when the Python environment successfully detects an NVIDIA GPU through PyTorch, but the underlying bitsandbytes C++/CUDA extension fails to load, locate, or execute its compiled GPU kernels. Because bitsandbytes operates as a Python wrapper over custom CUDA and C++ kernels designed for memory-efficient deep learning operations, initialization fails if the low-level execution environment does not satisfy its dynamic linking or hardware requirements.

    CUDA initialization failures are driven by three primary root causes:

    • Dynamic CUDA Library Discovery: At runtime, bitsandbytes searches system search paths for CUDA dynamic runtime shared libraries, such as libcudart.so on Linux or nvcuda.dll on Windows. If these libraries are absent from the linker search paths or are incompatible with the active PyTorch installation, the C++ extension cannot be loaded.
    • CUDA Version and Symbol Mismatches: When the CUDA version used to compile the installed PyTorch build does not match the system CUDA Toolkit or the pre-compiled binaries distributed within the bitsandbytes package, the runtime environment cannot properly bind the required symbols.
    • Compute Capability and Architecture Requirements: bitsandbytes requires specific minimum NVIDIA compute capability levels to execute its kernel backends. Attempting to initialize the library on an unsupported architecture or on a build missing compiled kernels for that specific architecture triggers a setup failure:
      • Compute capability 6.0+ (Pascal or newer): Required to run basic 8-bit optimizers and 4-bit/8-bit quantization routines.
      • Compute capability 7.5+ (Turing, Ampere, Ada Lovelace, Hopper, or newer): Required for accelerated LLM.int8() mixed-precision matrix multiplication.

    Sources: mistralai/Mixtral-8x7B-Instruct-v0.1 · [AUTOMATED] Model Memory Requirements, CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment! I, bitsandbytes · PyPI, RuntimeError: CUDA Setup failed despite correct environment paths on Windows · Issue #976 · bitsandbytes-found, stackoverflow.com/questions/79913258/cuda-setup-error-missing-libnvji…

    Diagnosing CUDA and Runtime Mismatches Using 'python -m bitsandbytes'

    To pinpoint which of these underlying root causes is responsible for the error in a specific environment, running the diagnostic command provided in the error message outputs detailed information about the environment, detected NVIDIA GPU hardware, and library search stages:

    python -m bitsandbytes
    

    Executing this command triggers bitsandbytes’ internal CUDA setup routine to inspect system paths, compute capability levels, and dynamic library dependencies.

    The resulting diagnostic log reveals the exact sequence of checks bitsandbytes performs. In one developer’s environment report, the diagnostic output captured the following path inspections and library loading stages:

    CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
    CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
    CUDA SETUP: Highest compute capability among GPUs detected: 7.5
    CUDA SETUP: Detected CUDA version 121
    CUDA SETUP: Loading binary /home/yed_dev/.local/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda121.so...
    /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /home/yed_dev/.local/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda121.so)
    CUDA SETUP: Something unexpected happened. Please compile from source:
    git clone [email protected]:TimDettmers/bitsandbytes.git
    cd bitsandbytes
    CUDA_VERSION=121
    python setup.py install
    

    As demonstrated in this log, running python -m bitsandbytes allows you to inspect several diagnostic points:

    • Directory and runtime discovery: It identifies missing or non-existent directories listed in environment paths (such as /usr/local/cuda/lib missing expected shared libraries like libcudart.so, libcudart.so.11.0, or libcudart.so.12.0) and notes when fallback searches find candidates like /usr/local/cuda/lib64/libcudart.so.
    • Hardware and version detection: It prints the detected CUDA version (e.g., CUDA 121) and the highest detected NVIDIA GPU compute capability (e.g., 7.5).
    • Binary loading and ABI mismatches: It shows which specific pre-compiled binary is being loaded (such as libbitsandbytes_cuda121.so) and captures underlying system loader failures, such as a missing C++ ABI symbol in the host’s libstdc++.so (e.g., version 'CXXABI_1.3.9' not found).

    If the setup encounters an unrecoverable failure during binary loading, diagnostic tools often suggest compiling bitsandbytes directly from source for the detected CUDA version.

    Sources: help!!!RuntimeError: CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    Fixing Linux Setup Errors: Configuring CUDA Paths and Missing libstdc++ Symbols

    Once the diagnostic output reveals the exact nature of the failure, addressing it requires targeting the specific platform constraints. In Linux environments, bitsandbytes can detect the local CUDA runtime and hardware specifications successfully but still fail to initialize due to missing system library symbols. In one reported setup log, the initialization routine correctly located the CUDA runtime path at /usr/local/cuda/lib64/libcudart.so, detected CUDA version 117, and identified an NVIDIA GPU compute capability of 7.5, yet failed when attempting to load the pre-compiled binary:

    CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
    CUDA SETUP: Highest compute capability among GPUs detected: 7.5
    CUDA SETUP: Detected CUDA version 117
    CUDA SETUP: Loading binary /usr/local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda117.so...
    /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /usr/local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda117.so)
    

    This log demonstrates an incompatibility where the system’s libstdc++.so lacks the required CXXABI_1.3.9 symbol needed by libbitsandbytes_cuda117.so. When this dynamic linking error occurs during execution—such as importing dependent libraries like peft that call import bitsandbytes as bnb—compiling the package directly from source against the local environment resolves the ABI mismatch.

    While legacy runtime diagnostic outputs in older package versions recommended running make and python setup.py install from the original repository, current official bitsandbytes documentation maintains source builds through bitsandbytes-foundation/bitsandbytes using CMake:

    git clone https://github.com/bitsandbytes-foundation/bitsandbytes.git
    cd bitsandbytes
    cmake -DCOMPUTE_BACKEND=cuda -S .
    make
    pip install -e .
    

    This compilation workflow builds the CUDA backend kernels directly with your host compiler and active CUDA SDK, ensuring complete symbol compatibility with your system’s libstdc++ runtime.

    Sources: CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment!

    Resolving Initialization Failures on Windows via Pre-compiled Wheels

    While Linux environments primarily encounter ABI and dynamic path issues, running bitsandbytes in native Windows environments presents a different set of challenges because the initialization routines often look for Linux-specific shared libraries and paths. In user reports running setups such as CUDA 11.8 with PyTorch 2.0.1 or CUDA 12.2.2, the initialization routine scans for .so binaries rather than Windows-compatible dynamic libraries, producing path warnings and failing to register the NVIDIA GPU:

    bin C:\...\bitsandbytes\libbitsandbytes_cpu.so
    UserWarning: C:\... did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
    CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
    UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
    CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed?
    UserWarning: WARNING: No GPU detected! Check your CUDA paths. Proceeding to load CPU-only library...
    CUDA SETUP: Loading binary ...\libbitsandbytes_cpu.so...
    argument of type 'WindowsPath' is not iterable
    CUDA SETUP: Problem: The main issue seems to be that the main CUDA library was not detected.
    CUDA SETUP: Solution 1): Your paths are probably not up-to-date. You can update them via: sudo ldconfig.
    

    This log excerpt illustrates bitsandbytes attempting to search Linux directories like /usr/local/cuda/lib64 and falling back to a CPU-only .so binary on Windows, ultimately raising a TypeError (argument of type 'WindowsPath' is not iterable) and presenting Linux-specific solutions like sudo ldconfig.

    In cases reported across Windows installations using tools like threestudio or stable-diffusion-webui (with dependencies like xformers 0.0.20), the setup routine fails when it cannot resolve Windows-native runtime bindings. For threestudio, project collaborators recommended running the workflow inside WSL2 to bypass native Windows library resolution failures. In native Windows environments, users typically require dedicated Windows-compatible wheel builds to properly load the CUDA dynamic libraries.

    Sources: CUDA Setup failed despite GPU being available, CUDA Setup failed despite GPU being available. Please run the following command to get more information

    Closing thoughts

    Ultimately, these platform-specific failures illustrate that PyTorch detecting a GPU is only half the battle; bitsandbytes relies on strict low-level dynamic linking that fails whenever host runtime libraries, ABI symbols, or operating system paths diverge from its pre-built binaries.

    Running the python -m bitsandbytes diagnostic tool is essential because it immediately clarifies whether the issue is an unlocated libcudart.so, an incompatible host libstdc++ version, or a Windows environment trying to load Linux-specific paths. When these mismatches occur, targeted source compilation via CMake on Linux or operating within WSL2 / utilizing Windows-compatible builds provides the most reliable way to align the library’s custom kernels with your active CUDA environment.

    Frequently Asked Questions

    Why does bitsandbytes throw a CUDA Setup failed error even when PyTorch detects a GPU?

    This error occurs because the underlying C++/CUDA extension fails to load, locate, or execute its GPU kernels. Common root causes include dynamic CUDA library discovery issues, CUDA version and C++ ABI symbol mismatches, or an unsupported GPU compute capability.

    How can I diagnose CUDA and runtime issues with bitsandbytes?

    You can run the command ‘python -m bitsandbytes’ in your terminal. This diagnostic tool inspects environment paths, checks for required shared libraries like libcudart.so, reports detected CUDA and compute capability versions, and identifies binary loading failures.

    What compute capability is required to run bitsandbytes?

    Bitsandbytes requires a minimum compute capability of 6.0+ (Pascal or newer) for basic 8-bit optimizers and 4-bit/8-bit quantization routines. For accelerated LLM.int8() mixed-precision matrix multiplication, a compute capability of 7.5+ (Turing or newer) is required.

    Why does bitsandbytes encounter initialization errors on Windows?

    On native Windows environments, bitsandbytes often defaults to searching for Linux-specific file paths and .so shared libraries rather than Windows-compatible dynamic libraries. Running inside WSL2 or installing compatible Windows wheels provides a practical workaround to avoid these path and dynamic linking issues.


    Keep reading: LLM Inference & Serving

    All guides in this series: LLM Inference & Serving Guides: vLLM, Ollama & Hugging Face

  • Fixing Quality Degradation When Merging 4-Bit QLoRA Adapters in PEFT

    Fixing Quality Degradation When Merging 4-Bit QLoRA Adapters in PEFT

    TL;DR

    • In QLoRA, the base model weights are stored in 4-bit precision (NormalFloat4 / NF4 via bitsandbytes) while the trained LoRA adapter matrices stay in 16-bit floating point (torch.float16 or torch.bfloat16).
    • Calling merge_and_unload directly on a model that is still loaded in 4-bit forces PEFT to dequantize each weight to 16-bit, add the adapter delta, and then re-quantize the result back into NF4.
    • The dequantize and add steps are effectively lossless; the final re-quantization is not. It rounds every merged weight to the nearest NF4 bucket, across every layer at once, and PEFT warns this “may get different generations due to rounding errors.”
    • The fix is to load the base model in 16-bit, run merge_and_unload there, save, and — only if deployment needs it — quantize the merged checkpoint once afterward.

    Understanding the Pitfall: Why Merging into 4-Bit Weights Hurts Output Quality

    In QLoRA configurations, the base model weights are stored in 4-bit precision formats such as NormalFloat4 (NF4) via bitsandbytes, whereas the trained LoRA adapter matrices A and B are kept in 16-bit floating-point precision (torch.float16 or torch.bfloat16).

    When you call merge_and_unload() on a model that is still loaded in 4-bit, PEFT cannot add a 16-bit delta to a 4-bit weight directly. Instead it dequantizes each base weight back to the compute dtype (bf16/fp16), adds the LoRA update ΔW = (α / r)·BA in floating point, and then re-quantizes the updated weight back into NF4. The dequantize and add steps are well defined and essentially lossless. The problem is the last step: re-quantization snaps every merged weight to the nearest value its NF4 bucket can represent, and that rounding is applied to every weight in every merged layer at once.

    So there is a single mechanism at work here — re-quantization rounding error — not a pile of unrelated failure modes. NF4 and fp16 are not “incompatible” formats; the conversion between them is well defined. What you lose is precision when a 16-bit sum is forced back into NF4’s limited set of representable values.

    PEFT itself flags this rather than forbidding it: merging a LoRA module into a 4-bit linear layer “may get different generations due to rounding errors.” In practice the merged model still runs, but its outputs can shift measurably from the same adapter running unmerged — enough to matter for evaluation or production, and impossible to recover once the merge is saved. Community threads on the PEFT tracker (issue #2321, issue #2105) are users asking why this warning appears and asking for the behavior to be documented, not reports of the model being destroyed. Treat it as a quality regression to design around, not a guaranteed failure.

    Sources: LoRA · Hugging Face, [Warning] `Merge lora module to 4-bit linear may get different generations` · Issue #2321 · huggingface/peft ·, merge_and_unload docs do not clarify behaviour for quantized base models · Issue #2105 · huggingface/peft · Gi

    The Correct Workflow: Reloading in Full Precision Before Merging

    The officially supported path is to merge the adapter into an unquantized 16-bit base model, never into the 4-bit model you trained against.

    1. Load the base model without quantization. Call transformers.AutoModelForCausalLM.from_pretrained with the compute dtype set to torch.bfloat16 or torch.float16 — use dtype= on current Transformers (torch_dtype= still works but is deprecated) — and omit quantization_config / load_in_4bit entirely for this phase. Set device_map="auto", or an explicit CPU/GPU map if memory is tight.
    2. Attach the trained adapter. peft.PeftModel.from_pretrained(base_model, adapter_path) loads the adapter weights onto the unquantized base.
    3. Call merge_and_unload(). This fuses ΔW = (α / r)·BA into the 16-bit base weights and returns a plain Transformers model with no adapter layers left. Useful arguments: safe_merge=True clones each weight matrix one layer at a time to check for NaN before committing that layer (peak overhead is one layer, not a second full model); progressbar=True shows layer-by-layer progress; adapter_names limits the merge to specific adapters.
    4. Save the merged model. model.save_pretrained(output_dir) writes a standalone 16-bit checkpoint that no longer depends on PEFT or bitsandbytes to load.
    5. Re-quantize only if deployment needs it. If you need a 4-bit or 8-bit artifact for inference, quantize this saved 16-bit checkpoint once, after the merge, with a current post-training method (bitsandbytes, torchao, AWQ, or GPTQ). That keeps quantization to a single, measurable step on the final model instead of one buried inside the merge.

    Sources: LoRA · Hugging Face, Quantization · Hugging Face

    System and Memory Requirements for 16-Bit Model Fusion

    The merge arithmetic itself is cheap. The real constraint is holding the base model in 16-bit, since the whole point of the correct workflow is that it is no longer quantized during the merge.

    At bf16/fp16 you need roughly two bytes per parameter, spread across VRAM and system RAM combined, plus headroom for the adapter and for the single weight matrix safe_merge clones per layer as it checks for NaN (one layer at a time, not a full second model). That is several times the footprint of the 4-bit model you trained with, which is the trade-off for avoiding re-quantization.

    You do not need a single GPU large enough to hold the whole model. device_map="auto" offloads layers to CPU RAM, and because the merge is a one-off operation, running it partly or entirely on CPU is fine — fitting matters far more than speed. If system RAM is also tight, pass an explicit device map that keeps most layers on CPU. Keep load_in_4bit off for this whole phase; the low-bit representation for deployment is produced later, from the saved checkpoint.

    Sources: LoRA · Hugging Face

    Comparing Quality: 4-Bit Merge, 16-Bit Merge, and Runtime QLoRA

    Three configurations are worth keeping distinct when you evaluate the result:

    • Naive 4-bit merge — adapter fused into the 4-bit model, with the re-quantization rounding baked into every layer. It runs, but generations can diverge from the trained adapter and the loss is permanent.
    • Proper 16-bit merge — adapter fused into the unquantized base, then saved. Generations track the unmerged adapter up to ordinary floating-point noise, because no re-quantization happens during the merge.
    • Runtime QLoRA inference — base kept in 4-bit, adapter applied on the fly, nothing merged. This is the natural reference point for adapter quality, but it carries adapter overhead on every forward pass and needs bitsandbytes at serving time.

    This is a conceptual comparison, not a numeric benchmark — the right numbers are the ones you measure on your own task. For deployment, do the 16-bit merge first, then quantize the merged checkpoint if you need a low-bit model for inference. Applying quantization once to a clean merged model (bitsandbytes NF4 for a calibration-free load-time path, or AWQ / GPTQ for calibration-based PTQ) gives you one rounding pass you can actually measure, rather than one hidden inside merge_and_unload. Before shipping, validate the deployed model against the runtime-QLoRA setup on your own evaluation set — a single quantization step on the final model is far easier to sign off on than rounding introduced mid-merge.

    Sources: LoRA · Hugging Face

    Closing thoughts

    Merging a QLoRA adapter straight into 4-bit base weights is not a shortcut worth taking. PEFT will do it, but it has to re-quantize every updated weight back into NF4, and that rounding pass — applied across every merged layer — is what pushes the fused model’s outputs away from what you trained.

    The reliable path is boring: load the base model in 16-bit, attach the adapter, run merge_and_unload there, save, and quantize once at the end if deployment needs it. That keeps quantization to a single measurable step on the final model instead of an invisible one inside the merge. When you do need a low-bit artifact, reach for a current post-training quantization method rather than merging inside an already-quantized model.

    Frequently Asked Questions

    What precisions are used for the base model and adapters in QLoRA?

    In QLoRA, the base model weights are stored in 4-bit precision formats like NormalFloat4 (NF4) via bitsandbytes. The trained LoRA adapter matrices are kept in 16-bit floating-point precision, such as torch.float16 or torch.bfloat16.

    Why does calling merge_and_unload directly on a 4-bit model change generation quality?

    PEFT cannot add a 16-bit adapter delta to a 4-bit weight, so it dequantizes each base weight to bf16/fp16, adds the delta, and re-quantizes the result back into NF4. That final re-quantization rounds every merged weight to the nearest NF4 bucket, across every layer at once, and PEFT warns it “may get different generations due to rounding errors.” The dequantize and add steps are lossless; only the re-quantization loses information.

    What stops the LoRA update from integrating exactly during a 4-bit merge?

    Nothing stops the addition itself — it happens in floating point and is accurate. What you lose is precision when the summed weight is forced back into NF4’s limited set of representable values. Merging into an unquantized 16-bit base skips that step entirely, so the adapter integrates without rounding loss.


    Keep reading: Training & Architectures

    All guides in this series: LLM Training & Architecture Guides: Fine-Tuning, RL & New Models

  • Resolving Pydantic State Deserialization Failures in LangGraph Checkpointers

    Resolving Pydantic State Deserialization Failures in LangGraph Checkpointers

    TL;DR

    • Pydantic state deserialization problems in LangGraph checkpointers come from four distinct mechanisms with different symptoms: msgpack allowlist restrictions under strict mode, classes that cannot be re-imported (dynamic or local scope), schema drift between old checkpoints and newer models, and non-serializable attributes that actually fail when the checkpoint is written.
    • The usual symptom is silent degradation, not a raised exception: JsonPlusSerializer returns the raw kwargs dictionary (or a validation-skipped model_construct object) when it cannot rehydrate a model, so failures often surface later as an AttributeError or KeyError inside a node rather than at load time.
    • Fixes are cause-specific: register models in the msgpack allowlist for allowlist blocks, move models to module top level for import failures, add field defaults plus a model_validator or reducer for schema drift, and keep runtime objects out of state fields for the save-time case.

    Understanding LangGraph State Serialization with JsonPlusSerializer

    In LangGraph, state persistence across execution supersteps is managed by checkpointer implementations extending BaseCheckpointSaver, such as MemorySaver, SqliteSaver, and PostgresSaver. These checkpointers delegate data encoding and decoding to serialization protocols located in langgraph.checkpoint.serde. The default serialization engine is JsonPlusSerializer (langgraph.checkpoint.serde.jsonplus.JsonPlusSerializer), which encodes state primarily through msgpack, with a JSON path as fallback, while preserving standard Python types, LangChain and LangGraph primitives, and Pydantic BaseModel instances.

    When a checkpoint is loaded, the serializer rebuilds each custom object from a stored (module, class_name, kwargs) triple. Success depends on two independent conditions: whether the module passes the deserialization allowlist, and whether the class can actually be imported and constructed. By default—when LANGGRAPH_STRICT_MSGPACK is unset—the msgpack policy is permissive (allowed_msgpack_modules=True): unregistered modules are still reconstructed and only a warning is logged. Strict mode, enabled by LANGGRAPH_STRICT_MSGPACK=true or by passing an explicit allowlist, blocks modules that are not listed. The important detail is that blocking does not raise—the serializer returns the raw data instead.

    Common Causes of Deserialization Failures

    Pydantic state deserialization issues within JsonPlusSerializer stem from four distinct mechanisms. They produce different symptoms and require different fixes, so they are worth separating:

    • msgpack Allowlist Restrictions (strict mode only): Under strict mode, JsonPlusSerializer checks each custom object’s module path before instantiating it. If a Pydantic model’s module is not on the allowlist, the msgpack Pydantic branch returns the stored kwargs dictionary unchanged—no exception, and no visible signal unless you are watching the warning log. The node then receives a dict where it expected a model. InvalidModuleError is a real type, but it is raised only on the separate JSON constructor path, and even there JsonPlusSerializer‘s reviver catches it, logs a warning, and falls back to the generic LangChain reviver—so it is not the symptom to code against.
    • Classes That Cannot Be Re-Imported: Rebuilding a class during deserialization requires resolving module.ClassName back to an importable symbol. Models defined inside a function body or closure, or parametrized generic models with no concrete top-level subclass, expose no such symbol. The lookup fails and the value deserializes as a raw dictionary—independent of allowlist configuration or strict mode. Adding the class to the allowlist does not help, because the problem is resolution, not permission.
    • Schema Drift Between Checkpoint and Model: When a state model gains a required field without a default, renames a field, or tightens a type, reading an older checkpoint does not surface a Pydantic ValidationError from the serializer. It tries cls(**kwargs), and on failure falls back to cls.model_construct(**kwargs), which skips validation and returns a partially populated model; if that also fails, it returns the raw dict. A ValidationError appears only later, when a reducer or node re-validates the value.
    • Non-Serializable Attributes (a save-time failure): Placing runtime artifacts that cannot be encoded—raw exception objects, thread locks, open client sessions, file handles—inside Pydantic BaseModel fields makes the checkpoint fail when it is written, not when it is read. Despite the “deserialization” label, the fix belongs on the encoding side: keep these objects out of persisted state.

    Resolution Strategies and Best Practices

    Each strategy targets a specific mechanism above rather than all of them at once.

    • Register Models in the msgpack Allowlist (cause 1): When running with strict msgpack mode, pass custom Pydantic models to allowed_msgpack_modules on JsonPlusSerializer—as class objects or (module, qualname) tuples—or call serde.with_msgpack_allowlist([...]) to obtain a new serializer with a merged allowlist. allowed_msgpack_modules and allowed_json_modules gate two different decode paths; msgpack carries the large majority of state, so it is the one that usually matters. Compiling a StateGraph with strict mode enabled already derives an allowlist from the state, input, output, and context schemas, so—when the chosen checkpointer supports with_allowlist(...)—top-level state models are covered without manual registration:
    from langgraph.checkpoint.serde.jsonplus import JsonPlusSerializer
    from langgraph.checkpoint.sqlite import SqliteSaver
    from my_app.schemas import CustomStateModel
    
    serde = JsonPlusSerializer(
        allowed_msgpack_modules=[CustomStateModel]  # or [("my_app.schemas", "CustomStateModel")]
    )
    checkpointer = SqliteSaver(conn, serde=serde)
    
    • Declare State Models at Top-Level Module Scope (cause 2): Define every Pydantic state schema, sub-model, and dataclass at the top level of an importable module rather than inside a nested function or closure. For generic models in Pydantic v2, subclass BaseModel together with typing.Generic[T] and declare an explicit concrete subclass at module scope, so the stored metadata maps to an importable symbol.
    • Design for Schema Resilience (cause 3): Give every new field a default via Field(default=...) or Field(default_factory=...) so old checkpoints still satisfy the current model. Then use @model_validator(mode="before") or a graph channel reducer to coerce any dictionary fallback payload back into the typed BaseModel before nodes observe it.
    • Keep Runtime Objects Out of State (cause 4): Persist identifiers or serializable configuration instead of live handles, and rebuild the runtime object inside the node. For genuinely custom encoding, implement SerializerProtocol (dumps_typed(obj) -> tuple[str, bytes] and loads_typed(data) -> Any) and pass it as the checkpointer’s serde.

    Sources: Unsafe msgpack deserialization in LangGraph checkpoint loading · Advisory · langchain-ai/langgraph · GitHub, langgraph.checkpoint, base | langgraph.checkpoint | LangChain Reference, SerdeConfig | langgraph_cli | LangChain Reference, jsonplus | langgraph.checkpoint | LangChain Reference

    Closing thoughts

    These four failure modes do not share a single root cause, and it is a mistake to treat them all as the price of security hardening. Only the msgpack allowlist check is a security control. Import-resolution failures are a packaging and code-organization problem; schema drift is a data-contract problem between a stored checkpoint and a newer model; non-serializable attributes are an encoding constraint that shows up before anything is ever read back. What the four have in common is only the outcome: JsonPlusSerializer prefers to hand back a raw dictionary or a validation-skipped object rather than raise, so a broken assumption travels silently into your graph.

    The practical consequence is that resilient graphs treat state models as version-stable data contracts and verify what actually comes out of the checkpointer. Define models at module top level so they can be re-imported, give every new field a default so old checkpoints still load, and add a model_validator or channel reducer that turns any dict fallback back into a typed model before a node touches it. Reserve allowlist configuration for the one case it addresses—strict msgpack mode—and enable strict mode deliberately rather than by accident.

    Frequently Asked Questions

    What are the primary causes of Pydantic state deserialization failures in LangGraph?

    Four distinct mechanisms: (1) under strict msgpack mode, a model whose module is not on the allowlist is returned as a raw dict; (2) classes defined in a function body, closure, or as an unsubclassed generic cannot be re-imported and also come back as dicts; (3) schema drift between an old checkpoint and a newer model, where the serializer falls back to model_construct or a raw dict rather than raising; and (4) non-serializable attributes such as locks, sessions, or exception objects in state fields, which fail when the checkpoint is written.

    What does a deserialization failure actually look like at runtime?

    Usually not an exception. JsonPlusSerializer returns the stored kwargs dictionary, or a model_construct object with validation skipped, when it cannot rehydrate a model. The failure typically surfaces later as an AttributeError or KeyError inside a node that expected a typed model. InvalidModuleError exists but is confined to the JSON constructor path and is caught internally; a ValidationError appears only if a reducer or node re-validates the value.

    How do you register custom Pydantic models for the msgpack allowlist?

    Pass them to allowed_msgpack_modules on JsonPlusSerializer, as class objects or (module, qualname) tuples, or call serde.with_msgpack_allowlist([...]) to get a new serializer with a merged allowlist. When you compile a StateGraph with strict mode enabled, LangGraph already derives an allowlist from the state, input, output, and context schemas, so top-level state models are covered without manual registration—provided the chosen checkpointer supports with_allowlist(...).

    Why must state models be defined at top-level module scope?

    Deserialization resolves a stored module.ClassName back to an importable symbol. Models defined inside nested functions, or generics without a concrete top-level subclass, have no such symbol, so they deserialize as raw dictionaries no matter how the allowlist is configured.

    How do you keep old checkpoints loading after a schema change?

    Add defaults with Field(default=...) or Field(default_factory=...) for every new field, and add a @model_validator(mode="before") or a channel reducer that coerces a plain-dict fallback back into the typed model before nodes run.


    Keep reading: LLM Agents & Tooling

    All guides in this series: LLM Agent Guides: LangChain, LangGraph & AI Tooling

  • Resolving SQLite Database Lock Errors in ChromaDB During Concurrent Writes

    Resolving SQLite Database Lock Errors in ChromaDB During Concurrent Writes

    TL;DR

    • SQLite enforces a global write lock in Embedded ChromaDB, which causes SQLITE_BUSY database lock errors during concurrent write operations and read-to-write lock upgrades.
    • Standard write contention can be mitigated by keeping insert batch sizes between 50 and 250 records and raising SQLite’s own PRAGMA busy_timeout to at least 5 seconds — this is a general SQLite setting, not a parameter ChromaDB itself exposes.
    • Transitioning to ChromaDB’s Client-Server mode using HttpClient or AsyncHttpClient resolves embedded SQLite locking limits by offloading concurrency control to a standalone server process.

    Understanding Embedded ChromaDB and SQLite Persistence Engine

    Embedded ChromaDB handles local database persistence using PersistentClient. This client automatically saves database files to the local machine and reloads them on startup if previous data exists.

    The storage location on disk is configured through the path parameter, which determines where database files are written and loaded from. If no path is explicitly provided, the client defaults to using .chroma.

    The persistence client exposes programmatic methods for system management:

    • heartbeat(): Returns a nanosecond heartbeat value, useful for confirming the client is still connected.
    • reset(): Empties and completely resets the database. Executing this method is destructive and cannot be reversed.

    Sources: docs.trychroma.com — Persistent Client

    Root Causes of SQLite Locking Issues in Concurrent Write Workloads

    Underneath this local persistence mechanism, SQLite manages concurrent write operations by enforcing a global write lock that permits only one writer at a time. When a write transaction begins, it holds this global lock for its entire duration, blocking all other write attempts until the transaction completes.

    The SQLITE_BUSY error (“database is locked”) occurs whenever a transaction cannot acquire this global write lock. If another transaction holds the lock, even a simple insert operation fails immediately with this error.

    A distinct edge case occurs when an active read transaction attempts to upgrade to a write transaction. If another database connection has already modified the database or is in the process of modifying it, this upgrade attempt fails immediately with SQLITE_BUSY. Unlike general write lock contention, this specific read-to-write lock upgrade failure is not helped by setting a busy timeout.

    To address standard write lock contention, SQLite itself — independent of ChromaDB, which does not expose this as a documented client parameter — provides a general pragma, PRAGMA busy_timeout. This sets the duration that transactions will wait to acquire the write lock before returning “database is locked” instead of failing immediately. In production systems, a PRAGMA busy_timeout setting of 5 seconds or more is recommended.

    Sources: tenthousandmeters.com — SQLite concurrent writes and "database is locked" errors

    Mitigating Write Contention Through Batch Size and Understanding SQLite Storage Behavior

    Beyond PRAGMA busy_timeout, how ingestion parameters are configured also shapes how often write locks get contended in the first place. ChromaDB documentation recommends keeping insert batch sizes on the smaller side, specifically between 50 and 250 records. Choosing batch sizes in this range yields lower and more consistent latency while making writes less likely to encounter timeout errors. Although overall throughput plateaus and remains fairly flat across batch sizes between 100 and 500, smaller batches are preferred to prevent latency spikes and timeouts.

    For insert write concurrency, ChromaDB records writes to a log and flushes them every N operations. Because of this logging and flushing mechanism, mean latency does not fluctuate as the number of concurrent writers increases.

    Separately from locking behavior, ChromaDB’s on-disk footprint is worth understanding on its own: the database saves metadata and documents via SQLite, and disk usage is highly variable, depending entirely on whether full documents and metadata are being retained. For example, a sample collection containing approximately 40,000 documents (averaging 1,000 words each) and roughly 600,000 metadata entries requires about 1.7GB of storage. SQLite handles database disk paging effectively and supports database sizes scaling into the terabyte range.

    Sources: docs.trychroma.com — Performance guide (single node)

    Architectural Alternatives: Embedded SQLite vs Client-Server Mode for Multi-Threaded Writes

    When local tuning strategies and parameter adjustments prove insufficient for multi-threaded or multi-process write workloads, ChromaDB supports Client-Server ChromaDB where applications interact with a standalone server process instead of managing local storage. Synchronous access is established using HttpClient, whereas non-blocking access is provided via AsyncHttpClient. The two client implementations maintain identical method signatures and behaviors, but AsyncHttpClient makes all methods that would otherwise block run asynchronously.

    Configuring the standalone server instance is handled through environment variables:

    • CHROMA_PERSIST_PATH: Dictates the directory used for persisted data. This defaults to ./chroma in the frontend configuration and is commonly set to /data in container deployments.
    • CHROMA_SQLITE_FILENAME: Specifies the SQLite database filename created under the persist path, defaulting to chroma.sqlite3.
    • CHROMA_LISTEN_ADDRESS: Sets the bind address for the frontend server, defaulting to 0.0.0.0.
    • CHROMA_PORT: Defines the HTTP listening port for the frontend server, defaulting to 8000.

    Adopting Client-Server ChromaDB provides an architectural alternative to running Embedded ChromaDB directly inside client processes. In one third-party case, developers maintaining the MemPalace project reported encountering chromadb.errors.InternalError: Error in compaction: Failed to apply logs to the hnsw segment writer during concurrent multi-process operation. Their assessment was that Embedded ChromaDB’s SQLite database and HNSW segment writers are inherently not thread- or process-safe for concurrent writers. To address this within their application, they implemented a workaround by switching from the embedded PersistentClient to HttpClient, offloading concurrency and execution control entirely to the server process.

    It’s worth being precise about what this MemPalace failure actually is: an HNSW compaction error is a different failure mode from the SQLITE_BUSY lock errors described earlier in this article. It isn’t SQLite’s write-lock contention — it originates from concurrent access to the HNSW vector-index segment writer instead. PRAGMA busy_timeout has no bearing on it. The two problems just happen to share the same fix here (moving to Client-Server mode), not the same underlying mechanism.

    Sources: docs.trychroma.com — Client-Server mode (HttpClient / AsyncHttpClient), docs.trychroma.com — Server environment variables, MemPalace/mempalace issue #832

    Closing thoughts

    Ultimately, while Embedded ChromaDB provides a convenient local persistence mechanism, attempting to push it through heavy concurrent write workloads reveals clear architectural limitations inherent to embedded SQLite. Keeping batch sizes between 50 and 250 records, and raising SQLite’s own PRAGMA busy_timeout (a general SQLite setting, not something ChromaDB exposes as a client parameter), can alleviate standard SQLite write contention, but neither helps with unhandled read-to-write lock upgrades, and neither has any bearing on HNSW compaction failures — a separate concurrency issue in the HNSW segment writer, illustrated so far only by one third party’s reported case, rather than a SQLite locking problem. In my judgment, pushing heavy concurrent write workloads through Embedded ChromaDB carries stability risk on both fronts — SQLite lock contention and (per that one reported case) HNSW segment-writer safety alike. Transitioning to ChromaDB’s Client-Server mode using HttpClient or AsyncHttpClient remains the most definitive solution: it doesn’t make the underlying SQLite database disappear (the server itself still persists to chroma.sqlite3), but it stops multiple application processes from contending directly over the same local SQLite file, and in the MemPalace case, moving to HttpClient was also reported to resolve the HNSW segment-writer failure.

    Frequently Asked Questions

    What causes the SQLITE_BUSY database locked error in ChromaDB?

    The SQLITE_BUSY error occurs when a transaction cannot acquire SQLite’s global write lock because another transaction is holding it. It also happens when an active read transaction attempts to upgrade to a write transaction while another connection is modifying the database.

    What batch size is recommended for ChromaDB insert operations?

    ChromaDB documentation recommends keeping insert batch sizes between 50 and 250 records. This range yields lower, more consistent latency and makes writes less likely to hit timeout errors.

    How does PRAGMA busy_timeout help handle SQLite write lock contention?

    PRAGMA busy_timeout sets the duration that transactions will wait to acquire the write lock before failing with a database locked error. A setting of 5 seconds or more is recommended in production systems.

    How does switching to Client-Server mode solve embedded SQLite locking limitations?

    Transitioning to Client-Server mode offloads concurrency and execution control to a dedicated server process using HttpClient or AsyncHttpClient. This architecture bypasses the embedded SQLite locking constraints and thread-safety limitations.


    Keep reading: LLM Agents & Tooling

    All guides in this series: LLM Agent Guides: LangChain, LangGraph & AI Tooling