Tag: Concurrent Writes

  • Troubleshooting LangGraph InvalidUpdateError: Resolving Concurrent State Updates with Annotated Reducers

    Troubleshooting LangGraph InvalidUpdateError: Resolving Concurrent State Updates with Annotated Reducers

    TL;DR

    • LangGraph raises InvalidUpdateError when multiple concurrent nodes write to the same state key in a single step without an aggregation rule.
    • To intentionally combine concurrent updates, define the state key using typing.Annotated and a reducer function such as operator.add.
    • Unintentional collisions can be prevented by assigning dedicated state keys to parallel branches or routing them into downstream fan-in nodes.

    Diagnosing InvalidUpdateError and the INVALID_CONCURRENT_GRAPH_UPDATE Error

    In LangGraph, conflicting writes during graph execution surface as the API class langgraph.errors.InvalidUpdateError. In LangGraph’s documentation, this issue is tracked under the standard error code INVALID_CONCURRENT_GRAPH_UPDATE.

    The runtime throws this error when multiple parallel nodes (such as Fan-out branches executing concurrently in the same super-step) return an update dictionary targeting the exact same state key. Because LangGraph cannot determine how to resolve the conflicting incoming values for a channel designed to handle only a single write per step, it halts execution to prevent silent data loss or non-deterministic overwrites.

    The exception produces the following message and troubleshooting reference:

    langgraph.errors.InvalidUpdateError: At key '<key>': Can receive only one value per step. Use an Annotated key to handle multiple values.
For troubleshooting, visit: https://python.langchain.com/docs/troubleshooting/errors/INVALID_CONCURRENT_GRAPH_UPDATE

    (Note: Runtime exceptions may print the legacy troubleshooting URL python.langchain.com/..., which corresponds to the current documentation hosted under docs.langchain.com/....)

    This traceback indicates that two or more concurrently executing nodes attempted to write to <key> within the exact same super-step without an aggregation rule.

    When diagnosing this error, confirm whether the concurrent writes to the same key are intentional or the result of graph design:

    • Unintentional Key Collisions: If parallel branches produce distinct results that should not be combined into the same field, alter the graph structure to use dedicated state keys (such as branch_a_output and branch_b_output). Alternatively, route edges from the parallel nodes into a downstream convergence or fan-in node that consumes both results and applies the unified update in a subsequent step.
    • Intentional Multi-Branch Updates: If multiple parallel branches must write to the same collection or field simultaneously, the target key must be declared using typing.Annotated with an appropriate Reducer function to dictate how the concurrent values should be combined.

    Sources: INVALID_CONCURRENT_GRAPH_UPDATE – Docs by LangChain, InvalidUpdateError, Errors – Docs by LangChain, GitHub Issue Title: InvalidUpdateError when using parallel subgraphs with shared state keys in LangGraph · Iss, How to modify specific item in a State – LangGraph – LangChain Forum, docs(langgraph): fixes fan-out example

    The Pregel Execution Model: Why Concurrent Updates Halt the Tick Loop

    To understand why this failure occurs under the hood, it helps to examine how the Pregel execution model processes state within a super-step. Specifically, an INVALID_CONCURRENT_GRAPH_UPDATE error arises when a StateGraph receives concurrent updates to a state property that does not support them, such as during a Fan-out or other parallel execution where multiple nodes return values for the same key within a single step. While a single node returning a value overwrites that key in the state, multiple nodes returning values for the same key create uncertainty about how to update the internal state, causing the graph to raise an error.

    For example, executing a graph via graph.stream() under these conditions can result in a langgraph.errors.InvalidUpdateError:

    langgraph.errors.InvalidUpdateError: At key 'jokes': Can receive only one value per step. Use an Annotated key to handle multiple values.
For troubleshooting, visit: https://python.langchain.com/docs/troubleshooting/errors/INVALID_CONCURRENT_GRAPH_UPDATE

    To prevent or resolve this error when executing nodes in parallel, relevant state keys can be defined with a Reducer function using typing.Annotated (such as Annotated[list, operator.add]), which specifies the logic for combining multiple values.

    Sources: INVALID_CONCURRENT_GRAPH_UPDATE – Docs by LangChain, Fix to graph-api docs Send API example

    Default Overwrite Channels vs. Reducer Channels

    This distinction highlights the core operational difference between default overwrite channels and reducer-enabled channels. By default, fields defined without an annotation in a TypedDict, dataclass, or Pydantic model operate as single-value channels backed internally by LastValue. When such a channel is configured without a Reducer function, receiving multiple update values in a single step causes LangGraph to raise an InvalidUpdateError:

    langgraph.errors.InvalidUpdateError: At key '<key>': Can receive only one value per step. Use an Annotated key to handle multiple values.

    To handle multiple values, fields can be configured with Reducer functions using typing.Annotated (such as Annotated[int, operator.add]).

    The presence or absence of a Reducer function also affects how bugs surface during execution. In one reported open issue (#9005), a developer observed a framework bug where overriding an inherited field in a dataclass or Pydantic subclass caused Command(update=...) to apply the update twice in a single step. When the overridden field (count) was configured without a Reducer function, this duplicate application triggered:

    langgraph.errors.InvalidUpdateError: At key 'count': Can receive only one value per step. Use an Annotated key to handle multiple values.

    However, when the field configured a Reducer function such as Annotated[int, operator.add], no error was raised, and the framework bug caused the reducer to apply the update twice (yielding a count of 5 instead of 3 from an initial value of 1 and an update of 2). Here, the reducer is not the source of the issue, but its normal aggregation behavior masks the framework’s double-invocation bug.

    Sources: `Command(update=…)` applies an overridden inherited state field twice, Receiving multiple inputs on a single node not working in a vanilla state

    Configuring Annotated State Keys with operator.add and Custom Reducers

    Building on this reducer mechanism, state schemas in LangGraph can use typing.Annotated within a TypedDict to specify a standard Reducer function such as operator.add:

    from typing import Annotated, Sequence
import operator
from typing_extensions import TypedDict

class AgentState(TypedDict):
    messages: Annotated[Sequence[dict], operator.add]

class State(TypedDict):
    my_key: Annotated[str, operator.add]

    When multiple parallel upstream nodes (such as up and side in a topology like graph.add_edge(["up", "side"], "down")) execute concurrently in the same super-step and both write updates to the same state key, a reducer is required on that key to prevent update conflicts across those parallel branches before control reaches the downstream down node.

    When list merging requires avoiding duplicate entries, custom reducer functions can be defined. For example, in a discussion around handling subgraph outputs, a community workaround demonstrated a custom reducer:

    def reduce_results(left: list[str], right: list[str]) -> list[str]:
    """This approach might not be the best, but it can solve repeat problems."""
    left_table = {result: i for i, result in enumerate(left)}
    merged = left.copy()
    for right_result in right:
        if (existing_idx := left_table.get(right_result)) is not None:
            merged[existing_idx] = right_result
        else:
            merged.append(right_result)
    return merged

class ParentState(TypedDict):
    question: str
    results: Annotated[list[str], reduce_results]

    In that same discussion, a contributor suggested separating input and output schemas when handling subgraphs: using the parent graph’s state for the input allows full access to parent state values, while the output schema contains only the required fields, typed as lists with an Annotated Reducer function.

    Sources: Sink node issue, if multiple subgraphs are used in parallel, two node flow to one node

    Edge Cases: Parallel Subgraphs, Conditional Self-Loops, and Shared State Keys

    Beyond standard fan-out configurations, several distinct edge cases and open bug reports illustrate scenarios where InvalidUpdateError surfaces unexpectedly. These scenarios stem from different factors and should not be conflated:

    In an open bug report (issue #6446), executing a compiled subgraph in parallel with another node in the parent graph resulted in an unexpected InvalidUpdateError. In the reporter’s minimal reproducer, a parent graph branched from an initial node to both a standard node (nodeb) and a compiled subgraph:

    def create_graph():
    builder = StateGraph(State)
    builder.add_node("nodea", nodea)
    builder.add_node("subgraph", subgraph())
    builder.add_node("nodeb", nodeb)

    builder.add_edge(START, "nodea")
    builder.add_edge("nodea", "nodeb")
    builder.add_edge("nodea", "subgraph")
    builder.add_edge("nodeb", END)
    builder.add_edge("subgraph", END)
    return builder.compile()

    When nodeb and subgraph executed concurrently, LangGraph raised:

    langgraph.errors.InvalidUpdateError: At key 'b': Can receive only one value per step. Use an Annotated key to handle multiple values.

    The reporter noted that nodeb only updated key b while the subgraph internal nodes only updated keys s1 and s2. However, the root cause has not been officially confirmed by maintainers, and the report remains an open bug rather than an established graph behavior.

    In another scenario (issue #2336), a user encountered an update collision when combining parallel branches with a conditional self-loop (not a RetryPolicy). An initial node branched into parallel nodes b and c, where node c evaluated a conditional edge that routed back to itself on a retry condition before proceeding downstream. Executing this flow produced InvalidUpdateError targeting key count_b. Because the issue was closed without an official explanation or confirmed fix, it is only presumed that a self-loop iteration coincided with another branch in the same super-step to cause multiple writes to count_b.

    Sources: GitHub Issue Title: InvalidUpdateError when using parallel subgraphs with shared state keys in LangGraph, langgraph.errors.InvalidUpdateError: Can receive only one value per step. Use an Annotated key to handle multiple values. error when running retry in graph parallel processing

    Closing thoughts

    In standard graph workflows, InvalidUpdateError serves as an essential safeguard within LangGraph’s Pregel model, preventing non-deterministic data overwrites when multiple concurrent nodes write to the same unannotated channel. When parallel writes to a shared key are intentional, configuring the field with an Annotated reducer provides a deterministic aggregation rule. For unintentional collisions, architectural adjustments such as dedicated state keys or downstream fan-in nodes cleanly separate updates across steps.

    However, reducers should not be treated as a blanket remedy. In specific bug paths such as subclass field inheritance with Command(update=...) (#9005), adding a reducer can mask duplicate updates that would otherwise trigger a visible error. Furthermore, edge cases like parallel subgraphs (#6446) and conditional self-loops (#2336) represent distinct issues with separate, unconfirmed causes rather than straightforward topology mistakes. Distinguishing between intended concurrent updates, graph architectural patterns, and unresolved framework bugs is crucial when troubleshooting update conflicts in LangGraph.

    Frequently Asked Questions

    Why does LangGraph raise an InvalidUpdateError?

    LangGraph raises this error when multiple parallel nodes attempt to update the exact same state key within the same super-step without an aggregation rule. It halts execution to prevent non-deterministic overwrites and silent data loss.

    How can you resolve InvalidUpdateError when multiple branches intentionally update the same key?

    You can declare the target state key using typing.Annotated along with a Reducer function, such as operator.add or a custom reducer. This tells LangGraph how to combine the conflicting concurrent values instead of attempting an overwrite.

    What should you do if parallel nodes are causing unintentional key collisions?

    You can alter the graph structure to give each branch its own dedicated state key, such as branch_a_output and branch_b_output. Alternatively, you can route the parallel nodes to a downstream fan-in node that aggregates the values in a subsequent step.


    Keep reading: LLM Agents & Tooling

    All guides in this series: LLM Agent Guides: LangChain, LangGraph & AI Tooling

  • Resolving SQLite Database Lock Errors in ChromaDB During Concurrent Writes

    Resolving SQLite Database Lock Errors in ChromaDB During Concurrent Writes

    TL;DR

    • SQLite enforces a global write lock in Embedded ChromaDB, which causes SQLITE_BUSY database lock errors during concurrent write operations and read-to-write lock upgrades.
    • Standard write contention can be mitigated by keeping insert batch sizes between 50 and 250 records and raising SQLite’s own PRAGMA busy_timeout to at least 5 seconds — this is a general SQLite setting, not a parameter ChromaDB itself exposes.
    • Transitioning to ChromaDB’s Client-Server mode using HttpClient or AsyncHttpClient resolves embedded SQLite locking limits by offloading concurrency control to a standalone server process.

    Understanding Embedded ChromaDB and SQLite Persistence Engine

    Embedded ChromaDB handles local database persistence using PersistentClient. This client automatically saves database files to the local machine and reloads them on startup if previous data exists.

    The storage location on disk is configured through the path parameter, which determines where database files are written and loaded from. If no path is explicitly provided, the client defaults to using .chroma.

    The persistence client exposes programmatic methods for system management:

    • heartbeat(): Returns a nanosecond heartbeat value, useful for confirming the client is still connected.
    • reset(): Empties and completely resets the database. Executing this method is destructive and cannot be reversed.

    Sources: docs.trychroma.com — Persistent Client

    Root Causes of SQLite Locking Issues in Concurrent Write Workloads

    Underneath this local persistence mechanism, SQLite manages concurrent write operations by enforcing a global write lock that permits only one writer at a time. When a write transaction begins, it holds this global lock for its entire duration, blocking all other write attempts until the transaction completes.

    The SQLITE_BUSY error (“database is locked”) occurs whenever a transaction cannot acquire this global write lock. If another transaction holds the lock, even a simple insert operation fails immediately with this error.

    A distinct edge case occurs when an active read transaction attempts to upgrade to a write transaction. If another database connection has already modified the database or is in the process of modifying it, this upgrade attempt fails immediately with SQLITE_BUSY. Unlike general write lock contention, this specific read-to-write lock upgrade failure is not helped by setting a busy timeout.

    To address standard write lock contention, SQLite itself — independent of ChromaDB, which does not expose this as a documented client parameter — provides a general pragma, PRAGMA busy_timeout. This sets the duration that transactions will wait to acquire the write lock before returning “database is locked” instead of failing immediately. In production systems, a PRAGMA busy_timeout setting of 5 seconds or more is recommended.

    Sources: tenthousandmeters.com — SQLite concurrent writes and "database is locked" errors

    Mitigating Write Contention Through Batch Size and Understanding SQLite Storage Behavior

    Beyond PRAGMA busy_timeout, how ingestion parameters are configured also shapes how often write locks get contended in the first place. ChromaDB documentation recommends keeping insert batch sizes on the smaller side, specifically between 50 and 250 records. Choosing batch sizes in this range yields lower and more consistent latency while making writes less likely to encounter timeout errors. Although overall throughput plateaus and remains fairly flat across batch sizes between 100 and 500, smaller batches are preferred to prevent latency spikes and timeouts.

    For insert write concurrency, ChromaDB records writes to a log and flushes them every N operations. Because of this logging and flushing mechanism, mean latency does not fluctuate as the number of concurrent writers increases.

    Separately from locking behavior, ChromaDB’s on-disk footprint is worth understanding on its own: the database saves metadata and documents via SQLite, and disk usage is highly variable, depending entirely on whether full documents and metadata are being retained. For example, a sample collection containing approximately 40,000 documents (averaging 1,000 words each) and roughly 600,000 metadata entries requires about 1.7GB of storage. SQLite handles database disk paging effectively and supports database sizes scaling into the terabyte range.

    Sources: docs.trychroma.com — Performance guide (single node)

    Architectural Alternatives: Embedded SQLite vs Client-Server Mode for Multi-Threaded Writes

    When local tuning strategies and parameter adjustments prove insufficient for multi-threaded or multi-process write workloads, ChromaDB supports Client-Server ChromaDB where applications interact with a standalone server process instead of managing local storage. Synchronous access is established using HttpClient, whereas non-blocking access is provided via AsyncHttpClient. The two client implementations maintain identical method signatures and behaviors, but AsyncHttpClient makes all methods that would otherwise block run asynchronously.

    Configuring the standalone server instance is handled through environment variables:

    • CHROMA_PERSIST_PATH: Dictates the directory used for persisted data. This defaults to ./chroma in the frontend configuration and is commonly set to /data in container deployments.
    • CHROMA_SQLITE_FILENAME: Specifies the SQLite database filename created under the persist path, defaulting to chroma.sqlite3.
    • CHROMA_LISTEN_ADDRESS: Sets the bind address for the frontend server, defaulting to 0.0.0.0.
    • CHROMA_PORT: Defines the HTTP listening port for the frontend server, defaulting to 8000.

    Adopting Client-Server ChromaDB provides an architectural alternative to running Embedded ChromaDB directly inside client processes. In one third-party case, developers maintaining the MemPalace project reported encountering chromadb.errors.InternalError: Error in compaction: Failed to apply logs to the hnsw segment writer during concurrent multi-process operation. Their assessment was that Embedded ChromaDB’s SQLite database and HNSW segment writers are inherently not thread- or process-safe for concurrent writers. To address this within their application, they implemented a workaround by switching from the embedded PersistentClient to HttpClient, offloading concurrency and execution control entirely to the server process.

    It’s worth being precise about what this MemPalace failure actually is: an HNSW compaction error is a different failure mode from the SQLITE_BUSY lock errors described earlier in this article. It isn’t SQLite’s write-lock contention — it originates from concurrent access to the HNSW vector-index segment writer instead. PRAGMA busy_timeout has no bearing on it. The two problems just happen to share the same fix here (moving to Client-Server mode), not the same underlying mechanism.

    Sources: docs.trychroma.com — Client-Server mode (HttpClient / AsyncHttpClient), docs.trychroma.com — Server environment variables, MemPalace/mempalace issue #832

    Closing thoughts

    Ultimately, while Embedded ChromaDB provides a convenient local persistence mechanism, attempting to push it through heavy concurrent write workloads reveals clear architectural limitations inherent to embedded SQLite. Keeping batch sizes between 50 and 250 records, and raising SQLite’s own PRAGMA busy_timeout (a general SQLite setting, not something ChromaDB exposes as a client parameter), can alleviate standard SQLite write contention, but neither helps with unhandled read-to-write lock upgrades, and neither has any bearing on HNSW compaction failures — a separate concurrency issue in the HNSW segment writer, illustrated so far only by one third party’s reported case, rather than a SQLite locking problem. In my judgment, pushing heavy concurrent write workloads through Embedded ChromaDB carries stability risk on both fronts — SQLite lock contention and (per that one reported case) HNSW segment-writer safety alike. Transitioning to ChromaDB’s Client-Server mode using HttpClient or AsyncHttpClient remains the most definitive solution: it doesn’t make the underlying SQLite database disappear (the server itself still persists to chroma.sqlite3), but it stops multiple application processes from contending directly over the same local SQLite file, and in the MemPalace case, moving to HttpClient was also reported to resolve the HNSW segment-writer failure.

    Frequently Asked Questions

    What causes the SQLITE_BUSY database locked error in ChromaDB?

    The SQLITE_BUSY error occurs when a transaction cannot acquire SQLite’s global write lock because another transaction is holding it. It also happens when an active read transaction attempts to upgrade to a write transaction while another connection is modifying the database.

    What batch size is recommended for ChromaDB insert operations?

    ChromaDB documentation recommends keeping insert batch sizes between 50 and 250 records. This range yields lower, more consistent latency and makes writes less likely to hit timeout errors.

    How does PRAGMA busy_timeout help handle SQLite write lock contention?

    PRAGMA busy_timeout sets the duration that transactions will wait to acquire the write lock before failing with a database locked error. A setting of 5 seconds or more is recommended in production systems.

    How does switching to Client-Server mode solve embedded SQLite locking limitations?

    Transitioning to Client-Server mode offloads concurrency and execution control to a dedicated server process using HttpClient or AsyncHttpClient. This architecture bypasses the embedded SQLite locking constraints and thread-safety limitations.


    Keep reading: LLM Agents & Tooling

    All guides in this series: LLM Agent Guides: LangChain, LangGraph & AI Tooling