Tag: OpenAI API

Understanding Silent Context Truncation in Ollama

In LLM serving frameworks like Ollama, context window size represents the maximum number of tokens (comprising both the input prompt and output response) that a model can retain in memory at a given time. When applications send requests to Ollama’s OpenAI-compatible API endpoint v1/chat/completions, input sequences that exceed the model’s assigned context window trigger silent context truncation. Instead of throwing an explicit error, Ollama trims earlier messages or system prompts to fit within the allocated context window, which can cause subtle response quality degradation or lost instructions.

This behavior is primarily driven by API schema discrepancies and default runtime boundaries:

• API Schema Discrepancy: While native Ollama endpoints (/api/generate or /api/chat) accept context window configuration directly via the num_ctx parameter, the standard OpenAI-compatible API schema does not natively support num_ctx. Standard parameters like max_tokens control output generation limits rather than input context allocation. Consequently, requests routed through v1/chat/completions rely on the model’s defined context window or global server settings unless overridden in a custom model.
• Default Context Limits: To prevent out-of-memory (OOM) errors, Ollama enforces default context window limits—traditionally 2048 or 4096 tokens, or variable limits based on available GPU VRAM. Even if a model natively supports a larger context window (such as 32k or 128k tokens), running it without explicit configuration binds it to a smaller runtime default.

Several diagnostic tools can be used to inspect active context window bounds and detect truncation events:

• Server Debug Mode: Launching the server with OLLAMA_DEBUG=1 ollama serve enables verbose logging. The server logs will explicitly flag truncation with messages such as truncating input prompt or truncating input messages which exceed context length.
• Runtime Verification: Running ollama ps in the CLI shows currently loaded models alongside their actively allocated memory bounds under the CONTEXT column.
• Modelfile Inspection: Running ollama show --modelfile <model_name> inspects a model’s base configuration to verify whether a default PARAMETER num_ctx has been saved.

To adjust and expand the active context window for OpenAI-compatible API client integrations, context window size can be configured via a custom model Modelfile or a server-wide environment variable.

Creating a custom model via a Modelfile is the recommended method for OpenAI-compatible API compatibility:

FROM llama3.2
PARAMETER num_ctx 16384

After building the model with ollama create my-custom-model, specify "model": "my-custom-model" in v1/chat/completions API calls.

Alternatively, setting the OLLAMA_CONTEXT_LENGTH environment variable before starting the server establishes a global context window default for loaded models:

OLLAMA_CONTEXT_LENGTH=32768 ollama serve

For native Ollama endpoints (/api/generate, /api/chat), runtime context window configurations resolve according to the following priority hierarchy — note this num_ctx request parameter is NOT available on the OpenAI-compatible v1/chat/completions endpoint, which can only be configured via the Modelfile or environment variable below:

Native API Request Parameters (num_ctx, native endpoints only) > Modelfile (PARAMETER num_ctx) > Environment Variable (OLLAMA_CONTEXT_LENGTH) > Default Allocation

Sources: ollama.com, serverman.co.uk, medium.com, reddit.com, ollama.com

Closing thoughts

Ultimately, while Ollama’s OpenAI-compatible API offers seamless integration, its conservative runtime defaults create a subtle trap where longer prompts are quietly trimmed without raising explicit errors. Relying on out-of-the-box settings is risky because standard OpenAI parameters like max_tokens affect output generation rather than expanding Ollama’s input context window. In my view, explicitly configuring context limits via custom Modelfiles or the OLLAMA_CONTEXT_LENGTH environment variable should be treated as a mandatory setup step for any production v1/chat/completions integration. Actively navigating this configuration hierarchy and validating allocated limits through debug tools is essential to prevent invisible prompt degradation while maintaining system stability.

Frequently Asked Questions

What is silent context truncation in Ollama?

Silent context truncation occurs when input sequences exceed a model’s allocated context window size. Instead of throwing an explicit error, Ollama trims earlier messages or system prompts to fit within the available context window.

Why does silent context truncation happen with Ollama’s OpenAI-compatible API?

The standard OpenAI-compatible API schema does not natively support Ollama’s num_ctx configuration parameter, and parameters like max_tokens only restrict output length. Consequently, requests default to conservative server or runtime context limits unless overridden.

How can I detect if context truncation is occurring in Ollama?

You can start the server with OLLAMA_DEBUG=1 to view verbose logs that explicitly flag truncation events. Additionally, running ollama ps shows loaded context bounds, and ollama show –modelfile displays default model parameters.

How do I expand the context window size for OpenAI-compatible API calls in Ollama?

You can define PARAMETER num_ctx in a custom model’s Modelfile or set the OLLAMA_CONTEXT_LENGTH environment variable before starting the Ollama server.