Ollama: Roo overrides Modelfile num_ctx by forcing options.num_ctx from modelInfo.contextWindow

[hannesrudolph]

Ollama: Roo overrides Modelfile num_ctx by forcing options.num_ctx from modelInfo.contextWindow

App Version
v3.27.0 (main 0ce4e89)

API Provider
Ollama

Model Used
Any Ollama model with a Modelfile num_ctx smaller than Roo’s inferred context (e.g., llama3.1 with num_ctx 4096)

Steps to Reproduce

1. Create an Ollama model with a reduced context:
   Modelfile:

   FROM llama3.1
   PARAMETER num_ctx 4096
   

   Build and start: ollama create my-small-ctx -f Modelfile then ensure Ollama is running.
2. In Roo Code, set provider to “Ollama” and select model my-small-ctx.
3. Start a chat and send a prompt large enough to exceed the Modelfile’s num_ctx (e.g., >4k tokens).
4. Observe GPU/CPU memory usage and/or server behavior.

Outcome Summary
Roo sends options.num_ctx equal to its inferred modelInfo.contextWindow, overriding the Modelfile’s num_ctx. This can allocate a much larger context window than intended (e.g., 128k/200k), causing failures or OOM on budget GPUs.

Technical Evidence

• Roo sets num_ctx on chat requests in native Ollama handler:
  • Request creation at src/api/providers/native-ollama.ts:186
  • num_ctx assignment at src/api/providers/native-ollama.ts:193
• modelInfo.contextWindow is inferred from Ollama /api/show via parseOllamaModel()
  • contextWindow assignment at src/api/providers/fetchers/ollama.ts:47
  • maxTokens mirror at src/api/providers/fetchers/ollama.ts:51
• Default fallback is ollamaDefaultModelInfo with contextWindow 200,000
• Non‑streaming path completePrompt() does not include num_ctx
• Provider selection routes “ollama” → NativeOllamaHandler

Expected Behavior

• Roo should not force num_ctx unless the user explicitly configures it.
• By default, Ollama should honor the Modelfile’s num_ctx or server-side defaults.

Actual Behavior

• Roo overrides the Modelfile by sending options.num_ctx = modelInfo.contextWindow on each chat request, ignoring user-defined num_ctx in the Modelfile.

Proposed Fix (non‑breaking)

• Remove num_ctx from default chat options in NativeOllamaHandler.createMessage() so Ollama respects the Modelfile.
• Add an explicit optional override (e.g., ApiHandlerOptions.ollamaNumCtx?: number) in ApiHandlerOptions; include num_ctx only when set.
• Optionally improve display-only inference in parseOllamaModel() to prefer a configured num_ctx (e.g., from parameters or model_info.num_ctx), but do not enforce it in requests.

Notes

• Matches reports of unexpected large context allocation on consumer GPUs.
• Opt‑in num_ctx keeps power users’ control while respecting defaults for everyone else.

Related

• Ollama model num_ctx is being ignored in the newest version #7159
• fix: do not override Ollama model num_ctx configuration #7160
• fix: respect user-configured num_ctx in Ollama modelfiles #7342

[roomote]

I'm exploring a potential fix for this issue where Roo overrides Ollama's Modelfile num_ctx setting. I'll work on implementing the proposed solution to respect the Modelfile's configuration by default while allowing optional overrides.

[roomote]

I've opened PR #7798 with a fix for this issue: #7798

The implementation:

• Removes the automatic num_ctx override in NativeOllamaHandler
• Adds an optional ollamaNumCtx parameter to ApiHandlerOptions for explicit overrides when needed
• Updates both streaming and non-streaming paths to respect Modelfile configuration by default
• Includes comprehensive test coverage

All CI checks have passed. This change allows Ollama to use the Modelfile-defined num_ctx by default while still providing users the ability to override it when needed. Feedback welcome!