feat(onboard): warn on Ollama models without tools capability (#2667)

[TonyLuo-NV]

Summary

Fixes #2667 — users who picked an Ollama model without tool-calling support (e.g. `phi4`) hit `HTTP 400 ... does not support tools` the first time the agent invoked a tool, with no early signal during onboarding that the chosen model would not work.

This adds an onboard-time capability check plus a friendlier runtime mapping so users find out as early as possible and get an actionable hint either way.

What changed

Onboard-time check (primary fix)

• New `probeOllamaModelCapabilities` POSTs `/api/show` against the local Ollama daemon and parses the `capabilities` array. Defensive parsing returns "unknown" on any failure (HTTP error, empty body, malformed JSON, missing field) so we never false-positive-block.
• New `checkOllamaModelToolSupport` runs inside `prepareOllamaModel` after the pull and before warmup. When the model lacks `tools`, it prints a warning naming the model and tools-capable alternatives, then branches:
  • `--yes` / `NEMOCLAW_YES=1`: continues with a stderr note
  • non-interactive default: blocks with actionable error pointing at the override env var
  • non-interactive + `NEMOCLAW_OLLAMA_REQUIRE_TOOLS=0`: continues with stderr warning
  • interactive: prompts `Use this model anyway? [y/N]` (default N)
• `NEMOCLAW_OLLAMA_REQUIRE_TOOLS=0` is the documented escape hatch (default `1`: block).

Runtime fallback

• `validateOllamaModel` now detects `/does not support tools/i` in the `/api/generate` error payload and rewrites the user-facing message to name the model and recommend a replacement — covers the case where capability metadata says tools but generation still fails.

No new CLI flags. One env var: `NEMOCLAW_OLLAMA_REQUIRE_TOOLS` (default `1`).

Architectural notes

• The host-side onboard probes Ollama on `localhost:11434` directly (network policy preset is irrelevant — onboard runs on host before the sandbox exists).
• The new check fits the existing `selectAndValidateOllamaModel` loop contract — `prepareOllamaModel` returning `{ok:false}` already loops back to model selection.
• Backward-compatible: existing onboarded sandboxes are untouched. Only fires during fresh model selection.
• Custom Ollama registries / older daemons without `/api/show` capabilities → "unknown" → graceful proceed (never blocks on probe failure).

Test plan

• `npx vitest run test/ollama-tools-capability.test.ts` — 13/13 covering the 13 architect-specified scenarios (tools=true/false/null, malformed JSON, correct curl argv, runtime error rewrite, interactive yes/no, non-interactive default + override, NEMOCLAW_YES, graceful unknown-degradation).
• Broader Ollama suite (`get-ollama-model-options`, `ollama-pull-timeout`, `ollama-proxy-recovery`, `ollama-gpu-cleanup`, `onboard-selection`, plus new file): 70/70 passing.
• `npm run typecheck:cli` passes.
• Security review: 9/9 PASS (no secrets, injection-safe via argv `spawnSync`, JSON parse wrapped, no SSRF — host comes from internal resolver, no TLS skipped).
• CI green.
• Manual repro on Ubuntu 22.04 with phi4 (deferred to maintainer review env).

Signed-off-by: Tony Luo xialuo@nvidia.com

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Automatic tool-capability detection for Ollama models before loading.
  • Pre-load gating that checks model tool support and can prompt, warn, or block loading.
  • Non-interactive and auto-approve overrides for different environments.
  • Improved user-facing guidance and error messages suggesting tool-capable alternatives.

• Tests

  • Added comprehensive tests for capability probing, prompts, override flows, and graceful degradation.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ed5101fd-a571-4966-8639-89b9f07ed776

📥 Commits

Reviewing files that changed from the base of the PR and between 353effc and 1d60c8c.

📒 Files selected for processing (3)

• src/lib/local-inference.ts
• src/lib/onboard-ollama-proxy.ts
• test/ollama-tools-capability.test.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• test/ollama-tools-capability.test.ts

---

📝 Walkthrough

Walkthrough

Adds Ollama tool-capability detection and gating: a /api/show probe returning a new OllamaCapabilities shape and exported probeOllamaModelCapabilities(), a probe-driven checkOllamaModelToolSupport() gating flow in the onboard proxy (with interactive/env overrides), and tests for probe and gating behavior.

Changes

Ollama Tool-Capability Detection

Layer / File(s)	Summary
Data Shape / Exports src/lib/local-inference.ts	Adds OllamaCapabilities interface and exports probeOllamaModelCapabilities.
Core Capability Probe src/lib/local-inference.ts	Implements probeOllamaModelCapabilities() that POSTs to /api/show, parses response into OllamaCapabilities, sets supportsTools (true/false/null), and maps known "does not support tools" errors into a user-facing suggestion message or rawError.
Tool Support Gating & Helpers src/lib/onboard-ollama-proxy.ts	Adds isProxyNonInteractive(), isProxyAutoYes(), promptProxyYesNo(), printToolsIncompatibleWarning() and checkOllamaModelToolSupport() which calls the probe, respects NEMOCLAW_OLLAMA_REQUIRE_TOOLS and interactive/auto-yes modes, and returns { ok, message }.
Model Loading Integration src/lib/onboard-ollama-proxy.ts	prepareOllamaModel() now invokes checkOllamaModelToolSupport(model) before pull/load and aborts with the returned message if the check fails. Exports checkOllamaModelToolSupport.
Tests & Verification test/ollama-tools-capability.test.ts	New test suite validating probeOllamaModelCapabilities() parsing, curl argv construction, error mapping, and checkOllamaModelToolSupport() interactive/non-interactive flows, env overrides, and graceful degradation. Includes shared stubs and a proxy harness for module patching.

Sequence Diagram

sequenceDiagram
    actor User
    participant TUI
    participant OnboardProxy as onboard-ollama-proxy
    participant LocalInf as local-inference
    participant OllamaAPI as Ollama /api/show

    User->>TUI: Request load model
    TUI->>OnboardProxy: prepareOllamaModel(model)
    OnboardProxy->>OnboardProxy: checkOllamaModelToolSupport(model)
    OnboardProxy->>LocalInf: probeOllamaModelCapabilities(model)
    LocalInf->>OllamaAPI: POST /api/show { name: model }
    OllamaAPI-->>LocalInf: { capabilities: [...], ... } or error
    LocalInf-->>OnboardProxy: OllamaCapabilities (supportsTools: true/false/null)

    alt supportsTools is false & interactive
        OnboardProxy->>User: print warning + prompt
        User-->>OnboardProxy: response
        alt User declines
            OnboardProxy-->>TUI: { ok: false, message }
        else User accepts or auto-yes
            OnboardProxy-->>TUI: { ok: true }
        end
    else supportsTools is true OR null
        OnboardProxy-->>TUI: { ok: true }
    end

    OnboardProxy->>OnboardProxy: proceed with pull/load sequence
    OnboardProxy-->>TUI: model ready / streaming responses

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Poem

🐰 I sniffed the Ollama paths beneath the dew,
I ask which models know the tools they use,
If one says "no", I tap a gentle clue,
Suggest starter names so streams stay true,
Hooray — the TUI hops along anew!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 35.71% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: adding a warning for Ollama models without tools capability during onboarding.
Linked Issues check	✅ Passed	The PR addresses the linked issue #2667 by detecting models without tool support and warning users with actionable guidance, including capability checking and user-friendly error messages.
Out of Scope Changes check	✅ Passed	All changes are directly aligned with the stated objectives: capability probing, warning mechanisms, tool-support gating, and comprehensive tests for Ollama tools capability.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

• Generate code and open pull requests
• Plan features and break down work
• Investigate incidents and troubleshoot customer tickets together
• Automate recurring tasks and respond to alerts with triggers
• Summarize progress and report instantly

Built for teams:

• Shared memory across your entire org—no repeating context
• Per-thread sandboxes to safely plan and execute work
• Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/ollama-tools-capability.test.ts`:
- Around line 236-238: The cast on localInference is too direct; update it to go
through unknown first to satisfy TypeScript (i.e., cast localInference to
unknown then to Record<string, unknown>) before assigning
probeOllamaModelCapabilities so the assignment to SHARED.scriptedCaps is
type-safe; locate the expression using localInference and
probeOllamaModelCapabilities and replace the direct cast with the intermediate
unknown cast.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a88c8b90-13dd-4243-9178-3f75cf64b1d3

📥 Commits

Reviewing files that changed from the base of the PR and between 3800964 and ad2615a.

📒 Files selected for processing (3)

• src/lib/local-inference.ts
• src/lib/onboard-ollama-proxy.ts
• test/ollama-tools-capability.test.ts

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/ollama-tools-capability.test.ts`:
- Around line 230-266: The proxy module is cached and can be loaded before our
stubs are installed, causing order-dependent tests; in loadProxyWithStubs() call
the installSharedStubs() first (already done) and then clear Node's require
cache for the onboard proxy module (resolve ONBOARD_OLLAMA_PROXY_PATH and delete
its entry from require.cache) before requiring it so the module re-evaluates and
picks up the patched probeOllamaModelCapabilities and credential.prompt stubs;
keep SHARED resets and then require the proxy as before.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f2d578d6-1436-46e4-8381-31d0630fe91a

📥 Commits

Reviewing files that changed from the base of the PR and between ad2615a and 5b4e9b7.

📒 Files selected for processing (1)

• test/ollama-tools-capability.test.ts