perf(inference): reuse selected chat capability

[HOYALIM]

Summary

Avoid repeating the same successful Chat Completions capability request during one onboarding run. Provider selection records a narrowly scoped, one-shot capability receipt that the immediately following host smoke consumes only when endpoint, model, authentication mode, and semantic requirements still match.

Changes

• Add an in-memory, one-shot onboarding capability cache that never stores credentials.
• Reuse only validated Chat Completions results for non-pinned, non-custom-header, credential-free URL shapes.
• Invalidate the receipt on provider or route setup failures and fall back to the existing smoke path on any mismatch.
• Keep SSRF-pinned, specialized streaming, custom-header, and accepted-but-unvalidated paths unchanged.

Verification

• npx vitest run --project cli src/lib/onboard/inference-capability-cache.test.ts src/lib/onboard/inference-selection-validation.test.ts
• npx vitest run --project integration test/onboard-smoke-verifier.test.ts
• npm run build:cli
• npm run typecheck:cli
• npm run check:diff

---

Signed-off-by: Ho Lim subhoya@gmail.com

Summary by CodeRabbit

• New Features
  • Reduced duplicate onboarding validation by reusing successful OpenAI-like Chat Completions checks across provider/model setup and smoke verification via an optional one-shot, in-memory capability cache.
• Bug Fixes
  • Automatically invalidates cached entries when provider setup or validation fails to prevent stale reuse.
• Tests
  • Added coverage for cache reuse (including benign endpoint normalization), one-time consumption, model/auth mismatches, security-sensitive rejection, invalidation, and skipping duplicate smoke probes; also added scenarios for safely/non-safely cacheable capability priming.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The PR adds a per-run, one-shot cache for successful OpenAI Chat Completions validation, propagates it through NIM onboarding, reuses matching results during smoke verification, and invalidates cached state on validation or setup failures.

Changes

Onboarding capability cache

Layer / File(s)	Summary
Cache key and one-shot storage src/lib/onboard/inference-capability-cache.ts, src/lib/onboard/inference-capability-cache.test.ts	Adds credential-safe cache-key generation, one-shot retrieval, invalidation, and tests for matching, mismatched, and sensitive inputs.
Selection validation recording src/lib/onboard/inference-selection-validation.ts, src/lib/onboard/inference-selection-validation.test.ts	Records eligible successful OpenAI-like selections and invalidates cached state when probing fails.
Onboarding cache propagation src/lib/onboard/setup-nim-flow.ts, src/lib/onboard/setup-nim-selection.ts, src/lib/onboard/machine/handlers/provider-inference.ts, src/lib/onboard/setup-inference.ts, src/lib/onboard/inference-providers/*	Creates a per-run cache and passes it through selection, provider setup, and smoke-verification paths; setup failures invalidate it.
Smoke verification reuse and coverage src/lib/inference/onboard-probes.ts, test/helpers/onboard-smoke-verifier-harness.ts, test/onboard-smoke-verifier.test.ts	Skips matching duplicate smoke probes, invalidates the cache after probe failure, and tests reuse, zero-probe behavior, and unsafe cache priming.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SelectionValidation
  participant OnboardInferenceCapabilityCache
  participant VerifyOnboardInferenceSmoke
  participant SmokeProbe
  SelectionValidation->>OnboardInferenceCapabilityCache: Record successful Chat Completions validation
  VerifyOnboardInferenceSmoke->>OnboardInferenceCapabilityCache: Consume matching validation
  OnboardInferenceCapabilityCache-->>VerifyOnboardInferenceSmoke: Return reusable result
  VerifyOnboardInferenceSmoke->>VerifyOnboardInferenceSmoke: Skip duplicate smoke probe
  VerifyOnboardInferenceSmoke->>SmokeProbe: Run probe when no reusable result exists
  SmokeProbe-->>OnboardInferenceCapabilityCache: Invalidate after probe failure

Loading

Possibly related PRs

• NVIDIA/NemoClaw#3603: Adds the smoke-probe flow that this PR extends with capability-cache reuse and invalidation.
• NVIDIA/NemoClaw#4289: Updates the overlapping provider-inference setup options path with a separate inference option.

Suggested labels: feature, integration: hermes, NV QA

Suggested reviewers: ericksoa, cv, jyaunches

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and matches the main change: reusing selected chat capability via an inference cache.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

E2E Advisor Recommendation

Required E2E: cloud-onboard, inference-routing, network-policy, onboard-repair, onboard-resume
Optional E2E: None

Dispatch hint: cloud-onboard,inference-routing,network-policy,onboard-repair,onboard-resume

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: target/main
Head: HEAD
Confidence: high

Required E2E

• cloud-onboard: Validate clean hosted onboarding with the changed provider-selection, inference setup, route verification, and smoke-check flow.
• inference-routing: Validate that inference routes remain usable when selection validation and onboarding smoke verification share capability state.
• network-policy: Validate intended inference connectivity and policy isolation, including the changed handling of pinned and security-sensitive probe paths.
• onboard-repair: Validate repair convergence after provider/inference setup failures invalidate transient validation state.
• onboard-resume: Validate resumed onboarding preserves correct provider route behavior without persisting or incorrectly reusing one-run capability state.

Optional E2E

• None.

New E2E recommendations

• inference-routing (medium): Existing unit and harness coverage verifies one-shot cache matching and invalidation, but the selected-provider live flow should explicitly prove that an eligible validation is reused exactly once and that a retry/failure forces a fresh real probe.
  • Suggested test: Extend an existing live onboarding/inference-routing scenario with an instrumented OpenAI-compatible endpoint that records selection and smoke probes, asserting one reuse only for an identical non-sensitive route and a fresh probe after invalidation.

Dispatch hint

• Workflow: .github/workflows/e2e.yaml
• jobs input: cloud-onboard,inference-routing,network-policy,onboard-repair,onboard-resume

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/onboard/inference-capability-cache.test.ts`:
- Around line 21-27: Remove both conditional request increments in the cache
test and assert the boolean results of takeCompletedOpenAiChat directly,
preserving the expected first and second one-shot outcomes and request counts.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f33d1909-75e5-4467-aeba-955caf4b8cfa

📥 Commits

Reviewing files that changed from the base of the PR and between 13bece6 and ef6a203.

📒 Files selected for processing (13)

• src/lib/inference/onboard-probes.ts
• src/lib/onboard/inference-capability-cache.test.ts
• src/lib/onboard/inference-capability-cache.ts
• src/lib/onboard/inference-providers/remote.ts
• src/lib/onboard/inference-providers/types.ts
• src/lib/onboard/inference-selection-validation.test.ts
• src/lib/onboard/inference-selection-validation.ts
• src/lib/onboard/machine/handlers/provider-inference.ts
• src/lib/onboard/setup-inference.ts
• src/lib/onboard/setup-nim-flow.ts
• src/lib/onboard/setup-nim-selection.ts
• test/helpers/onboard-smoke-verifier-harness.ts
• test/onboard-smoke-verifier.test.ts

[wscurran]

✨ Thanks for the performance work, @HOYALIM. Adding a one-shot capability cache to avoid repeated Chat Completions requests during onboarding could reduce latency. Ready for maintainer review.

[coderabbitai]

🧹 Nitpick comments (1)

src/lib/onboard/setup-nim-flow.test.ts (1)

284-286: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Prefer a behavioral cache assertion over toBeInstanceOf.

This couples the setup-flow test to the concrete cache class without proving that the returned cache is usable or correctly propagated. Exercise its public behavior, or cover propagation at the public onboarding boundary, while keeping this test focused on observable setup results.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard/setup-nim-flow.test.ts` around lines 284 - 286, Update the
assertion in the setup-flow test around resultWithoutCache to verify the
returned inferenceCapabilityCache through its public behavior rather than
checking its concrete OnboardInferenceCapabilityCache type. Exercise a
meaningful cache operation or assert propagation through the public onboarding
result, while keeping the existing observable setup-result assertions unchanged.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lib/onboard/setup-nim-flow.test.ts`:
- Around line 284-286: Update the assertion in the setup-flow test around
resultWithoutCache to verify the returned inferenceCapabilityCache through its
public behavior rather than checking its concrete
OnboardInferenceCapabilityCache type. Exercise a meaningful cache operation or
assert propagation through the public onboarding result, while keeping the
existing observable setup-result assertions unchanged.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5793c17f-7c95-4fed-8fc6-0130d1475eab

📥 Commits

Reviewing files that changed from the base of the PR and between 93e94c9 and 231eca3.

📒 Files selected for processing (14)

• src/lib/inference/onboard-probes.ts
• src/lib/onboard/inference-capability-cache.test.ts
• src/lib/onboard/inference-capability-cache.ts
• src/lib/onboard/inference-providers/remote.ts
• src/lib/onboard/inference-providers/types.ts
• src/lib/onboard/inference-selection-validation.test.ts
• src/lib/onboard/inference-selection-validation.ts
• src/lib/onboard/machine/handlers/provider-inference.ts
• src/lib/onboard/setup-inference.ts
• src/lib/onboard/setup-nim-flow.test.ts
• src/lib/onboard/setup-nim-flow.ts
• src/lib/onboard/setup-nim-selection.ts
• test/helpers/onboard-smoke-verifier-harness.ts
• test/onboard-smoke-verifier.test.ts

🚧 Files skipped from review as they are similar to previous changes (13)

• src/lib/onboard/inference-providers/types.ts
• src/lib/inference/onboard-probes.ts
• src/lib/onboard/inference-providers/remote.ts
• src/lib/onboard/setup-nim-selection.ts
• src/lib/onboard/inference-capability-cache.test.ts
• test/onboard-smoke-verifier.test.ts
• src/lib/onboard/setup-inference.ts
• src/lib/onboard/machine/handlers/provider-inference.ts
• src/lib/onboard/setup-nim-flow.ts
• src/lib/onboard/inference-capability-cache.ts
• src/lib/onboard/inference-selection-validation.test.ts
• test/helpers/onboard-smoke-verifier-harness.ts
• src/lib/onboard/inference-selection-validation.ts

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/helpers/onboard-smoke-verifier-harness.ts`:
- Around line 94-108: Update the capability-cache priming in the invocation flow
around selectedChatCapability to use the effective endpoint, model, provider,
and authMode values from input rather than hard-coded values. Ensure
rememberCompletedOpenAiChat receives the same capability key that
verifyOnboardInferenceSmoke will use after applying ...input, so the selected
capability path exercises cache reuse.
- Around line 97-101: Update the cache priming call in the harness around
capabilityCache.rememberCompletedOpenAiChat to capture and assert its boolean
result. Fail the test harness immediately when the call returns false, ensuring
the probe mock’s successful response actually primes the cache before reuse is
tested.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c406b73a-5242-478e-9ec6-b0119c4d4564

📥 Commits

Reviewing files that changed from the base of the PR and between 231eca3 and 6143acf.

📒 Files selected for processing (14)

• src/lib/inference/onboard-probes.ts
• src/lib/onboard/inference-capability-cache.test.ts
• src/lib/onboard/inference-capability-cache.ts
• src/lib/onboard/inference-providers/remote.ts
• src/lib/onboard/inference-providers/types.ts
• src/lib/onboard/inference-selection-validation.test.ts
• src/lib/onboard/inference-selection-validation.ts
• src/lib/onboard/machine/handlers/provider-inference.ts
• src/lib/onboard/setup-inference.ts
• src/lib/onboard/setup-nim-flow.test.ts
• src/lib/onboard/setup-nim-flow.ts
• src/lib/onboard/setup-nim-selection.ts
• test/helpers/onboard-smoke-verifier-harness.ts
• test/onboard-smoke-verifier.test.ts

🚧 Files skipped from review as they are similar to previous changes (12)

• src/lib/onboard/inference-capability-cache.test.ts
• src/lib/onboard/inference-selection-validation.test.ts
• test/onboard-smoke-verifier.test.ts
• src/lib/onboard/setup-nim-flow.test.ts
• src/lib/onboard/inference-selection-validation.ts
• src/lib/onboard/inference-providers/types.ts
• src/lib/onboard/setup-nim-selection.ts
• src/lib/onboard/setup-inference.ts
• src/lib/inference/onboard-probes.ts
• src/lib/onboard/inference-providers/remote.ts
• src/lib/onboard/setup-nim-flow.ts
• src/lib/onboard/machine/handlers/provider-inference.ts