Skip to content

test: add unit tests for Bedrock provider - #1204

Closed
mvincentbb wants to merge 5 commits into
NVIDIA:mainfrom
mvincentbb:test/bedrock-provider
Closed

test: add unit tests for Bedrock provider#1204
mvincentbb wants to merge 5 commits into
NVIDIA:mainfrom
mvincentbb:test/bedrock-provider

Conversation

@mvincentbb

@mvincentbb mvincentbb commented Mar 31, 2026

Copy link
Copy Markdown
Contributor
  • inference-config: verify bedrock returns correct config and default model
  • onboard: verify getSandboxInferenceConfig maps bedrock with supportsStore:false
  • onboard-selection: verify curated model list, region check, and endpoint probe

Summary

Add unit tests for the Amazon Bedrock inference provider introduced in #963. Covers provider config, sandbox inference mapping,
and interactive selection UX.

Related Issue

Relates to #963

Changes

  • test/inference-config.test.js: verify getProviderSelectionConfig("bedrock") returns correct OpenAI-compatible config,
    BEDROCK_API_KEY credential env, and default model (nvidia.nemotron-super-3-120b)
  • test/onboard.test.js: verify getSandboxInferenceConfig maps bedrock to routed inference provider with supportsStore: false
  • test/onboard-selection.test.js: verify interactive setup shows curated Bedrock models, enforces BEDROCK_REGION check, and
    probes endpoint via Chat Completions API

Type of Change

  • Code change for a new feature, bug fix, or refactor.
  • Code change with doc updates.
  • Doc only. Prose changes without code sample modifications.
  • Doc only. Includes code sample changes.

Testing

  • npx prek run --all-files passes (or equivalently make check).
  • npm test passes.
  • make docs builds without warnings. (for doc-only changes)

Checklist

General

Code Changes

  • Formatters applied — npx prek run --all-files auto-fixes formatting (or make format for targeted runs).
  • Tests added or updated for new or changed behavior.
  • No secrets, API keys, or credentials committed.
  • Doc pages updated for any user-facing behavior changes (new commands, changed defaults, new features, bug fixes that
    contradict existing docs).

Doc Changes

N/A — no doc changes in this PR.

Signed-off-by: BAKPATINA Vincent de Paul 47268786+mvincentbb@users.noreply.github.com

Summary by CodeRabbit

  • Tests
    • Updated Bedrock model selection test to reflect a revised default model choice.
    • Added an onboarding test that simulates selecting Bedrock and verifies user-facing confirmation messages and availability of the Chat Completions API.
    • Added a sandbox inference test to confirm Bedrock is routed to the expected inference path and that API/compatibility flags are set.

- inference-config: verify bedrock returns correct config and default model
- onboard: verify getSandboxInferenceConfig maps bedrock with supportsStore:false
- onboard-selection: verify curated model list, region check, and endpoint probe
@coderabbitai

coderabbitai Bot commented Mar 31, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 0dbcdc3c-6f29-4451-95db-21d0ad3d79b0

📥 Commits

Reviewing files that changed from the base of the PR and between 0fe0d30 and 9f2ac6d.

📒 Files selected for processing (1)
  • test/onboard.test.js
✅ Files skipped from review due to trivial changes (1)
  • test/onboard.test.js

📝 Walkthrough

Walkthrough

Updated and added tests to validate Amazon Bedrock model selection (model id changed to nvidia.nemotron-super-3-120b), simulate interactive onboarding selecting Bedrock and Chat Completions, and verify sandbox inference mapping routes Bedrock models through the inference provider.

Changes

Cohort / File(s) Summary
Bedrock provider config tests
test/inference-config.test.js
Updated assertions to expect custom.model = nvidia.nemotron-super-3-120b for bedrock and adjusted the fallback getProviderSelectionConfig("bedrock").model.
Onboard selection flow test
test/onboard-selection.test.js
Added a subprocess test that sets BEDROCK_API_KEY/BEDROCK_REGION, stubs interactive prompts to pick Bedrock and nvidia.nemotron-super-3-120b, injects a fake curl, captures logs, and asserts provider/model selection plus Chat Completions API availability.
Sandbox inference mapping test
test/onboard.test.js
Added a unit test asserting getSandboxInferenceConfig("nvidia.nemotron-super-3-120b", "bedrock") maps to routed inference: providerKey: "inference", primaryModelRef prefixed with inference/, inferenceBaseUrl = https://inference.local/v1, inferenceApi = openai-completions, and inferenceCompat.supportsStore = false.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~15 minutes

Poem

🐰 I hopped through tests with nimble feet,
Bedrock picked, a model tidy and neat,
A fake curl chirps, prompts answered with glee,
Sandbox routes whisper "inference" to me,
Hooray — the burrow's checks all pass! 🥕

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately and specifically describes the main change: adding unit tests for the Bedrock provider, which is reflected across all three modified test files.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
test/onboard-selection.test.js (1)

2375-2465: Test title overstates BEDROCK_REGION validation coverage.

Line 2375 says this validates the region check, but the case only tests the success path with BEDROCK_REGION already set (Line 2430). Add a companion negative-path test (missing BEDROCK_REGION) to ensure the guard is actually enforced.

✅ Suggested follow-up test
+  it("requires BEDROCK_REGION before accepting Bedrock selection", () => {
+    // same setup as the Bedrock success test, but omit BEDROCK_REGION
+    // assert that setup does not complete successfully and prompts/logs region guidance
+  });
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/onboard-selection.test.js` around lines 2375 - 2465, The test title is
misleading because it only covers the success path with BEDROCK_REGION set; add
a negative-path test that mirrors the existing "shows curated Bedrock models..."
test but does not set process.env.BEDROCK_REGION (remove or unset BEDROCK_REGION
in the inline script before calling setupNim) and assert that setupNim fails
(non-zero process status) and emits an error or message indicating a
missing/invalid BEDROCK_REGION; reference the same symbols used in the existing
test (setupNim, BEDROCK_API_KEY, BEDROCK_REGION, and the spawnSync-based test
harness) so the new test reuses the fake curl and prompt runner scaffolding but
expects failure when BEDROCK_REGION is absent.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@test/onboard-selection.test.js`:
- Around line 2375-2465: The test title is misleading because it only covers the
success path with BEDROCK_REGION set; add a negative-path test that mirrors the
existing "shows curated Bedrock models..." test but does not set
process.env.BEDROCK_REGION (remove or unset BEDROCK_REGION in the inline script
before calling setupNim) and assert that setupNim fails (non-zero process
status) and emits an error or message indicating a missing/invalid
BEDROCK_REGION; reference the same symbols used in the existing test (setupNim,
BEDROCK_API_KEY, BEDROCK_REGION, and the spawnSync-based test harness) so the
new test reuses the fake curl and prompt runner scaffolding but expects failure
when BEDROCK_REGION is absent.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 78a413ad-4ebe-41aa-9a1d-6dbe6f59f834

📥 Commits

Reviewing files that changed from the base of the PR and between 82789a6 and 05aec40.

📒 Files selected for processing (3)
  • test/inference-config.test.js
  • test/onboard-selection.test.js
  • test/onboard.test.js

@cv

cv commented Mar 31, 2026

Copy link
Copy Markdown
Collaborator

@mvincentbb can you make sure your commits are signed and DCO tag has been added to the PR description, please?

Signed-off-by: BAKPATINA Vincent de Paul <47268786+mvincentbb@users.noreply.github.com>
@mvincentbb
mvincentbb force-pushed the test/bedrock-provider branch from 0fe0d30 to 9f2ac6d Compare March 31, 2026 21:19
cv pushed a commit that referenced this pull request Mar 31, 2026
Bedrock's OpenAI-compatible endpoint works through the existing
"compatible-endpoint" option — the same path every other CSP wrapper
(Vertex, Azure OpenAI, etc.) should use. Adding a dedicated provider
key for each CSP does not scale and creates maintenance overhead for
config, tests, UX, and docs that is better avoided.

Also adds a guard test that enumerates the approved provider set and
fails if a new provider key is introduced, steering contributors toward
the compatible-endpoint / compatible-anthropic-endpoint options instead.

Reverts: #963
Closes: #1204
@cv

cv commented Mar 31, 2026

Copy link
Copy Markdown
Collaborator

@mvincentbb Thanks for the work on the Bedrock tests and the original provider PR (#963) — the testing was thorough and the implementation was clean.

After reviewing the direction this takes us, we've decided to revert Bedrock as a first-class provider and close this PR. The core issue: adding a dedicated provider key for each CSP (Bedrock, Vertex, Azure OpenAI, etc.) doesn't scale. Each one requires config in onboard.js, a case in inference-config.js, UX menu entries, curated model lists, validation logic, tests, and docs. Bedrock's OpenAI-compatible endpoint already works through the existing compatible-endpoint option, which is the path all CSP wrappers should use.

We've added a guard test in #1212 that enforces this — it'll fail if anyone tries to add a new provider key, pointing them to the compatible-endpoint options instead.

No issue with the quality of the contribution; this is a design direction decision. If you run into friction using compatible-endpoint for Bedrock, let us know and we can improve the generic path.

@cv cv closed this Mar 31, 2026
cv added a commit that referenced this pull request Mar 31, 2026
## Summary

Reverts #963 and adds a guard test that prevents new CSP-specific
provider keys from being added to the installer.

## Rationale

Bedrock's OpenAI-compatible endpoint already works through the existing
`compatible-endpoint` option — the same path every other CSP wrapper
(Vertex, Azure OpenAI, etc.) should use. Adding a dedicated provider key
for each CSP does not scale: each one requires config in `onboard.js`, a
case in `inference-config.js`, UX menu entries, model curation,
validation logic, tests, and docs. The `compatible-endpoint` and
`compatible-anthropic-endpoint` options already cover these use cases
generically.

## Changes

- **`bin/lib/onboard.js`**: Remove `BEDROCK_ENDPOINT_URL`, `bedrock`
from `REMOTE_PROVIDER_CONFIG`, `REMOTE_MODEL_OPTIONS`, `validProviders`,
interactive menu, region check, validation, `setupInference`, and
dashboard label.
- **`bin/lib/inference-config.js`**: Remove `bedrock` case from
`getProviderSelectionConfig()`.
- **`test/inference-config.test.js`**: Remove Bedrock-specific
assertions; add `does not grow beyond the approved provider set` guard
test that fails if any new provider key (bedrock, vertex, azure, etc.)
is introduced.

## Related

- Reverts #963
- Closes #1204

## Type of Change

- [x] Code change for a new feature, bug fix, or refactor.

## Checklist

- [x] Formatters applied
- [x] Tests added (guard test)
- [x] No secrets committed
- [x] `npm test` passes (725 passed)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Breaking Changes**
  * Removed support for Amazon Bedrock as an inference provider.

* **Documentation**
* Updated command documentation to recommend `nemoclaw onboard` for new
installations.
  * Marked legacy `nemoclaw setup` command as deprecated.
* Updated deployment guides to reference the `nemoclaw onboard` setup
wizard.

* **Tests**
  * Updated test suite to validate approved inference providers.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Deepak Jain <deepujain@users.noreply.github.com>
laitingsheng pushed a commit that referenced this pull request Apr 2, 2026
## Summary

Reverts #963 and adds a guard test that prevents new CSP-specific
provider keys from being added to the installer.

## Rationale

Bedrock's OpenAI-compatible endpoint already works through the existing
`compatible-endpoint` option — the same path every other CSP wrapper
(Vertex, Azure OpenAI, etc.) should use. Adding a dedicated provider key
for each CSP does not scale: each one requires config in `onboard.js`, a
case in `inference-config.js`, UX menu entries, model curation,
validation logic, tests, and docs. The `compatible-endpoint` and
`compatible-anthropic-endpoint` options already cover these use cases
generically.

## Changes

- **`bin/lib/onboard.js`**: Remove `BEDROCK_ENDPOINT_URL`, `bedrock`
from `REMOTE_PROVIDER_CONFIG`, `REMOTE_MODEL_OPTIONS`, `validProviders`,
interactive menu, region check, validation, `setupInference`, and
dashboard label.
- **`bin/lib/inference-config.js`**: Remove `bedrock` case from
`getProviderSelectionConfig()`.
- **`test/inference-config.test.js`**: Remove Bedrock-specific
assertions; add `does not grow beyond the approved provider set` guard
test that fails if any new provider key (bedrock, vertex, azure, etc.)
is introduced.

## Related

- Reverts #963
- Closes #1204

## Type of Change

- [x] Code change for a new feature, bug fix, or refactor.

## Checklist

- [x] Formatters applied
- [x] Tests added (guard test)
- [x] No secrets committed
- [x] `npm test` passes (725 passed)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Breaking Changes**
  * Removed support for Amazon Bedrock as an inference provider.

* **Documentation**
* Updated command documentation to recommend `nemoclaw onboard` for new
installations.
  * Marked legacy `nemoclaw setup` command as deprecated.
* Updated deployment guides to reference the `nemoclaw onboard` setup
wizard.

* **Tests**
  * Updated test suite to validate approved inference providers.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Deepak Jain <deepujain@users.noreply.github.com>
lakamsani pushed a commit to lakamsani/NemoClaw that referenced this pull request Apr 4, 2026
## Summary

Reverts NVIDIA#963 and adds a guard test that prevents new CSP-specific
provider keys from being added to the installer.

## Rationale

Bedrock's OpenAI-compatible endpoint already works through the existing
`compatible-endpoint` option — the same path every other CSP wrapper
(Vertex, Azure OpenAI, etc.) should use. Adding a dedicated provider key
for each CSP does not scale: each one requires config in `onboard.js`, a
case in `inference-config.js`, UX menu entries, model curation,
validation logic, tests, and docs. The `compatible-endpoint` and
`compatible-anthropic-endpoint` options already cover these use cases
generically.

## Changes

- **`bin/lib/onboard.js`**: Remove `BEDROCK_ENDPOINT_URL`, `bedrock`
from `REMOTE_PROVIDER_CONFIG`, `REMOTE_MODEL_OPTIONS`, `validProviders`,
interactive menu, region check, validation, `setupInference`, and
dashboard label.
- **`bin/lib/inference-config.js`**: Remove `bedrock` case from
`getProviderSelectionConfig()`.
- **`test/inference-config.test.js`**: Remove Bedrock-specific
assertions; add `does not grow beyond the approved provider set` guard
test that fails if any new provider key (bedrock, vertex, azure, etc.)
is introduced.

## Related

- Reverts NVIDIA#963
- Closes NVIDIA#1204

## Type of Change

- [x] Code change for a new feature, bug fix, or refactor.

## Checklist

- [x] Formatters applied
- [x] Tests added (guard test)
- [x] No secrets committed
- [x] `npm test` passes (725 passed)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Breaking Changes**
  * Removed support for Amazon Bedrock as an inference provider.

* **Documentation**
* Updated command documentation to recommend `nemoclaw onboard` for new
installations.
  * Marked legacy `nemoclaw setup` command as deprecated.
* Updated deployment guides to reference the `nemoclaw onboard` setup
wizard.

* **Tests**
  * Updated test suite to validate approved inference providers.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Deepak Jain <deepujain@users.noreply.github.com>
gemini2026 pushed a commit to gemini2026/NemoClaw that referenced this pull request Apr 14, 2026
## Summary

Reverts NVIDIA#963 and adds a guard test that prevents new CSP-specific
provider keys from being added to the installer.

## Rationale

Bedrock's OpenAI-compatible endpoint already works through the existing
`compatible-endpoint` option — the same path every other CSP wrapper
(Vertex, Azure OpenAI, etc.) should use. Adding a dedicated provider key
for each CSP does not scale: each one requires config in `onboard.js`, a
case in `inference-config.js`, UX menu entries, model curation,
validation logic, tests, and docs. The `compatible-endpoint` and
`compatible-anthropic-endpoint` options already cover these use cases
generically.

## Changes

- **`bin/lib/onboard.js`**: Remove `BEDROCK_ENDPOINT_URL`, `bedrock`
from `REMOTE_PROVIDER_CONFIG`, `REMOTE_MODEL_OPTIONS`, `validProviders`,
interactive menu, region check, validation, `setupInference`, and
dashboard label.
- **`bin/lib/inference-config.js`**: Remove `bedrock` case from
`getProviderSelectionConfig()`.
- **`test/inference-config.test.js`**: Remove Bedrock-specific
assertions; add `does not grow beyond the approved provider set` guard
test that fails if any new provider key (bedrock, vertex, azure, etc.)
is introduced.

## Related

- Reverts NVIDIA#963
- Closes NVIDIA#1204

## Type of Change

- [x] Code change for a new feature, bug fix, or refactor.

## Checklist

- [x] Formatters applied
- [x] Tests added (guard test)
- [x] No secrets committed
- [x] `npm test` passes (725 passed)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Breaking Changes**
  * Removed support for Amazon Bedrock as an inference provider.

* **Documentation**
* Updated command documentation to recommend `nemoclaw onboard` for new
installations.
  * Marked legacy `nemoclaw setup` command as deprecated.
* Updated deployment guides to reference the `nemoclaw onboard` setup
wizard.

* **Tests**
  * Updated test suite to validate approved inference providers.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Deepak Jain <deepujain@users.noreply.github.com>
@wscurran wscurran added area: providers Inference provider integrations and provider behavior feature PR adds or expands user-visible functionality and removed enhancement: provider labels Jun 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: providers Inference provider integrations and provider behavior feature PR adds or expands user-visible functionality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants