Skip to content

feat: add Amazon Bedrock as inference provider #963

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cv merged 8 commits into from
Mar 31, 2026
Merged

feat: add Amazon Bedrock as inference provider #963

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
6a47995
feat: add Amazon Bedrock as inference provider
mvincentbb Mar 26, 2026
a7845c9
fix: address review — require explicit region, validate model and end…
mvincentbb Mar 26, 2026
d5826d6
fix: address review — use distinct credential key, require region, va…
mvincentbb Mar 26, 2026
0efd1ef
fix: align sandbox config, credential key, and dashboard label for Be…
mvincentbb Mar 27, 2026
5b47b98
fix: move Bedrock region check before model validation
mvincentbb Mar 27, 2026
8d50da1
Merge branch 'main' into feat/bedrock-provider
cv Mar 30, 2026
e58e0bc
fix: move bedrock menu entry after vLLM/NIM to preserve existing opti…
mvincentbb Mar 30, 2026
4ba56ff
Merge branch 'main' into feat/bedrock-provider
cv Mar 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions bin/lib/inference-config.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,17 @@ function getProviderSelectionConfig(provider, model) {
provider,
providerLabel: "Local Ollama",
};
case "bedrock":
return {
endpointType: "custom",
endpointUrl: INFERENCE_ROUTE_URL,
ncpPartner: null,
model: model || "nvidia.nemotron-super-3-120b",
profile: DEFAULT_ROUTE_PROFILE,
credentialEnv: "BEDROCK_API_KEY",
provider,
providerLabel: "Amazon Bedrock (OpenAI-compatible)",
};
Comment thread
coderabbitai[bot] marked this conversation as resolved.
default:
return null;
}
Expand Down
59 changes: 54 additions & 5 deletions bin/lib/onboard.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ const BUILD_ENDPOINT_URL = "https://integrate.api.nvidia.com/v1";
const OPENAI_ENDPOINT_URL = "https://api.openai.com/v1";
const ANTHROPIC_ENDPOINT_URL = "https://api.anthropic.com";
const GEMINI_ENDPOINT_URL = "https://generativelanguage.googleapis.com/v1beta/openai/";
const BEDROCK_ENDPOINT_URL = `https://bedrock-mantle.${process.env.BEDROCK_REGION || process.env.AWS_REGION || ""}.api.aws/v1`;
Comment thread
coderabbitai[bot] marked this conversation as resolved.

const REMOTE_PROVIDER_CONFIG = {
build: {
Expand Down Expand Up @@ -116,6 +117,17 @@ const REMOTE_PROVIDER_CONFIG = {
defaultModel: "",
skipVerify: true,
},
bedrock: {
label: "Amazon Bedrock (OpenAI-compatible endpoint)",
providerName: "bedrock",
providerType: "openai",
credentialEnv: "BEDROCK_API_KEY",
endpointUrl: BEDROCK_ENDPOINT_URL,
helpUrl: "https://docs.aws.amazon.com/bedrock/latest/userguide/api-keys-generate.html",
modelMode: "curated",
defaultModel: "nvidia.nemotron-super-3-120b",
skipVerify: true,
},
Comment thread
coderabbitai[bot] marked this conversation as resolved.
};

const REMOTE_MODEL_OPTIONS = {
Expand All @@ -138,6 +150,15 @@ const REMOTE_MODEL_OPTIONS = {
"gemini-2.5-flash",
"gemini-2.5-flash-lite",
],
bedrock: [
"nvidia.nemotron-nano-3-30b",
"nvidia.nemotron-super-3-120b",
"deepseek.v3.2",
"openai.gpt-oss-120b",
"mistral.devstral-2-123b",
"moonshotai.kimi-k2.5",
"minimax.minimax-m2.5",
],
};

// Non-interactive mode: set by --non-interactive flag or env var.
Expand Down Expand Up @@ -562,6 +583,7 @@ function getSandboxInferenceConfig(model, provider = null, preferredInferenceApi
supportsStore: false,
};
break;
case "bedrock":
case "compatible-endpoint":
providerKey = "inference";
primaryModelRef = `inference/${model}`;
Expand Down Expand Up @@ -1472,10 +1494,10 @@ function getNonInteractiveProvider() {
anthropiccompatible: "anthropicCompatible",
};
const normalized = aliases[providerKey] || providerKey;
const validProviders = new Set(["build", "openai", "anthropic", "anthropicCompatible", "gemini", "ollama", "custom", "nim-local", "vllm"]);
const validProviders = new Set(["build", "openai", "anthropic", "anthropicCompatible", "gemini", "ollama", "custom", "nim-local", "vllm", "bedrock"]);
if (!validProviders.has(normalized)) {
console.error(` Unsupported NEMOCLAW_PROVIDER: ${providerKey}`);
console.error(" Valid values: build, openai, anthropic, anthropicCompatible, gemini, ollama, custom, nim-local, vllm");
console.error(" Valid values: build, openai, anthropic, anthropicCompatible, gemini, ollama, custom, nim-local, vllm, bedrock");
process.exit(1);
}

Expand Down Expand Up @@ -2028,7 +2050,7 @@ async function setupNim(gpu) {
label: "Local vLLM [experimental] — running",
});
}

options.push({ key: "bedrock", label: "Amazon Bedrock (OpenAI-compatible endpoint)" });
// On macOS without Ollama, offer to install it
if (!hasOllama && process.platform === "darwin") {
options.push({ key: "install-ollama", label: "Install Ollama (macOS)" });
Expand Down Expand Up @@ -2076,6 +2098,18 @@ async function setupNim(gpu) {
endpointUrl = remoteConfig.endpointUrl;
preferredInferenceApi = null;

if (selected.key === "bedrock") {
const bedrockRegion = process.env.BEDROCK_REGION || process.env.AWS_REGION;
if (!bedrockRegion) {
console.error(" BEDROCK_REGION or AWS_REGION must be set for Amazon Bedrock.");
if (isNonInteractive()) {
process.exit(1);
}
continue selectionLoop;
}
endpointUrl = `https://bedrock-mantle.${bedrockRegion}.api.aws/v1`;
}

if (selected.key === "custom") {
endpointUrl = isNonInteractive()
? (process.env.NEMOCLAW_ENDPOINT_URL || "").trim()
Expand Down Expand Up @@ -2115,7 +2149,7 @@ async function setupNim(gpu) {
}
const defaultModel = requestedModel || remoteConfig.defaultModel;
let modelValidator = null;
if (selected.key === "openai" || selected.key === "gemini") {
if (selected.key === "openai" || selected.key === "gemini" || selected.key === "bedrock") {
modelValidator = (candidate) =>
validateOpenAiLikeModel(remoteConfig.label, endpointUrl, candidate, getCredential(credentialEnv));
} else if (selected.key === "anthropic") {
Expand Down Expand Up @@ -2145,6 +2179,20 @@ async function setupNim(gpu) {
if (validation.retry === "selection") {
continue selectionLoop;
}
} else if (selected.key === "bedrock") {
// Bedrock Mantle exposes an OpenAI-compatible API
const retryMessage = "Please choose a provider/model again.";
preferredInferenceApi = await validateOpenAiLikeSelection(
remoteConfig.label,
endpointUrl,
model,
credentialEnv,
retryMessage
);
if (preferredInferenceApi) {
break;
}
continue selectionLoop;
Comment on lines +2182 to +2195

@coderabbitai coderabbitai Bot Mar 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Fresh non-interactive Bedrock runs will probe without the env-supplied key.

Line 1740 accepts BEDROCK_API_KEY from process.env, but validateOpenAiLikeSelection() resolves the probe secret via getCredential(credentialEnv) only (Line 712). The rest of this file treats getCredential() as store-backed and falls back to process.env when env values are needed, e.g. Line 1522, so a CI run that only exports BEDROCK_API_KEY will hit Bedrock unauthenticated and exit before setupInference() persists the provider.

💡 One way to keep env-only CI runs working 
+function resolveCredentialValue(envName) {
+  return envName ? (process.env[envName] || getCredential(envName) || "") : "";
+}
+
 async function validateOpenAiLikeSelection(
   label,
   endpointUrl,
   model,
   credentialEnv = null,
   retryMessage = "Please choose a provider/model again."
 ) {
-  const apiKey = credentialEnv ? getCredential(credentialEnv) : "";
+  const apiKey = resolveCredentialValue(credentialEnv);
   const probe = probeOpenAiLikeEndpoint(endpointUrl, model, apiKey);
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@bin/lib/onboard.js` around lines 1778 - 1799, The Bedrock probe can run
unauthenticated because validateOpenAiLikeSelection() only uses
getCredential(credentialEnv) which may not read process.env; update the
credential resolution so the probe uses the env-supplied key when present.
Concretely, modify either getCredential(symbol credentialEnv) or
validateOpenAiLikeSelection(remoteConfig.label, endpointUrl, model,
credentialEnv, ...) to fall back to process.env[credentialEnv] (e.g.,
BEDROCK_API_KEY) when getCredential returns falsy before attempting the probe,
ensuring non-interactive/CI runs that only export BEDROCK_API_KEY succeed and
that setupInference() still persists the provider afterward.

@mvincentbb mvincentbb Mar 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getCredential() in credentials.js:28 already checks process.env[key] before the credential store — the probe runs authenticated. So validateOpenAiLikeSelection() at line 802 calling getCredential("BEDROCK_API_KEY") will pick up the env var. The probe runs authenticated.

@coderabbitai coderabbitai Bot Mar 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧩 Analysis chain

Let me verify by checking the getCredential() implementation directly.

🏁 Script executed:

#!/bin/bash
fd -e js credentials -g '*credentials*' | xargs grep -n "getCredential" | head -20
fd -e js -g '*credentials*' | head -5

Length of output: 148

🏁 Script executed:

#!/bin/bash
cat bin/lib/credentials.js

Length of output: 6749

@mvincentbb, you're absolutely right — I apologize for the noise. getCredential() in credentials.js (Line 30–33) is:

function getCredential(key) {
  if (process.env[key]) return process.env[key];   // ← env-first
  const creds = loadCredentials();
  return creds[key] || null;
}

It checks process.env[key] before the file-backed credential store, so getCredential("BEDROCK_API_KEY") will correctly pick up a CI-exported BEDROCK_API_KEY. The probe runs authenticated and my concern was a false positive. No change needed here.

@mvincentbb mvincentbb Mar 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you made another review then

@coderabbitai coderabbitai Bot Mar 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
echo "=== getSandboxInferenceConfig switch cases ==="
grep -n "case\|providerKey\|primaryModelRef\|inferenceCompat\|providerName" bin/lib/onboard.js | head -60

echo ""
echo "=== printDashboard providerLabel ==="
sed -n '/function printDashboard/,/^}/p' bin/lib/onboard.js | grep -n "providerLabel\|bedrock"

echo ""
echo "=== REMOTE_PROVIDER_CONFIG bedrock credentialEnv ==="
grep -n "credentialEnv\|bedrock" bin/lib/onboard.js | head -30

echo ""
echo "=== inference-config.js bedrock case ==="
grep -n "bedrock\|credentialEnv" bin/lib/inference-config.js | head -20

Length of output: 6160

@mvincentbb, here's a fresh look at the Bedrock additions. I found three issues beyond the already-resolved credential concern:

🔴 1. getSandboxInferenceConfig is missing a "bedrock" case — wrong Dockerfile ARGs baked into every sandbox

The switch statement in getSandboxInferenceConfig() (Lines 484–516) has explicit cases for openai-api, anthropic-prod, gemini-api, compatible-endpoint, etc., but no case for "bedrock". Since REMOTE_PROVIDER_CONFIG.bedrock.providerName is "bedrock", it silently falls into the default branch alongside nvidia-prod, producing:

providerKey = "inference";
primaryModelRef = `inference/${model}`;   // ← wrong; should be openai/${model} or similar
inferenceCompat = null;                   // ← supportsStore flag missing

This gets baked directly into the sandbox Dockerfile as ARG NEMOCLAW_PROVIDER_KEY / ARG NEMOCLAW_PRIMARY_MODEL_REF, so the agent inside the sandbox would be configured as if it were talking to an NVIDIA endpoint rather than an OpenAI-compatible one. The fix should mirror the compatible-endpoint or gemini-api case:

+    case "bedrock":
     case "compatible-endpoint":
       providerKey = "inference";
       primaryModelRef = `inference/${model}`;
       inferenceCompat = {
         supportsStore: false,
       };
       break;

🟠 2. credentialEnv is inconsistent across onboard, sandbox config, and docs

Location Value
REMOTE_PROVIDER_CONFIG.bedrock.credentialEnv (Line 120, onboard.js) "BEDROCK_API_KEY"
inference-config.js case "bedrock": (Line 117) "OPENAI_API_KEY"
PR description usage example export OPENAI_API_KEY=<bedrock-api-key>

During onboarding, the wizard checks/stores BEDROCK_API_KEY. Inside the sandbox, the config references OPENAI_API_KEY. A user following the PR docs (export OPENAI_API_KEY=...) will hit the onboarding credential prompt because BEDROCK_API_KEY is unset — and vice versa in CI. Pick one env var name and use it consistently in all three places.

🟡 3. printDashboard has no label for "bedrock" — shows raw provider name

Lines 6–13 of printDashboard map each known providerName to a human-readable string, but "bedrock" is absent, so the dashboard line reads Model nvidia.nemotron-super-3-120b (bedrock) instead of something like Amazon Bedrock. One-liner fix:

  else if (provider === "ollama-local") providerLabel = "Local Ollama";
+ else if (provider === "bedrock") providerLabel = "Amazon Bedrock";

} else if (selected.key === "anthropicCompatible") {
const validation = await validateCustomAnthropicSelection(
remoteConfig.label,
Expand Down Expand Up @@ -2405,7 +2453,7 @@ async function setupInference(sandboxName, model, provider, endpointUrl = null,
step(4, 7, "Setting up inference provider");
runOpenshell(["gateway", "select", GATEWAY_NAME], { ignoreError: true });

if (provider === "nvidia-prod" || provider === "nvidia-nim" || provider === "openai-api" || provider === "anthropic-prod" || provider === "compatible-anthropic-endpoint" || provider === "gemini-api" || provider === "compatible-endpoint") {
if (provider === "nvidia-prod" || provider === "nvidia-nim" || provider === "openai-api" || provider === "anthropic-prod" || provider === "compatible-anthropic-endpoint" || provider === "gemini-api" || provider === "compatible-endpoint" || provider === "bedrock") {
const config = provider === "nvidia-nim"
? REMOTE_PROVIDER_CONFIG.build
: Object.values(REMOTE_PROVIDER_CONFIG).find((entry) => entry.providerName === provider);
Expand Down Expand Up @@ -2835,6 +2883,7 @@ function printDashboard(sandboxName, model, provider, nimContainer = null) {
else if (provider === "gemini-api") providerLabel = "Google Gemini";
else if (provider === "compatible-endpoint") providerLabel = "Other OpenAI-compatible endpoint";
else if (provider === "vllm-local") providerLabel = "Local vLLM";
else if (provider === "bedrock") providerLabel = "Amazon Bedrock";
else if (provider === "ollama-local") providerLabel = "Local Ollama";

const token = fetchGatewayAuthTokenFromSandbox(sandboxName);
Expand Down