feat(inference): route DNS-backed HTTPS custom endpoints through a pinned adapter

ci/env-var-doc-allowlist.json

@@ -27,6 +27,18 @@
     "name": "NEMOCLAW_BEDROCK_RUNTIME_REGION",
     "reason": "Internal child-process setting used only to pass the resolved Bedrock Runtime region to the hidden local adapter. Users should rely on the endpoint URL or standard AWS region environment variables."
   },
+  {
+    "name": "NEMOCLAW_HTTPS_PIN_RUNTIME_ADAPTER_PORT",
+    "reason": "Internal child-process setting used only when launching the hidden HTTPS Pin Runtime adapter. The port is a fixed internal constant, not a public user-facing configuration knob."
+  },
+  {
+    "name": "NEMOCLAW_HTTPS_PIN_RUNTIME_ADAPTER_BOOTSTRAP_ROUTE",
+    "reason": "Internal child-process setting carrying a JSON-encoded route (including a credential value) used only to seed the hidden HTTPS Pin Runtime adapter at startup. Never user-set."
+  },
+  {
+    "name": "NEMOCLAW_HTTPS_PIN_RUNTIME_ADAPTER_ORPHANED_ROUTE_IDS",
+    "reason": "Internal child-process setting carrying a JSON-encoded array of route ids (no credentials) that a fresh HTTPS Pin Runtime adapter respawn could not recover, so it can respond to them distinctly from a route that never existed. Never user-set."
+  },
   {
     "name": "NEMOCLAW_RESTORE_LATEST_BACKUP_ON_RECREATE",
     "reason": "Internal installer sentinel exported only during OpenShell gateway replacement so onboard restores the pre-upgrade sandbox backup. Not user-facing."

docs/inference/custom-endpoint-security.mdx

@@ -9,6 +9,7 @@ keywords: ["nemoclaw endpoint security", "inference endpoint ssrf", "custom endp
 content:
   type: "concept"
 ---
+import { AgentOnly } from "../_components/AgentGuide";
 
 NemoClaw keeps provider credentials on the host and validates explicit custom endpoint URLs before saving them through security-sensitive configuration paths.
 
@@ -44,9 +45,22 @@ Legacy entries without a source, endpoints recorded by `inference set`, and diff
 
 For a public HTTP URL, NemoClaw stores the validated IP address so the downstream runtime cannot resolve the hostname again and reach another address.
 
-NemoClaw rejects DNS-backed HTTPS URLs in these paths because it cannot pin the downstream peer address while preserving TLS Server Name Indication and host validation across the OpenShell runtime boundary.
-Use an HTTPS IP-literal endpoint with a certificate valid for that address.
-Use a public HTTP endpoint only when your deployment permits unencrypted traffic.
+NemoClaw rejects DNS-backed HTTPS URLs for Hermes Provider setup, host-side `config set`, and a direct blueprint run because it cannot pin the downstream peer address while preserving TLS Server Name Indication and host validation across the OpenShell runtime boundary.
+For those paths, use an HTTPS IP-literal endpoint with a certificate valid for that address, or use a public HTTP endpoint if your deployment permits unencrypted traffic.
+
+<AgentOnly variant="openclaw,hermes">
+
+## Use a DNS-Backed HTTPS Endpoint with `inference set`
+
+`$$nemoclaw inference set --endpoint-url <url>` on an already-onboarded sandbox supports a DNS-backed HTTPS custom endpoint through the HTTPS Pin Runtime adapter.
+After SSRF validation passes, NemoClaw starts a local reverse-proxy adapter on the host that terminates a pinned, SNI-correct outbound TLS connection to the real upstream hostname, re-validating that the resolved peer IP is still public.
+The sandbox, its OpenShell provider configuration and network policy, and the persisted sandbox registry only ever see a local route through `http://host.openshell.internal:<port>/route/<id>/...`.
+The real upstream hostname never reaches the sandbox or the persisted registry.
+
+This support is specific to `inference set` on an already-onboarded sandbox.
+Hermes Provider setup, host-side `config set`, and a direct blueprint run still reject DNS-backed HTTPS URLs as described above.
+
+</AgentOnly>
 
 ## Use the Sandbox Host Alias
 

docs/reference/commands.mdx

@@ -2968,7 +2968,8 @@ NemoClaw rejects loopback, link-local, private, and internal endpoint addresses,
 For a same-provider model change, pass `--endpoint-url` with the exact canonical endpoint URL that the target sandbox registry identifies as onboarding-established.
 Missing or `inference set` provenance and every different URL remain subject to the full address validation above.
 For public HTTP URLs, NemoClaw stores the validated IP address to prevent DNS rebinding.
-DNS-backed HTTPS URLs are rejected because NemoClaw cannot pin the downstream peer address while preserving TLS SNI and host validation across the OpenShell runtime boundary; HTTPS IP-literal URLs remain supported.
+For a DNS-backed HTTPS URL, NemoClaw routes the endpoint through a local HTTPS Pin Runtime adapter that terminates a pinned, SNI-correct outbound connection to the real upstream hostname; the sandbox and the persisted registry only ever see a local `host.openshell.internal` route, never the real hostname.
+HTTPS IP-literal URLs remain supported and do not need the adapter.
 NemoClaw accepts `http://host.openshell.internal:<port>` only with an explicit port from `1024` through `65535`; this narrow exception supports NemoClaw's sandbox-to-host inference routes and is not a general private-endpoint bypass.
 `--credential-env` may also be supplied for compatible provider metadata; supported `--inference-api` values are `openai-completions`, `anthropic-messages`, and `openai-responses`.
 

docs/reference/troubleshooting.mdx

@@ -1922,7 +1922,8 @@ After the upgrade, recreate the sandbox with `$$nemoclaw onboard`.
 NemoClaw rejects an explicit custom endpoint when it resolves a public HTTPS hostname but cannot pin the same peer address across the downstream OpenShell runtime boundary while preserving TLS SNI and host validation.
 This can appear during a direct blueprint run, custom-endpoint onboarding, or a host-side `config set` write.
 <AgentOnly variant="openclaw,hermes">
-It can also appear during a runtime `$$nemoclaw inference set` switch.
+It does not appear during a runtime `$$nemoclaw inference set` switch on an already-onboarded sandbox; that command routes a DNS-backed HTTPS endpoint through a local HTTPS Pin Runtime adapter instead of rejecting it.
+Refer to [Commands](commands) for details.
 </AgentOnly>
 
 Use an HTTPS IP-literal endpoint whose certificate is valid for that address.

nemoclaw/src/blueprint/ssrf.ts

@@ -112,11 +112,12 @@ export async function validateEndpointUrl(url: string): Promise<ValidatedEndpoin
 export function safeEndpointUrlForDownstream(validated: ValidatedEndpoint): string {
   if (validated.protocol === "https:" && validated.dnsResolved) {
     throw new Error(
-      `DNS-backed HTTPS endpoint '${validated.hostname}' is not supported yet because ` +
-        "NemoClaw cannot guarantee the downstream provider connects to the same IP " +
-        "that passed SSRF validation across the OpenShell runtime boundary. " +
-        "Use an HTTPS IP-literal endpoint, an HTTP endpoint that can be DNS-pinned, " +
-        "or wait for the runtime-aware HTTPS pinning transport.",
+      `DNS-backed HTTPS endpoint '${validated.hostname}' is not supported at onboarding ` +
+        "time because this path cannot guarantee the downstream provider connects to the " +
+        "same IP that passed SSRF validation across the OpenShell runtime boundary. " +
+        "Use an HTTPS IP-literal endpoint or an HTTP endpoint that can be DNS-pinned to " +
+        "onboard. After onboarding, `inference set --endpoint-url` supports a DNS-backed " +
+        "HTTPS endpoint through the HTTPS Pin Runtime adapter.",
     );
   }
 

src/lib/actions/inference-set-compatible-provider.test.ts

@@ -2,6 +2,8 @@
 // SPDX-License-Identifier: Apache-2.0
 
 import { describe, expect, it, vi } from "vitest";
+import { ensureHttpsPinRuntimeAdapter as realEnsureHttpsPinRuntimeAdapter } from "../inference/https-pin-runtime-adapter";
+import { HTTPS_PIN_RUNTIME_ADAPTER_PROVIDER_CREDENTIAL_ENV } from "../inference/https-pin-runtime";
 import type { ConfigObject } from "../security/credential-filter";
 import { runInferenceSet } from "./inference-set";
 import { baseSession, createDeps } from "./inference-set.test-support";
@@ -218,21 +220,29 @@ describe("runInferenceSet compatible providers", () => {
         },
       },
     });
+    // The DNS-backed HTTPS endpoint is pinned via the HTTPS-pin runtime
+    // adapter, so the persisted endpointUrl is the adapter's local route base
+    // URL, not the raw operator-supplied hostname — mirroring the existing
+    // HTTP precedent of persisting the validated/pinned address. The
+    // persisted credentialEnv is the adapter's own bearer-token env var, not
+    // the real upstream secret: the sandbox only ever authenticates to the
+    // local adapter, which injects the real credential on the outbound leg
+    // (PRA-1, #6141).
     expect(deps.calls.updateSandbox.mock.calls.at(-1)).toEqual([
       "alpha",
       expect.objectContaining({
         provider: "compatible-endpoint",
         model: "mock-responses-model",
-        endpointUrl: "https://compatible.example/v1",
-        credentialEnv: "COMPATIBLE_API_KEY",
+        endpointUrl: "http://host.openshell.internal:11438/route/test-route",
+        credentialEnv: HTTPS_PIN_RUNTIME_ADAPTER_PROVIDER_CREDENTIAL_ENV,
         preferredInferenceApi: "openai-responses",
       }),
     ]);
     expect(deps.getSession()).toMatchObject({
       provider: "compatible-endpoint",
       model: "mock-responses-model",
-      endpointUrl: "https://compatible.example/v1",
-      credentialEnv: "COMPATIBLE_API_KEY",
+      endpointUrl: "http://host.openshell.internal:11438/route/test-route",
+      credentialEnv: HTTPS_PIN_RUNTIME_ADAPTER_PROVIDER_CREDENTIAL_ENV,
       preferredInferenceApi: "openai-responses",
     });
     expect(deps.calls.restartSandboxGateway).toHaveBeenCalledWith("alpha");
@@ -317,6 +327,12 @@ describe("runInferenceSet compatible providers", () => {
         },
         rewriteConfigUrlsWithDnsPinning: (value) =>
           actualConfig.rewriteConfigUrlsWithDnsPinning(value, lookup),
+        // DNS-backed HTTPS endpoints (the "DNS-private" case below) route
+        // through the HTTPS-pin runtime adapter instead of
+        // rewriteConfigUrlsWithDnsPinning, so its real SSRF preflight is
+        // exercised here too, with the same injected DNS lookup.
+        ensureHttpsPinRuntimeAdapter: (adapterOptions) =>
+          realEnsureHttpsPinRuntimeAdapter({ ...adapterOptions, lookup }),
       });
 
       await expect(

src/lib/actions/inference-set-endpoint-security.test.ts

@@ -57,4 +57,23 @@ describe("custom inference endpoint DNS pinning", () => {
       ),
     ).rejects.toThrow(/DNS-backed HTTPS URLs are not supported/);
   });
+
+  it("adds the HTTPS Pin Runtime adapter hint only at the inference-set call site, not in the generic config validator's own message (#6141)", async () => {
+    const lookup = vi.fn(async () => [{ address: "93.184.216.34", family: 4 }]);
+
+    // The generic validator (also used by plain `config set` for arbitrary
+    // fields) must not mention inference set or the adapter -- it has no way
+    // to know the field it's validating is an inference endpoint.
+    await expect(
+      rewriteConfigUrlsWithDnsPinning("https://public-endpoint.example/v1/", lookup),
+    ).rejects.toThrow(/^(?!.*(?:inference set|HTTPS Pin Runtime adapter)).*$/is);
+
+    // normalizeCustomEndpointUrl is only ever called for `inference set
+    // --endpoint-url`, so it appends the adapter-specific hint itself.
+    await expect(
+      normalizeCustomEndpointUrl("https://public-endpoint.example/v1/", (value) =>
+        rewriteConfigUrlsWithDnsPinning(value, lookup),
+      ),
+    ).rejects.toThrow(/HTTPS Pin Runtime adapter/);
+  });
 });

src/lib/actions/inference-set-gateway-route-containment.test.ts

@@ -6,6 +6,7 @@ import os from "node:os";
 import path from "node:path";
 import { afterEach, describe, expect, it, vi } from "vitest";
 import { withGatewayRouteMutationLock } from "../inference/gateway-route-mutation-lock";
+import { HTTPS_PIN_RUNTIME_ADAPTER_PROVIDER_CREDENTIAL_ENV } from "../inference/https-pin-runtime";
 import type { ConfigObject } from "../security/credential-filter";
 import type { SandboxEntry } from "../state/registry";
 import { runInferenceSet } from "./inference-set";
@@ -28,6 +29,7 @@ const entry = (name: string, overrides: Partial<SandboxEntry> = {}): SandboxEntr
 describe("runtime shared gateway route containment", () => {
   afterEach(() => {
     vi.unstubAllEnvs();
+    delete process.env[HTTPS_PIN_RUNTIME_ADAPTER_PROVIDER_CREDENTIAL_ENV];
   });
 
   it("rejects an ambient gateway endpoint before OpenShell prep or state mutation", async () => {
@@ -213,7 +215,7 @@ describe("runtime shared gateway route containment", () => {
     const peer = entry("late-peer", {
       provider: "compatible-endpoint",
       model: "custom/model",
-      endpointUrl: "https://peer.example.test/v1",
+      endpointUrl: "http://peer.example.test/v1",
       credentialEnv: "COMPATIBLE_API_KEY",
       preferredInferenceApi: "openai-completions",
     });
@@ -224,13 +226,16 @@ describe("runtime shared gateway route containment", () => {
       .mockReturnValue({ sandboxes: [alpha, peer], defaultSandbox: "alpha" });
     deps.listSandboxes = listSandboxes;
 
+    // HTTP (not HTTPS) so this test exercises rewriteUrlWithDnsPinning directly
+    // to create the async validation gap; DNS-backed HTTPS endpoints route
+    // through the HTTPS-pin runtime adapter instead.
     await expect(
       runInferenceSet(
         {
           provider: "compatible-endpoint",
           model: "custom/model",
           sandboxName: "alpha",
-          endpointUrl: "https://alpha.example.test/v1",
+          endpointUrl: "http://alpha.example.test/v1",
           credentialEnv: "COMPATIBLE_API_KEY",
           inferenceApi: "openai-completions",
         },
@@ -279,8 +284,11 @@ describe("runtime shared gateway route containment", () => {
   });
 
   it("catches a DNS change between the preliminary and finalized gateway route checks", async () => {
-    const firstEndpoint = "https://first.example.test/v1";
-    const secondEndpoint = "https://second.example.test/v1";
+    // HTTP (not HTTPS) so this test exercises rewriteUrlWithDnsPinning directly;
+    // DNS-backed HTTPS endpoints route through the HTTPS-pin runtime adapter
+    // instead (see inference-set-https-pin-runtime.test.ts).
+    const firstEndpoint = "http://first.example.test/v1";
+    const secondEndpoint = "http://second.example.test/v1";
     const customRoute = {
       provider: "compatible-endpoint",
       model: "custom/model",
@@ -304,6 +312,7 @@ describe("runtime shared gateway route containment", () => {
       sandboxes: [alpha, peer],
     });
     const rewriteUrlWithDnsPinning = vi.fn().mockResolvedValueOnce(secondEndpoint);
+    const ensureHttpsPinRuntimeAdapter = vi.fn();
 
     await expect(
       finalizeInferenceSetRoute({
@@ -315,11 +324,113 @@ describe("runtime shared gateway route containment", () => {
         onboardEndpointUrl: null,
         getSandboxes: () => [alpha, peer],
         rewriteUrlWithDnsPinning,
+        ensureHttpsPinRuntimeAdapter,
       }),
     ).rejects.toThrow("custom-peer");
 
     expect(rewriteUrlWithDnsPinning).toHaveBeenCalledOnce();
     expect(rewriteUrlWithDnsPinning).toHaveBeenCalledWith(firstEndpoint);
+    expect(ensureHttpsPinRuntimeAdapter).not.toHaveBeenCalled();
+  });
+
+  it("serializes concurrent HTTPS-pin adapter route provisions for different gateways so their process.env token writes cannot interleave (#6141)", async () => {
+    const firstEndpoint = "https://race-a.example.test/v1";
+    const secondEndpoint = "https://race-b.example.test/v1";
+    const routeFor = (endpointUrl: string) => ({
+      provider: "compatible-endpoint",
+      model: "custom/model",
+      endpointUrl,
+      credentialEnv: "COMPATIBLE_API_KEY",
+      preferredInferenceApi: "openai-completions",
+    });
+    // Different gateways (distinguished by gatewayPort -- resolveSandboxGatewayName
+    // derives the effective gateway name from the port, not the gatewayName field,
+    // whenever a valid port is present): the per-gateway route-mutation lock does
+    // not serialize these two against each other, so only the credential-env lock
+    // added for this race can prevent their process.env writes from interleaving.
+    const alpha = entry("alpha", routeFor(firstEndpoint));
+    const beta = entry("beta", {
+      ...routeFor(secondEndpoint),
+      gatewayName: "nemoclaw-9091",
+      gatewayPort: 9091,
+    });
+    const preparedFor = (target: SandboxEntry, endpointUrl: string) =>
+      prepareInferenceSetRoute({
+        entry: target,
+        sandboxName: target.name,
+        provider: "compatible-endpoint",
+        model: "custom/model",
+        customRoute: {
+          endpointUrl,
+          credentialEnv: "COMPATIBLE_API_KEY",
+          inferenceApi: "openai-completions",
+        },
+        session: null,
+        sandboxes: [alpha, beta],
+      });
+
+    let releaseA: () => void = () => {};
+    const aGate = new Promise<void>((resolve) => {
+      releaseA = resolve;
+    });
+    const callOrder: string[] = [];
+    const adapterBehaviorByEndpoint: Record<
+      string,
+      () => Promise<{ baseUrl: string; credentialEnv: string; token: string }>
+    > = {
+      [firstEndpoint]: async () => {
+        callOrder.push("a-start");
+        await aGate;
+        callOrder.push("a-end");
+        return {
+          baseUrl: "http://host.openshell.internal:1/route/a",
+          credentialEnv: HTTPS_PIN_RUNTIME_ADAPTER_PROVIDER_CREDENTIAL_ENV,
+          token: "token-a",
+        };
+      },
+      [secondEndpoint]: async () => {
+        callOrder.push("b-start");
+        return {
+          baseUrl: "http://host.openshell.internal:1/route/b",
+          credentialEnv: HTTPS_PIN_RUNTIME_ADAPTER_PROVIDER_CREDENTIAL_ENV,
+          token: "token-b",
+        };
+      },
+    };
+    const ensureHttpsPinRuntimeAdapter = vi.fn(async (options: { endpointUrl: string }) =>
+      adapterBehaviorByEndpoint[options.endpointUrl](),
+    );
+    const rewriteUrlWithDnsPinning = vi.fn(async (value: unknown) => value as string);
+
+    const finalize = (target: SandboxEntry, endpointUrl: string) =>
+      finalizeInferenceSetRoute({
+        prepared: preparedFor(target, endpointUrl),
+        sandboxName: target.name,
+        provider: "compatible-endpoint",
+        model: "custom/model",
+        canReuseRecordedRoute: false,
+        onboardEndpointUrl: null,
+        getSandboxes: () => [alpha, beta],
+        rewriteUrlWithDnsPinning,
+        ensureHttpsPinRuntimeAdapter,
+      });
+
+    const callA = finalize(alpha, firstEndpoint);
+    await vi.waitFor(() => expect(callOrder).toContain("a-start"));
+    const callB = finalize(beta, secondEndpoint);
+
+    // While A is gated inside the lock, B must not reach its own adapter
+    // call (and process.env write) yet -- proving the two invocations are
+    // serialized rather than interleaved.
+    await new Promise((resolve) => setTimeout(resolve, 20));
+    expect(callOrder).toEqual(["a-start"]);
+
+    releaseA();
+    const [resultA, resultB] = await Promise.all([callA, callB]);
+
+    expect(callOrder).toEqual(["a-start", "a-end", "b-start"]);
+    expect(resultA.registryMetadata.endpointUrl).toBe("http://host.openshell.internal:1/route/a");
+    expect(resultB.registryMetadata.endpointUrl).toBe("http://host.openshell.internal:1/route/b");
   });
 
   it("blocks an incomplete legacy custom target even without a peer (#6315)", async () => {