Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,7 @@
COPY scripts/patch-openclaw-mcp-npx.mts /usr/local/lib/nemoclaw/patch-openclaw-mcp-npx.mts
COPY scripts/patch-openclaw-mcp-reliability.mts /usr/local/lib/nemoclaw/patch-openclaw-mcp-reliability.mts
COPY scripts/patch-openclaw-issue-4434-diagnostics.mts /usr/local/lib/nemoclaw/patch-openclaw-issue-4434-diagnostics.mts
COPY scripts/patch-openclaw-managed-transport-diagnostics.mts /usr/local/lib/nemoclaw/patch-openclaw-managed-transport-diagnostics.mts
COPY scripts/patch-openclaw-device-self-approval.mts /usr/local/lib/nemoclaw/patch-openclaw-device-self-approval.mts
COPY scripts/extract-semver.sh /usr/local/lib/nemoclaw/extract-semver
COPY scripts/patch-openclaw-shared-state-permissions.mts /usr/local/lib/nemoclaw/patch-openclaw-shared-state-permissions.mts
Expand Down Expand Up @@ -390,6 +391,7 @@
/usr/local/lib/nemoclaw/patch-openclaw-mcp-npx.mts \
/usr/local/lib/nemoclaw/patch-openclaw-mcp-reliability.mts \
/usr/local/lib/nemoclaw/patch-openclaw-issue-4434-diagnostics.mts \
/usr/local/lib/nemoclaw/patch-openclaw-managed-transport-diagnostics.mts \
/usr/local/lib/nemoclaw/patch-openclaw-device-self-approval.mts \
/usr/local/lib/nemoclaw/extract-semver \
/usr/local/lib/nemoclaw/patch-openclaw-shared-state-permissions.mts \
Expand Down Expand Up @@ -1019,6 +1021,20 @@
RUN node --experimental-strip-types /usr/local/lib/nemoclaw/patch-openclaw-mcp-reliability.mts \
/usr/local/lib/node_modules/openclaw/dist

# Emit a redacted managed-transport diagnostic when a remote Streamable HTTP MCP
# request fails. OpenClaw 2026.7.1 surfaces only the transport error text, which
# does not say whether policy, CONNECT, TLS, the upstream connection, the
# request, or response headers failed. The fetch-boundary wrapper is
# failure-only, never retries, never alters the request, and never reads a 2xx
# body, so streaming responses stay behaviorally unchanged. It is inert unless
# OPENSHELL_SANDBOX=1.
#
# Removal criterion: drop when upstream OpenClaw emits phase-classified,
# redacted transport diagnostics for remote MCP fetch failures.
# hadolint ignore=DL3059
RUN node --experimental-strip-types /usr/local/lib/nemoclaw/patch-openclaw-managed-transport-diagnostics.mts \
/usr/local/lib/node_modules/openclaw/dist

# Run the compact tool catalog shim for OpenClaw selection runtimes that still
# need it. OpenClaw 2026.7.1 ships a built-in catalog surface, so the script
# skips cleanly after classifying the compiled selection-*.js shape.
Expand Down Expand Up @@ -1173,10 +1189,10 @@
# CHAT_UI_URL is a non-loopback address (Brev Launchable, remote deployments)
# since terminal-based pairing is impossible in those contexts.
# Default: "0" (device auth enabled for local deployments — secure by default).
ARG NEMOCLAW_DISABLE_DEVICE_AUTH=0

Check warning on line 1192 in Dockerfile

View workflow job for this annotation

GitHub Actions / PR build and direct managed startup (OpenClaw)

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "NEMOCLAW_DISABLE_DEVICE_AUTH") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/
# Internal audit provenance for the opt-out above. Standard onboarding rewrites
# this to managed-onboard; direct image builders retain operator provenance.
ARG NEMOCLAW_DEVICE_AUTH_OPT_OUT_SOURCE=operator

Check warning on line 1195 in Dockerfile

View workflow job for this annotation

GitHub Actions / PR build and direct managed startup (OpenClaw)

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "NEMOCLAW_DEVICE_AUTH_OPT_OUT_SOURCE") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/
# Compatibility build arg for older custom Dockerfiles and rebuild tooling.
# NemoClaw-managed images intentionally do not consume it; gateway auth tokens
# are generated at container startup and are never baked into image layers.
Expand Down Expand Up @@ -1210,7 +1226,7 @@
# NEMOCLAW_MESSAGING_PLAN_B64 intentionally remains ARG-only: Docker exposes it
# to build RUN processes without retaining the full plan in the final image env.
# Direct ARG interpolation into inline source is a code injection vector (C-2).
ENV NEMOCLAW_MODEL=${NEMOCLAW_MODEL} \

Check warning on line 1229 in Dockerfile

View workflow job for this annotation

GitHub Actions / PR build and direct managed startup (OpenClaw)

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "NEMOCLAW_DEVICE_AUTH_OPT_OUT_SOURCE") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 1229 in Dockerfile

View workflow job for this annotation

GitHub Actions / PR build and direct managed startup (OpenClaw)

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "NEMOCLAW_DISABLE_DEVICE_AUTH") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/
NEMOCLAW_INFERENCE_PROVIDER_ID=${NEMOCLAW_INFERENCE_PROVIDER_ID} \
NEMOCLAW_UPSTREAM_PROVIDER=${NEMOCLAW_UPSTREAM_PROVIDER} \
NEMOCLAW_PRIMARY_MODEL_REF=${NEMOCLAW_PRIMARY_MODEL_REF} \
Expand Down
1 change: 1 addition & 0 deletions ci/full-e2e-cold-path-calibration.json
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,7 @@
"scripts/patch-openclaw-mcp-npx.mts",
"scripts/patch-openclaw-mcp-reliability.mts",
"scripts/patch-openclaw-issue-4434-diagnostics.mts",
"scripts/patch-openclaw-managed-transport-diagnostics.mts",
"scripts/patch-openclaw-device-self-approval.mts",
"scripts/verify-wechat-runtime-lock.mts",
"scripts/lib/reviewed-npm-archive.mts",
Expand Down
2 changes: 1 addition & 1 deletion ci/source-architecture-budget.json
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,6 @@
"src/lib/actions/sandbox": 183,
"src/lib/state": 37,
"src/lib/inference": 62,
"scripts": 45
"scripts": 46
}
}
67 changes: 67 additions & 0 deletions docs/reference/troubleshoot-mcp-servers.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,73 @@ A refused or unreachable endpoint is also not retried inside a turn; check the e
A catalog that contains a server diagnostic is not cached for the session.
NemoClaw rebuilds the catalog on the next agent turn and contacts the affected server again, so a brief upstream outage clears on its own and a repaired credential or policy takes effect without a new session.
A catalog without server diagnostics is cached and reused as usual.

## A Remote MCP Request Fails Without Saying Where

A transport error such as `fetch failed` does not say which step failed.
For OpenClaw remote Streamable HTTP requests, NemoClaw adds a failure-only diagnostic at the `fetch` boundary.
When that request fails inside a sandbox, OpenClaw attempts to write one `managed_transport_failure` line group to the sandbox log:

```bash
$$nemoclaw <sandbox> logs --tail 100
```

```text
[nemoclaw] managed_transport_failure
[nemoclaw] consumer=mcp
[nemoclaw] route=proxy_configured
[nemoclaw] proxy=127.0.0.1:3128
[nemoclaw] target=mcp.example.com:443
[nemoclaw] transport_phase=response_headers
[nemoclaw] http_status=503
[nemoclaw] elapsed_ms=1512
[nemoclaw] server=envoy
[nemoclaw] x_envoy_response_flags=UF,URX
[nemoclaw] session_present=true
[nemoclaw] diagnostic_id=<generated-id>
```

Read `transport_phase` first.
It states which transport step failed:

| Transport phase | What failed |
|-----------------|-------------|
| `policy` | The network policy refused the destination |
| `connect` | The proxy CONNECT tunnel failed for a reason other than a denial |
| `tls` | TLS setup or certificate validation failed |
| `app_connect` | The upstream connection was refused, unreachable, or unresolved |
| `request` | The request failed before any response arrived |
| `response_headers` | A response arrived and its status or headers report the failure |

`route=proxy_configured` states that OpenClaw received a supported proxy environment variable.
`route=unknown` states that the diagnostic did not observe one.
These values report configuration evidence and do not prove whether the failed request used a proxy.
When present, `proxy` contains the configured proxy endpoint without credentials.
`server`, `via`, `x_request_id`, and the `x_envoy_*` fields are the proxy's own account of the failure and appear only when the proxy sends them.
A proxy-generated response does not by itself prove whether the proxy or its upstream produced the failure; keep both the status and the cause chain when reporting one.

The line group is written only for a failure, so normal traffic adds no logging.
A 2xx response does not emit this diagnostic or read its body.
This fetch boundary cannot report a failure that occurs while the caller later reads a 2xx response body.
For a non-2xx response, OpenClaw does not wait for asynchronous body sampling before it receives the response.
Non-2xx response diagnostics are best-effort.
If detached collection fails, OpenClaw still returns the original response and the line group can be absent.
The line group can also be absent if OpenClaw exits before collection completes.
The sampler waits at most 250 ms and retains at most 2,048 response bytes.
The redacted `error_body` value contains at most 2,048 UTF-8 bytes before JSON encoding.
It redacts session identifiers, bearer tokens, structured credentials such as `access_token`, `refresh_token`, and `client_secret`, and known token prefixes.
The diagnostic does not include a JSON-RPC operation because this boundary sees only the endpoint.

To find the matching sandbox audit record, read the audit log for the same endpoint around the failure time:

```bash
$$nemoclaw <sandbox> logs --audit --tail 200
```

Match on `target` and the timestamp.
`diagnostic_id` is a local identifier created for one line group.
It is not a distributed trace identifier and does not appear in OpenShell audit events.
Correlate with OpenShell by endpoint and time until OpenShell records a shared identifier.
</AgentOnly>

## Policy or Provider Ownership Drifted
Expand Down
66 changes: 66 additions & 0 deletions docs/security/openclaw-2026.7.1-dependency-review.md
Original file line number Diff line number Diff line change
Expand Up @@ -433,6 +433,72 @@ The proof asserts that the rejected-credential scenario is contacted once per ru

Removal criterion: drop this patch when the reviewed OpenClaw release provides equivalent bounded startup retry, negative-catalog invalidation, and temporary-transport failure attribution.

## Managed Outbound Transport Diagnostics

`scripts/patch-openclaw-managed-transport-diagnostics.mts` is a version-scoped, fail-closed compatibility patch for issue #7957.
In `2026.7.1`, a failed remote Streamable HTTP MCP request surfaces only the transport error text, such as `fetch failed` or a request timeout.
That text does not say whether policy evaluation, proxy CONNECT, TLS setup, the upstream connection, the request, or the response headers failed.
An operator therefore has to correlate agent output with OpenShell audit logs by hand.

The patch wraps the `fetch` passed to `StreamableHTTPClientTransport` and identifies the compiled target by the `"openclaw-bundle-mcp"` client identity.
It requires the rewritten anchor to appear exactly once.
The sibling SSE transport boundary is deliberately left unwrapped.
An unrecognized compiled shape fails the image build instead of silently skipping.
`--audit` re-verifies the applied state.

Reviewed behavior:

- Failure-only.
A 2xx response returns untouched and emits nothing, so normal traffic produces no per-request logging.
- The wrapper never retries, never alters the request, never changes proxy selection, and never weakens TLS verification.
It rethrows a transport error unchanged.
- `route=proxy_configured` means that `HTTPS_PROXY`, `https_proxy`, `HTTP_PROXY`, or `http_proxy` was configured.
`route=unknown` means that the diagnostic did not observe one of those variables.
These values report configuration evidence and do not prove whether the failed request used a proxy.
- For a non-2xx response, the wrapper returns the original response without waiting for asynchronous sampling of `response.clone()`.
The diagnostic task waits at most 250 ms and retains at most 2,048 response bytes for an allowed content type.
It limits the redacted `error_body` value to 2,048 UTF-8 bytes before JSON encoding.
- Non-2xx response diagnostics are best-effort.
If detached collection fails, the wrapper still returns the original response and the diagnostic can be absent.
The diagnostic can also be absent if the process exits before collection completes.
- The wrapper does not inspect a 2xx response body.
This fetch boundary cannot classify a failure that occurs while the caller later reads that body.
- Response metadata is an allowlist: `content-type`, `retry-after`, `server`, `via`, `x-request-id`, `x-envoy-attempt-count`, `x-envoy-decorator-operation`, `x-envoy-response-flags`, and `x-envoy-upstream-service-time`.
Emitted header keys use underscores.
The wrapper does not access any other response header for diagnostics.
- Allowed header values, error bodies, and cause messages pass the same bounded redaction.
It removes session identifiers, bearer tokens, known token prefixes, and structured credentials such as `access_token`, `refresh_token`, and `client_secret`.
- The cause chain is bounded to 8 entries and keeps only error name, code, errno, syscall, address family, port, and a redacted message.
The peer address is not recorded.
- Session state is reported as a boolean.
The `mcp-session-id` value is never emitted.
- The `transport_phase` field classifies a thrown failure as `policy`, `connect`, `tls`, `app_connect`, `response_headers`, or `request`.
A policy denial takes precedence over its accompanying transport code.
Without a higher-priority transport-phase signal, a thrown `UND_ERR_HEADERS_TIMEOUT` failure is classified as `response_headers`.
That diagnostic has no response headers or `http_status` because `fetch` did not return a response.
- A returned non-2xx response sets `transport_phase=response_headers`.
It carries `http_status` and any allowlisted response headers that are present.
- The fetch boundary does not expose the JSON-RPC operation, so the diagnostic records the endpoint without an `operation` field.
- Each emitted diagnostic receives a local 32-character hexadecimal `diagnostic_id`.
- The wrapper is inert unless `OPENSHELL_SANDBOX=1`, so it does not change host-side behavior.

`diagnostic_id` is not a distributed trace identifier and does not correlate with an OpenShell audit event.
`NVIDIA/OpenShell#2508` tracks span emission from the sandbox supervisor, and the OCSF `http_request` object in the pinned OpenShell `0.0.85` has no slot for a request-scoped correlation identifier, so a shared identifier is not representable today.
The local identifier distinguishes application-side diagnostics, but operators still correlate each diagnostic with OpenShell audit events by endpoint and time.

Managed transport diagnostics remains separate from `scripts/patch-openclaw-mcp-reliability.mts`.
The diagnostics patch wraps every failed remote Streamable HTTP fetch and has its own exact-shape audit and removal condition.
The reliability patch owns startup catalog and retry behavior.
The two patches compose independently.

The injected helper in `scripts/patch-openclaw-managed-transport-diagnostics.mts` is the shipped runtime source of truth.
`test/openclaw-managed-transport-diagnostics-patch.test.ts` executes that exact helper.
It pins the compiled preimage, patch idempotence, fail-closed rejection of an unrecognized shape, and the untouched SSE boundary.
It also covers failure-only emission, no-retry and unchanged-response contracts, asynchronous body sampling, byte and time bounds, redaction, the header allowlist, local diagnostic identifiers, session-presence reporting, transport-phase classification, route evidence, and sandbox gating.
A reusable source schema is deferred until a production consumer requires one.

Removal criterion: drop this patch when the reviewed OpenClaw release emits redacted diagnostics classified by transport phase for remote MCP fetch failures.

## Gateway Startup Migration Compatibility

OpenClaw `2026.7.1` requires its migration checkpoint to complete without
Expand Down
Loading
Loading