Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -1049,8 +1049,10 @@
# request fails. OpenClaw 2026.7.1 surfaces only the transport error text, which
# does not say whether policy, CONNECT, TLS, the upstream connection, the
# request, or response headers failed. The fetch-boundary wrapper is
# failure-only, never retries, never alters the request, and never reads a 2xx
# body, so streaming responses stay behaviorally unchanged. It is inert unless
# failure-only by default, never retries, never alters the request, and never
# reads a 2xx body, so streaming responses stay behaviorally unchanged. Successful request
# timing is silent unless NEMOCLAW_MCP_SHADOW_DIAGNOSTICS=1 is explicitly
# forwarded into an OpenClaw sandbox. The wrapper is inert unless
# OPENSHELL_SANDBOX=1.
#
# Removal criterion: drop when upstream OpenClaw emits phase-classified,
Expand Down Expand Up @@ -1215,10 +1217,10 @@
# CHAT_UI_URL is a non-loopback address (Brev Launchable, remote deployments)
# since terminal-based pairing is impossible in those contexts.
# Default: "0" (device auth enabled for local deployments — secure by default).
ARG NEMOCLAW_DISABLE_DEVICE_AUTH=0

Check warning on line 1220 in Dockerfile

View workflow job for this annotation

GitHub Actions / PR build and direct managed startup (OpenClaw)

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "NEMOCLAW_DISABLE_DEVICE_AUTH") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 1220 in Dockerfile

View workflow job for this annotation

GitHub Actions / PR build and direct managed startup (OpenClaw)

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "NEMOCLAW_DISABLE_DEVICE_AUTH") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/
# Internal audit provenance for the opt-out above. Standard onboarding rewrites
# this to managed-onboard; direct image builders retain operator provenance.
ARG NEMOCLAW_DEVICE_AUTH_OPT_OUT_SOURCE=operator

Check warning on line 1223 in Dockerfile

View workflow job for this annotation

GitHub Actions / PR build and direct managed startup (OpenClaw)

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "NEMOCLAW_DEVICE_AUTH_OPT_OUT_SOURCE") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 1223 in Dockerfile

View workflow job for this annotation

GitHub Actions / PR build and direct managed startup (OpenClaw)

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "NEMOCLAW_DEVICE_AUTH_OPT_OUT_SOURCE") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/
# Compatibility build arg for older custom Dockerfiles and rebuild tooling.
# NemoClaw-managed images intentionally do not consume it; gateway auth tokens
# are generated at container startup and are never baked into image layers.
Expand Down Expand Up @@ -1252,7 +1254,7 @@
# NEMOCLAW_MESSAGING_PLAN_B64 intentionally remains ARG-only: Docker exposes it
# to build RUN processes without retaining the full plan in the final image env.
# Direct ARG interpolation into inline source is a code injection vector (C-2).
ENV NEMOCLAW_MODEL=${NEMOCLAW_MODEL} \

Check warning on line 1257 in Dockerfile

View workflow job for this annotation

GitHub Actions / PR build and direct managed startup (OpenClaw)

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "NEMOCLAW_DEVICE_AUTH_OPT_OUT_SOURCE") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 1257 in Dockerfile

View workflow job for this annotation

GitHub Actions / PR build and direct managed startup (OpenClaw)

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "NEMOCLAW_DISABLE_DEVICE_AUTH") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 1257 in Dockerfile

View workflow job for this annotation

GitHub Actions / PR build and direct managed startup (OpenClaw)

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "NEMOCLAW_DEVICE_AUTH_OPT_OUT_SOURCE") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 1257 in Dockerfile

View workflow job for this annotation

GitHub Actions / PR build and direct managed startup (OpenClaw)

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "NEMOCLAW_DISABLE_DEVICE_AUTH") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/
NEMOCLAW_INFERENCE_PROVIDER_ID=${NEMOCLAW_INFERENCE_PROVIDER_ID} \
NEMOCLAW_UPSTREAM_PROVIDER=${NEMOCLAW_UPSTREAM_PROVIDER} \
NEMOCLAW_PRIMARY_MODEL_REF=${NEMOCLAW_PRIMARY_MODEL_REF} \
Expand Down
1 change: 1 addition & 0 deletions docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4139,6 +4139,7 @@ OpenClaw-specific onboarding configuration:
| `BRAVE_API_KEY` | Brave Search API key | Supplies and implicitly selects Brave Search when no web search provider is set. NemoClaw validates the key and stores it in OpenShell rather than the sandbox. |
| `TAVILY_API_KEY` | Tavily Search API key | Supplies and implicitly selects Tavily Search when no provider is set and no Brave key is available. NemoClaw validates the key and stores it in OpenShell rather than the sandbox. |
| `NEMOCLAW_AGENT_TIMEOUT` | positive integer (seconds) | Overrides `agents.defaults.timeoutSeconds` in the built OpenClaw config. Raise for slow inference. |
| `NEMOCLAW_MCP_SHADOW_DIAGNOSTICS` | literal `1` to enable | Forwards opt-in successful Streamable HTTP MCP timing diagnostics to a newly created or rebuilt OpenClaw sandbox. It does not change timeouts, retries, requests, or responses. Unset it and rebuild after evidence collection to restore failure-only logging. Other values are ignored. |
| `NEMOCLAW_AUTO_PAIR_SLOW_INTERVAL_SECS` | positive number of seconds | Sets the post-pairing poll cadence for the in-sandbox OpenClaw auto-pair watcher. Defaults to `5` so late allowlisted CLI and browser scope upgrades are approved before clients time out. Raise only on load-sensitive gateways. |
| `NEMOCLAW_AUTO_PAIR_FAST_REENTRY_POLLS` | positive integer | Sets how many fast polls run after the watcher observes a fresh allowlisted scope-upgrade request. Defaults to `5`; set lower only when you need to reduce gateway polling. |
| `NEMOCLAW_AUTO_PAIR_FAST_REENTRY_INTERVAL_SECS` | positive number of seconds | Sets the fast-reentry interval after a fresh allowlisted scope-upgrade request. Defaults to `1`. |
Expand Down
76 changes: 68 additions & 8 deletions docs/reference/troubleshoot-mcp-servers.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
# SPDX-License-Identifier: Apache-2.0
title: "Troubleshoot MCP Servers"
sidebar-title: "Troubleshoot MCP Servers"
description: "Diagnose managed MCP credential resolution, DNS pin drift, incomplete transactions, capability gaps, policy drift, and lifecycle-lock failures."
description-agent: "Provides symptom-based remediation for NemoClaw-managed MCP servers, including trusted private endpoints. Use when mcp add, status, restart, remove, rebuild, or destroy does not converge."
keywords: ["troubleshoot nemoclaw mcp", "mcp credential resolution", "mcp dns pin drift", "mcp policy drift", "mcp transaction"]
description: "Diagnose managed MCP transport failures, timeout evidence, credential resolution, DNS pin drift, incomplete transactions, capability gaps, policy drift, and lifecycle-lock failures."
description-agent: "Provides symptom-based remediation for NemoClaw-managed MCP servers, including transport diagnostics and trusted private endpoints. Use when MCP requests, add, status, restart, remove, rebuild, or destroy do not converge."
keywords: ["troubleshoot nemoclaw mcp", "mcp transport diagnostics", "mcp timeout evidence", "mcp credential resolution", "mcp dns pin drift", "mcp policy drift", "mcp transaction"]
content:
type: "troubleshooting"
---
Expand Down Expand Up @@ -102,12 +102,20 @@ $$nemoclaw <sandbox> logs --tail 100
```text
[nemoclaw] managed_transport_failure
[nemoclaw] consumer=mcp
[nemoclaw] mcp_server=gitlab
[nemoclaw] operation=tools/list
[nemoclaw] transport_generation=1
[nemoclaw] request_sequence=3
[nemoclaw] route=proxy_configured
[nemoclaw] proxy=127.0.0.1:3128
[nemoclaw] target=mcp.example.com:443
[nemoclaw] connection_timeout_ms=30000
[nemoclaw] request_timeout_ms=60000
[nemoclaw] catalog_list_timeout_ms=1500
[nemoclaw] effective_timeout_ms=1500
[nemoclaw] transport_phase=response_headers
[nemoclaw] http_status=503
[nemoclaw] elapsed_ms=1512
[nemoclaw] elapsed_ms=412
[nemoclaw] server=envoy
[nemoclaw] x_envoy_response_flags=UF,URX
[nemoclaw] session_present=true
Expand All @@ -130,11 +138,18 @@ It states which transport step failed:
`route=unknown` states that the diagnostic did not observe one.
These values report configuration evidence and do not prove whether the failed request used a proxy.
When present, `proxy` contains the configured proxy endpoint without credentials.
`mcp_server` contains the configured MCP server name when it passes validation and redaction.
`operation` contains a validated JSON-RPC method, such as `initialize`, `tools/list`, or `tools/call`.
For other request shapes, it contains `rpc/unknown`, `transport/listen`, or `transport/close` when that transport action is known.
The diagnostic never records JSON-RPC parameters, tool names, tool arguments, or successful response bodies.
`transport_generation` identifies one constructed transport for the configured server.
`request_sequence` counts requests through that transport.
The connection, request, catalog-list, and effective timeout fields report the resolved millisecond budgets for the request operation.
`server`, `via`, `x_request_id`, and the `x_envoy_*` fields are the proxy's own account of the failure and appear only when the proxy sends them.
A proxy-generated response does not by itself prove whether the proxy or its upstream produced the failure; keep both the status and the cause chain when reporting one.

The line group is written only for a failure, so normal traffic adds no logging.
A 2xx response does not emit this diagnostic or read its body.
By default, the line group is written only for a failure, so normal traffic adds no logging.
A 2xx response does not emit this failure diagnostic or read its body.
This fetch boundary cannot report a failure that occurs while the caller later reads a 2xx response body.
For a non-2xx response, OpenClaw does not wait for asynchronous body sampling before it receives the response.
Non-2xx response diagnostics are best-effort.
Expand All @@ -143,7 +158,51 @@ The line group can also be absent if OpenClaw exits before collection completes.
The sampler waits at most 250 ms and retains at most 2,048 response bytes.
The redacted `error_body` value contains at most 2,048 UTF-8 bytes before JSON encoding.
It redacts session identifiers, bearer tokens, structured credentials such as `access_token`, `refresh_token`, and `client_secret`, and known token prefixes.
The diagnostic does not include a JSON-RPC operation because this boundary sees only the endpoint.

### Collect Shadow Timeout Evidence

Set `NEMOCLAW_MCP_SHADOW_DIAGNOSTICS=1` before you create or rebuild an OpenClaw sandbox to collect successful request timing without applying a timeout change.
NemoClaw accepts only the literal value `1` and forwards it only to OpenClaw sandboxes.
The setting is disabled by default and affects only remote Streamable HTTP MCP requests inside an OpenShell sandbox.

<Warning>
Enabling shadow diagnostics attempts to record one timing event for every successful remote Streamable HTTP MCP request and increases log volume.
Each event includes the target host and port, configured server name, operation, and timeout metadata, but excludes URL paths, tool arguments, and successful response bodies.
Rebuilding also interrupts the agent while NemoClaw replaces the sandbox and restores its managed state.
</Warning>

For an existing sandbox, export the setting and rebuild:

```bash
export NEMOCLAW_MCP_SHADOW_DIAGNOSTICS=1
$$nemoclaw <sandbox> rebuild
```

For a new sandbox, export the setting before `$$nemoclaw onboard` creates it.

When enabled, each successful request attempts to emit a `managed_transport_shadow` line group with the same validated operation, transport, timeout, and elapsed-time fields.
The diagnostic does not read the successful response body or change the returned response.
Identifier generation, serialization, or standard-error output failure can omit the line group without blocking the request or changing its response.

For `tools/list`, NemoClaw retains up to 64 successful elapsed-time samples per target host and port in the OpenClaw process.
Different MCP URL paths on the same target host and port share this sample set because the diagnostic does not retain URL paths.
After five samples, the event reports `shadow_p95_ms` and `shadow_recommended_timeout_ms`.
The proposed value is 1.5 times the observed p95 and is rounded up to 100 ms.
It cannot be less than the active catalog-list budget or 1,500 ms, and it cannot exceed 10,000 ms.
If the active budget already exceeds 10,000 ms, NemoClaw does not emit a recommendation.
An abort observed near the effective `tools/list` budget proposes twice that budget under the same constraints.
An explicit HTTP 503 does not produce a timeout recommendation because a longer timeout does not repair a returned upstream failure.

The shadow recommendation is evidence only.
It does not change a timeout, retry a request, or persist samples across an OpenClaw process restart.
After you collect the evidence, remove the setting and rebuild to restore failure-only logging:

```bash
unset NEMOCLAW_MCP_SHADOW_DIAGNOSTICS
$$nemoclaw <sandbox> rebuild
```

Verify that later successful MCP requests emit no `managed_transport_shadow` line groups.

To find the matching sandbox audit record, read the sandbox log for the same endpoint around the failure time.
The `logs` command reads both OpenClaw gateway output and OpenShell audit events, so it needs no separate audit flag.
Expand All @@ -154,7 +213,8 @@ $$nemoclaw <sandbox> logs --tail 200

Match on `target` and the timestamp.
If NemoClaw cannot enable OpenShell audit logs, it prints a warning and the policy denial events can be missing from the output.
`diagnostic_id` is a local identifier created for one line group.
`diagnostic_id` is a local identifier created for one line group when identifier generation succeeds.
NemoClaw attempts to create it before the request starts, but does not add it to the request.
It is not a distributed trace identifier and does not appear in OpenShell audit events.
Correlate with OpenShell by endpoint and time until OpenShell records a shared identifier.

Expand Down
30 changes: 24 additions & 6 deletions docs/security/openclaw-2026.7.1-dependency-review.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

Review date: 2026-07-21

Last updated: 2026-08-03
Last updated: 2026-08-06

## Decision

Expand Down Expand Up @@ -448,8 +448,12 @@ An unrecognized compiled shape fails the image build instead of silently skippin

Reviewed behavior:

- Failure-only.
A 2xx response returns untouched and emits nothing, so normal traffic produces no per-request logging.
- Failure-only by default.
A 2xx response returns untouched and emits nothing unless the OpenClaw gateway process has `NEMOCLAW_MCP_SHADOW_DIAGNOSTICS=1`.
- The opt-in shadow mode attempts to emit successful-request `managed_transport_shadow` timing events without reading their response bodies.
It does not change a timeout, retry a request, alter a response, or persist samples across an OpenClaw process restart.
Identifier generation, serialization, or standard-error output failure can omit an event without blocking the request or changing its response.
Sandbox creation forwards only the literal value `1`, and only for OpenClaw.
- The wrapper never retries, never alters the request, never changes proxy selection, and never weakens TLS verification.
It rethrows a transport error unchanged.
- `route=proxy_configured` means that `HTTPS_PROXY`, `https_proxy`, `HTTP_PROXY`, or `http_proxy` was configured.
Expand Down Expand Up @@ -478,8 +482,22 @@ Reviewed behavior:
That diagnostic has no response headers or `http_status` because `fetch` did not return a response.
- A returned non-2xx response sets `transport_phase=response_headers`.
It carries `http_status` and any allowlisted response headers that are present.
- The fetch boundary does not expose the JSON-RPC operation, so the diagnostic records the endpoint without an `operation` field.
- Each emitted diagnostic receives a local 32-character hexadecimal `diagnostic_id`.
- The wrapper parses only a string request body of at most 16,384 characters to report a validated JSON-RPC method.
It never reports JSON-RPC parameters, tool names, tool arguments, or successful response bodies.
An unsupported request shape reports `rpc/unknown`, and known GET and DELETE transport actions report `transport/listen` and `transport/close`.
- Each event reports the configured server name as `mcp_server` when validation and redaction retain it.
It also reports the transport generation, the request sequence, and the resolved connection, request, catalog-list, and effective timeout budgets.
- Shadow recommendations apply only to `tools/list`.
The wrapper retains up to 64 successful elapsed-time samples per target host and port and reports p95 after five samples.
Different MCP URL paths on the same target host and port share this sample set because the wrapper does not retain URL paths.
It proposes p95 times 1.5, rounded up to 100 ms, with a 1,500 ms floor and a 10,000 ms ceiling.
A proposal cannot be less than the active catalog-list budget.
The wrapper emits no recommendation when that active budget already exceeds 10,000 ms.
A near-budget abort proposes twice the effective budget under the same constraints.
An explicit HTTP 503 produces no timeout recommendation.
- The wrapper attempts to create a local 32-character hexadecimal `diagnostic_id` for each request before `fetch` starts.
If identifier generation fails, the wrapper omits the field and continues the request.
The wrapper does not add that identifier to the request.
- The wrapper is inert unless `OPENSHELL_SANDBOX=1`, so it does not change host-side behavior.

`diagnostic_id` is not a distributed trace identifier and does not correlate with an OpenShell audit event.
Expand All @@ -494,7 +512,7 @@ The two patches compose independently.
The injected helper in `scripts/patch-openclaw-managed-transport-diagnostics.mts` is the shipped runtime source of truth.
`test/openclaw-managed-transport-diagnostics-patch.test.ts` executes that exact helper.
It pins the compiled preimage, patch idempotence, fail-closed rejection of an unrecognized shape, and the untouched SSE boundary.
It also covers failure-only emission, no-retry and unchanged-response contracts, asynchronous body sampling, byte and time bounds, redaction, the header allowlist, local diagnostic identifiers, session-presence reporting, transport-phase classification, route evidence, and sandbox gating.
It also covers default failure-only emission, opt-in successful timing events, bounded shadow recommendations, explicit 503 exclusion, no-retry and unchanged-response contracts, validated operation reporting, asynchronous body sampling, byte and time bounds, redaction, the header allowlist, local diagnostic identifiers, session-presence reporting, transport-phase classification, route evidence, and sandbox gating.
A reusable source schema is deferred until a production consumer requires one.

Removal criterion: drop this patch when the reviewed OpenClaw release emits redacted diagnostics classified by transport phase for remote MCP fetch failures.
Expand Down
Loading
Loading