Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
3a49dc0
test(mcp): reproduce Hermes credential revision loss
rsliter Aug 25, 2026
ff64f5c
fix(mcp): preserve Hermes credential revisions
cjagwani Aug 25, 2026
dbdfb29
fix(mcp): align Hermes credential revision contract
prekshivyas Aug 25, 2026
3fa0229
test(mcp): execute Hermes config assertions
apurvvkumaria Aug 25, 2026
8a0127e
merge: refresh from main
apurvvkumaria Aug 25, 2026
0a17735
fix(mcp): preserve revision during teardown rollback
ericksoa Aug 25, 2026
bd3ce57
fix(hermes): avoid duplicate applied anchor writes
prekshivyas Aug 25, 2026
bcf9ebd
fix(mcp): retain scrub rollback revision state
ericksoa Aug 25, 2026
2547e0f
merge: resolve conflicts with main
github-actions[bot] Aug 25, 2026
1692806
fix(mcp): preserve invoked recovery command
apurvvkumaria Aug 25, 2026
98374ee
fix(mcp): reconcile Hermes revision metadata after main merge
prekshivyas Aug 25, 2026
9a4668e
merge: synchronize current main
apurvvkumaria Aug 25, 2026
6c6f0e1
test(mcp): cover Hermes lifecycle rollback revisions
prekshivyas Aug 25, 2026
7d735f9
merge: preserve concurrent PR repairs
apurvvkumaria Aug 25, 2026
6861810
merge: preserve concurrent lifecycle coverage
apurvvkumaria Aug 25, 2026
f445921
merge: synchronize current main
apurvvkumaria Aug 25, 2026
e8c1200
fix(mcp): remove Hermes helper metadata
apurvvkumaria Aug 25, 2026
d4ce801
refactor(mcp): use Hermes branding owner directly
prekshivyas Aug 25, 2026
057e5e8
merge: synchronize current main
apurvvkumaria Aug 25, 2026
f78f84a
refactor(snapshot): inject CLI name into Hermes hint
ericksoa Aug 25, 2026
2f6fc2b
fix(mcp): keep recovery branding within budget
apurvvkumaria Aug 25, 2026
9904cda
merge: preserve concurrent architecture repair
apurvvkumaria Aug 25, 2026
7b45f8a
merge: synchronize current main
apurvvkumaria Aug 25, 2026
78a6282
chore(architecture): lower branding fan-in budget
apurvvkumaria Aug 25, 2026
35ddb05
refactor(mcp): remove recovery command wrapper
apurvvkumaria Aug 25, 2026
594a508
refactor(mcp): inline Hermes recovery branding
prekshivyas Aug 25, 2026
6505f05
merge: preserve contributor architecture repair
apurvvkumaria Aug 25, 2026
302435a
chore(architecture): retain branding fan-in budget
apurvvkumaria Aug 25, 2026
42bc649
docs(mcp): describe Hermes credential revisions
rsliter Aug 25, 2026
1219dc2
fix(mcp): wait for stable credential revision
prekshivyas Aug 25, 2026
4b59c3d
test(mcp): expect stable credential revision
rsliter Aug 25, 2026
e5263dd
Merge branch 'main' into codex/fix-hermes-mcp-runtime-contract-10155
rsliter Aug 26, 2026
4d7ac91
fix(mcp): await exact credential revision
prekshivyas Aug 26, 2026
0cff15f
fix(mcp): keep credential revisions opaque
rsliter Aug 26, 2026
da46392
ci(e2e): reuse exact PR image cohort
ericksoa Aug 26, 2026
f152596
merge: synchronize exact E2E evidence
rsliter Aug 26, 2026
c87835b
ci(e2e): remove temporary exact cohort reuse
rsliter Aug 26, 2026
557b3c6
fix(mcp): fence opaque credential revisions
prekshivyas Aug 26, 2026
f7cf37c
test(mcp): mock credential revision fence
prekshivyas Aug 26, 2026
1613083
merge: synchronize latest main
prekshivyas Aug 26, 2026
286818d
fix(mcp): keep teardown revisions opaque
prekshivyas Aug 26, 2026
cbf312d
docs(mcp): document teardown revision recovery
rsliter Aug 26, 2026
90cc104
Merge remote-tracking branch 'origin/main' into codex/10155-hermes-mc…
prekshivyas Aug 26, 2026
d7a0a6a
test(repository): remove duplicate advisor watch trigger
prekshivyas Aug 26, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 26 additions & 31 deletions agents/hermes/mcp-config-transaction.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,6 @@
REVISIONED_ENV_PLACEHOLDER_RE = re.compile(
r"^Bearer openshell:resolve:env:(v[0-9]{1,20})_([A-Za-z_][A-Za-z0-9_]{0,127})$"
)
OPENSHELL_CREDENTIAL_REVISION_RE = re.compile(r"^v[0-9]{1,20}$")
OPENSHELL_REVISIONED_CREDENTIAL_NAME_RE = re.compile(r"^v[0-9]+_[A-Za-z0-9_]+$")
BOUNDARY_MANIFEST_NAME = "openshell-child-visible-credentials.v0.0.106.json"
ANSI_ESCAPE_RE = re.compile(
Expand Down Expand Up @@ -365,8 +364,6 @@ def _validate_payload(action: str, payload: dict[str, object]) -> None:
raise ValueError("Unsupported MCP config action")
allowed = {"server", "url", "headers"}
allowed.add("replace_existing" if action == "add" else "force")
if action == "add":
allowed.update({"credential_name", "credential_revision"})
unexpected = sorted(set(payload) - allowed)
if unexpected:
raise ValueError(
Expand Down Expand Up @@ -459,38 +456,35 @@ def _validate_payload(action: str, payload: dict[str, object]) -> None:
if not isinstance(headers, dict) or set(headers) != {"Authorization"}:
raise ValueError("MCP mutation payload must contain one Authorization header")
authorization = headers.get("Authorization")
declared_credential_name = payload.get("credential_name")
credential_revision = payload.get("credential_revision")
if credential_revision is not None and (
not isinstance(credential_revision, str)
or OPENSHELL_CREDENTIAL_REVISION_RE.fullmatch(credential_revision) is None
):
raise ValueError("Hermes MCP credential revision is invalid")
authorization_match = None
credential_name = None
if isinstance(authorization, str) and credential_revision is not None:
authorization_match = REVISIONED_ENV_PLACEHOLDER_RE.fullmatch(authorization)
if (
authorization_match is not None
and authorization_match.group(1) == credential_revision
and isinstance(declared_credential_name, str)
and authorization_match.group(2) == declared_credential_name
):
credential_name = authorization_match.group(2)
else:
authorization_match = None
elif isinstance(authorization, str) and declared_credential_name is None:
authorization_match = ENV_PLACEHOLDER_RE.fullmatch(authorization)
if authorization_match is not None:
credential_name = authorization_match.group(1)
revisioned_authorization_match = (
REVISIONED_ENV_PLACEHOLDER_RE.fullmatch(authorization)
if isinstance(authorization, str)
else None
)
canonical_authorization_match = (
ENV_PLACEHOLDER_RE.fullmatch(authorization)
if isinstance(authorization, str)
else None
)
authorization_match = (
revisioned_authorization_match or canonical_authorization_match
)
if authorization_match is None:
raise ValueError(
"Hermes MCP Authorization must contain an OpenShell environment placeholder"
)
if action == "add" and (
not isinstance(credential_name, str)
or _credential_name_is_reserved(credential_name)
):
credential_name = (
revisioned_authorization_match.group(2)
if revisioned_authorization_match is not None
else canonical_authorization_match.group(1)
)
if action == "add" and revisioned_authorization_match is not None:
expected_child_value = authorization.removeprefix("Bearer ")
if os.environ.get(credential_name) != expected_child_value:
raise ValueError(
"Hermes MCP Authorization revision does not match the OpenShell child environment"
)
if action == "add" and _credential_name_is_reserved(credential_name):
raise ValueError(
"Hermes MCP Authorization uses a reserved credential environment name"
)
Expand All @@ -515,6 +509,7 @@ def _managed_candidate(payload: dict[str, object]) -> dict[str, object]:
def _managed_candidate_matches(
actual: object, expected: dict[str, object], allow_revisioned: bool
) -> bool:
"""Compare managed config with bounded revision equivalence when requested."""
if actual == expected:
return True
if not allow_revisioned or not isinstance(actual, dict):
Expand Down
22 changes: 22 additions & 0 deletions agents/hermes/runtime-config-guard.py
Original file line number Diff line number Diff line change
Expand Up @@ -1455,6 +1455,28 @@ def refresh_hashes(
source_hash_text, config_path, env_path
)
current_mcp, _ = _current_mcp_servers_digest(config_path)
# The transaction helper and the managed supervisor can both observe the
# same pending reload. The helper may commit it first while the supervisor
# still retains its earlier pending observation. Replacing an already-current
# anchor with byte-identical content needlessly changes its inode and can
# make a concurrent host reconciliation reject an otherwise coherent
# snapshot. Keep the repeated apply fail-closed, but make it validation-only:
# authenticate config, env, and every relevant anchor as one stable current
# snapshot before returning without an atomic replacement.
if mcp_transition == "apply" and secrets.compare_digest(
state.intended, state.applied
):
integrity = inspect_mcp_integrity_snapshot(
hermes_dir,
state_path,
compat_hash if mode == "both" else None,
)
if integrity.state != "current":
raise UnsafePathError(
"Hermes MCP applied-state commit did not observe current state"
)
assert_mcp_integrity_snapshot_current(integrity)
return
if mcp_transition == "preserve":
if not secrets.compare_digest(current_mcp, state.intended):
raise UnsafePathError(
Expand Down
2 changes: 1 addition & 1 deletion ci/source-architecture-budget.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
"src/lib/adapters/openshell/runtime.ts": 55,
"src/lib/adapters/openshell/timeouts.ts": 39,
"src/lib/agent/defs.ts": 33,
"src/lib/cli/branding.ts": 87,
"src/lib/cli/branding.ts": 86,
"src/lib/cli/nemoclaw-oclif-command.ts": 107,
"src/lib/cli/terminal-style.ts": 43,
"src/lib/core/json-types.ts": 37,
Expand Down
12 changes: 10 additions & 2 deletions docs/deployment/set-up-mcp-bridge.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,14 @@ The decision record is tracked in [NVIDIA/NemoClaw#566](https://github.com/NVIDI
## Authenticated MCP Security Boundary

Authenticated MCP is the intended configuration.
The agent stores only the `openshell:resolve:env:KEY` placeholder.
The agent stores only an OpenShell resolver placeholder for the recorded credential key.
OpenShell keeps the raw credential in its provider store and combines credential replacement with generated MCP policy at egress.

<AgentOnly variant="hermes">
Hermes includes the credential revision from the readiness check in this placeholder.
Registration, inspection, rollback, and lifecycle reconciliation preserve that revision while OpenShell still reports it.
</AgentOnly>

For the normal MCP client path, OpenShell evaluates the effective policy for the destination host and port, adapter binary, literal endpoint path, and MCP method before replacing placeholders in allowed HTTP request headers.
The generated policy grants only the configured destination, path, adapter binaries, pinned addresses, explicit MCP method profile, and a 131,072-byte maximum request body.

Expand Down Expand Up @@ -134,9 +139,12 @@ mcp_servers:
resources: true
prompts: true
headers:
Authorization: Bearer openshell:resolve:env:GITHUB_MCP_TOKEN
Authorization: Bearer openshell:resolve:env:v12_GITHUB_MCP_TOKEN
```

The `v12_` prefix is an illustrative OpenShell credential revision.
Hermes records the revision that the readiness check proves for the live provider.

### Prepare Hermes for Mutation

When Hermes shields are up, run `nemohermes <sandbox> shields down --timeout 15m --reason "MCP maintenance"` before `mcp add`, `mcp restart`, `mcp remove`, or destroy, then restore shields if the sandbox still exists.
Expand Down
7 changes: 5 additions & 2 deletions docs/manage-sandboxes/add-mcp-server.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ The early capability check identifies managed image version only; NemoClaw still
</AgentOnly>
<AgentOnly variant="hermes">
Hermes performs its managed runtime probe before an active add or restart changes a live provider or policy.
Hermes stores the credential revision from the readiness check in the OpenShell resolver placeholder.
Registration, inspection, rollback, and lifecycle reconciliation preserve that revision while it remains observable.
</AgentOnly>
<AgentOnly variant="openclaw">
OpenClaw verifies the pinned `mcporter` during adapter registration and rolls back an incomplete add.
Expand All @@ -43,7 +45,8 @@ The assignment is illustrative.
Load real values from an approved secret manager or masked prompt so the credential is not recorded in shell history.

`--env KEY` reads the value from the host process environment and stores it in OpenShell's provider store.
NemoClaw persists only the variable name, writes `openshell:resolve:env:KEY` into sandbox-side MCP config, and relies on OpenShell to resolve the placeholder at egress.
NemoClaw persists only the variable name and writes an OpenShell resolver placeholder for that key into sandbox-side MCP configuration.
OpenShell resolves the placeholder at egress.

After `mcp add` commits, NemoClaw performs a fresh status inspection and runs the gated credential-resolution probe once unless you pass `--no-probe`.
If readiness is inconclusive, the command reports a probe skip without failing the committed add.
Expand Down Expand Up @@ -103,7 +106,7 @@ The command records the resulting exact trust intent, so later lifecycle command

NemoClaw records the normalized trust intent and every validated address as exact policy pins.
The raw bearer value passes transiently to the OpenShell provider and remains absent from NemoClaw state, sandbox configuration, command arguments, and logs.
The sandbox configuration contains only the `openshell:resolve:env:LOCAL_MCP_TOKEN` credential placeholder.
The sandbox configuration contains only an OpenShell resolver placeholder for `LOCAL_MCP_TOKEN`, not the raw bearer value.
You can unset the host variable after `mcp add` returns because OpenShell retains the credential.
An exported replacement updates it during restart, and `mcp remove` or sandbox destroy deletes the registry-owned provider.

Expand Down
11 changes: 10 additions & 1 deletion docs/manage-sandboxes/manage-mcp-servers.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ Bounded partial results set `truncated` to `true` and include a redacted `detail

## Verify Credential Resolution

Provider presence and metadata cannot prove that OpenShell rewrites `openshell:resolve:env:KEY` when a request leaves the sandbox.
Provider presence and metadata cannot prove that OpenShell rewrites the recorded resolver placeholder when a request leaves the sandbox.
`mcp status <server>` requests a differential wire-level credential-resolution probe by default.

Before sending probe traffic, NemoClaw verifies exact generated policy, expected provider attachment, provider ID, `nemoclaw-mcp-v1` type, valid resource version, and exactly one matching credential key.
Expand Down Expand Up @@ -189,6 +189,11 @@ It never detaches the provider from other sandboxes.

`rebuild` preserves providers that match the recorded ID and credential-key metadata and the active `nemoclaw-mcp-v1` type.
It removes adapter entries and exact owned policies, then detaches providers before replacing the sandbox.
Before changing each managed adapter, a fresh sandbox process must expose a revision-scoped OpenShell credential placeholder for that adapter.
If an observation is absent, unscoped, or unavailable, NemoClaw leaves the affected adapter and every provider unchanged.
It attempts to restore adapter entries prepared earlier in the operation and reports rollback failures.
It does not substitute the provider resource version for the credential revision.
Follow the reported [credential-revision recovery](../../reference/troubleshoot-mcp-servers#rebuild-or-destroy-cannot-prove-a-credential-revision), then retry rebuild.
Restoration applies the credential-free policy, reattaches each provider, applies the endpoint-bound policy, waits for credential readiness, and restores adapters.
For a trusted-private entry, the restored policy uses the recorded exact address pins and does not widen them from current DNS answers.
Public entries continue to resolve and validate their endpoint addresses during restoration.
Expand All @@ -209,6 +214,10 @@ A later `mcp restart` can retry an incomplete post-rebuild restore.
## Destroy a Sandbox with MCP State

Destroy removes adapter entries and exact owned policies, then detaches providers that match recorded metadata before asking OpenShell to delete the sandbox.
Before changing each managed adapter, a fresh sandbox process must expose a revision-scoped OpenShell credential placeholder for that adapter.
If an observation is absent, unscoped, or unavailable, NemoClaw leaves the affected adapter and every provider unchanged.
It attempts to restore adapter entries prepared earlier in the operation and reports rollback failures.
Follow the reported [credential-revision recovery](../../reference/troubleshoot-mcp-servers#rebuild-or-destroy-cannot-prove-a-credential-revision), then retry destroy.
If deletion is refused, NemoClaw attempts to restore previous MCP state, reports rollback failures, and preserves recovery state.
Provider deletion and registry cleanup happen only after OpenShell confirms the sandbox is gone.

Expand Down
5 changes: 3 additions & 2 deletions docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3183,7 +3183,7 @@ The declaration must equal the normalized host from `--url`.
For managed MCP, use a DNS hostname for an IPv6 unique local address because NemoClaw has not qualified direct IPv6-literal MCP URLs.
You can supply exact hosts through `NEMOCLAW_TRUSTED_PRIVATE_HOSTS` instead, and NemoClaw combines the variable with command options.
NemoClaw records the resulting exact trust intent and address pins, so restart, rebuild, and restore do not depend on the ambient environment.
NemoClaw registers that credential in an OpenShell provider, installs a generated OpenShell `protocol: mcp` policy for the target endpoint, attaches the provider to the running sandbox, and writes only an `openshell:resolve:env:KEY` placeholder into the agent config.
NemoClaw registers that credential in an OpenShell provider, installs a generated OpenShell `protocol: mcp` policy for the target endpoint, attaches the provider to the running sandbox, and writes only an OpenShell resolver placeholder for the recorded key into the agent configuration.
Inline `--env KEY=VALUE` is rejected because it would expose the value in NemoClaw process arguments.
Load the variable from a secret manager or masked prompt, export it without recording the value in shell history, and pass only `--env KEY`.
All endpoints must use HTTPS.
Expand All @@ -3203,6 +3203,7 @@ For full setup details, see [Add an MCP Server](../manage-sandboxes/mcp-servers/
<AgentOnly variant="hermes">

Hermes MCP add, restart, and remove mutate managed config and are refused while shields are up.
Hermes includes the credential revision from the readiness check in the resolver placeholder and preserves it through inspection, rollback, and lifecycle reconciliation while OpenShell still reports it.
Run `$$nemoclaw <name> shields down --timeout 15m --reason "MCP maintenance"` before the mutation, then run `$$nemoclaw <name> shields up` after it; list and status remain read-only.
Allow at least 15 minutes per configured server for an all-server restart or destroy; rebuild opens its own crash-recoverable maintenance window.
Keep shields down until the command returns; a concurrent relock refuses the config commit and can leave an earlier policy/provider stage for the next retry to converge.
Expand Down Expand Up @@ -3242,7 +3243,7 @@ Text output reports `private address pins: match`, `drift`, or `unresolved`.
JSON output reports the same value in `trustedPrivateTarget.state` and includes the recorded pins.
When a single server is named, status requests a differential wire-level credential-resolution probe.
It sends no probe traffic unless the exact generated policy matches the effective gateway policy, the expected provider attachment is confirmed, and the live provider has the recorded ID, `nemoclaw-mcp-v1` type, a valid resource version, and exactly one credential key matching the recorded key; a readiness failure reports `unknown` with a `probe skipped` detail.
When ready, the same MCP `initialize` is sent from inside the sandbox once with the `openshell:resolve:env:KEY` placeholder header and once with a deliberately-unresolvable control bearer.
When ready, the same MCP `initialize` is sent from inside the sandbox once with the recorded resolver placeholder header and once with a deliberately-unresolvable control bearer.
Classification uses the two HTTP status codes plus curl exit codes for transport, timeout, and policy-denial outcomes; response bodies are never captured or printed.
A `verified` verdict requires the placeholder request to be accepted (HTTP 2xx) while the control is rejected — the only outcome that proves a valid credential was on the wire.
Identical HTTP 400, 401, or 403 rejections raise a warning that names the hypotheses — the placeholder forwarded verbatim, an expired or revoked credential that resolved correctly, or (for HTTP 400) endpoint request validation — and tells you to verify the stored credential first.
Expand Down
27 changes: 26 additions & 1 deletion docs/reference/troubleshoot-mcp-servers.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Use the reported status or lifecycle error to choose the matching remediation.
If `mcp status <server>` reports identical placeholder and control rejections, first confirm the stored credential is valid.
Rotate it with `mcp restart` when in doubt.

For identical HTTP 401 or 403 responses, a confirmed-valid credential means the OpenShell gateway on this host is not rewriting `openshell:resolve:env:KEY` on egress.
For identical HTTP 401 or 403 responses, a confirmed-valid credential means the OpenShell gateway on this host is not rewriting the recorded resolver placeholder on egress.
Every agent request receives the same authentication failure even when provider, attachment, readiness, and adapter checks report healthy.

This is a host-side OpenShell defect rather than a NemoClaw registration problem.
Expand All @@ -40,6 +40,31 @@ If restart reports a missing provider and the original credential is not registe
If restart or rebuild reports that the provider uses the profile-less legacy `generic` type, remove that server and add it again with the recorded credential variable exported.
OpenShell cannot bind that provider to an MCP endpoint, and NemoClaw accepts it only for exact cleanup.

## Rebuild or Destroy Cannot Prove a Credential Revision

If rebuild or destroy reports that it could not prove a revision-scoped credential, a fresh sandbox process did not expose the exact OpenShell credential identity required for adapter cleanup.
NemoClaw leaves the affected adapter and every provider unchanged.
It attempts to restore adapter entries prepared earlier in the operation and reports rollback failures.

Inspect the managed server:

```bash
$$nemoclaw <sandbox> mcp status <server> --json
```

If status cannot reach the sandbox, repair the reported gateway condition first.
Export the recorded environment variable, then restart the server to publish and verify a fresh credential revision:

```bash
export <CREDENTIAL_ENV>='replacement-value'
$$nemoclaw <sandbox> mcp restart <server>
unset <CREDENTIAL_ENV>
```

Run status again.
Retry rebuild or destroy only after `provider.credentialReady` and `adapter.registered` are both `true`.
If restart still cannot prove the revision, repair the reported OpenShell provider, policy, or adapter condition before retrying teardown.

## Add Transaction Is Incomplete

If status reports an incomplete add transaction, rerun the original `mcp add` command with the same URL and environment-variable name.
Expand Down
2 changes: 1 addition & 1 deletion src/commands/credentials/list.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ export default class CredentialsListCommand extends NemoClawCommand {

public async run(): Promise<void> {
await this.parse(CredentialsListCommand);
const result = await runCredentialsListAction();
const result = await runCredentialsListAction(this.config.bin);
if (result.exitCode !== 0) {
this.failWithLines(result.failureLines, result.exitCode);
return;
Expand Down
Loading
Loading