test(e2e): restart Hermes before MCP rediscovery

[jyaunches]

Summary

After the DNS rebinding proof removes its temporary MCP route, explicitly restart Hermes before invoking the deferred MCP tool. Hermes builds its MCP registry at gateway startup, so repeated tool_search calls in one model turn cannot discover a missing tool; the live shard now proves fresh authenticated rediscovery after restart and then executes one real deferred tool call.

Changes

• Move the existing Hermes MCP restart ahead of the post-removal real adapter tool call.
• Capture the rediscovery request offset after temporary route removal and immediately before explicit restart, so route-removal side effects cannot satisfy the restart proof.
• Require one tool_search → tool_describe → tool_call sequence and fail immediately on a missing or malformed search result.
• Verify fresh authenticated initialize → notifications/initialized → tools/list evidence with successful responses, complete negotiated metadata, and one session.
• Protect the lifecycle order and missing-metadata branches with focused support and integration tests.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: This changes live E2E fixtures, orchestration, and assertions only; no supported product behavior changes.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: The DNS rebinding denial and route-removal assertions remain before restart. Rediscovery evidence remains fail-closed and requires authenticated, ordered, successful requests in one negotiated session plus exactly one real tool call.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: no-docs-needed
• Evidence: Independent review covered the complete six-file PR and the exact two-file follow-up at commit 06406a3a2f960e37f6de24d591f7ce3902d28094. The follow-up captures the Hermes rediscovery offset after temporary MCP route removal and immediately before explicit restart; its source-contract test pins route removal → offset capture → restart → real tool call → authenticated rediscovery. This changes only live E2E evidence collection and support tests; no user-visible API, CLI, configuration, UI, workflow, default, error, or supported product behavior changes. The committed follow-up diff SHA-256 a8437d836fce7be9a3fc1a8f403bb35b93a91f780bddc038ec4271daf3ee9b95 exactly matches the reviewed worktree diff. Focused E2E-support and test:changed passed 14/14; the conditional scan, Biome, CLI typecheck, normal commit and pre-push hooks, and git diff --check passed. The prior head passed the trusted mcp-bridge (hermes) live E2E in 12m38s.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: E2E-support passed 25/25; the compatible-mock integration fixture passed 10/10; npm run test:changed passed 25/25. The exact offset-isolation follow-up passed its focused E2E-support and changed-test suites 14/14. Repository checks, conditional scan, CLI typecheck, clean CLI build, Biome, and git diff --check passed. The prior head passed trusted mcp-bridge (hermes) live E2E in 12m38s.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result:
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Julie Yaunches jyaunches@nvidia.com
Signed-off-by: Carlos Villela cvillela@nvidia.com

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Authenticated MCP request evidence now records negotiated metadata and response outcomes. Hermes tool calls retry deferred tool-search misses, and the DNS rebinding scenario verifies rediscovery after route removal and the real adapter tool call.

Changes

MCP rediscovery and Hermes E2E flow

Layer / File(s)	Summary
Fake MCP request ledger test/e2e/live/mcp-bridge-servers.ts	Typed request records capture response outcomes and negotiated session metadata through shared response helpers.
Authenticated rediscovery helper test/e2e/live/mcp-bridge-tool-discovery.ts, test/e2e/support/mcp-bridge-tool-discovery.test.ts	Adds ordered authenticated discovery detection, optional rediscovery targets, polling integration, and validation of ordering and negotiated session matching.
Hermes deferred tool retry test/e2e/live/mcp-bridge-tool-discovery.ts, test/e2e/live/mcp-bridge.test.ts, test/e2e/support/mcp-bridge-tool-discovery.test.ts	Retries Hermes tool calls after the specified deferred tool-search miss and tests retry, stop, and attempt-limit behavior.
DNS rebinding teardown flow test/e2e/live/mcp-bridge.test.ts	Restores DNS before MCP route removal, returns the surviving request offset, and verifies authenticated Hermes rediscovery afterward.
Sandbox ordering assertions test/e2e/support/mcp-bridge-sandbox.test.ts	Checks that rediscovery follows the Hermes shard, DNS denial assertion, and real adapter tool call.

Estimated code review effort: 3 (Moderate) | ~30 minutes

Possibly related PRs

• NVIDIA/NemoClaw#7168: Both changes modify Hermes E2E progress flow and test-harness ordering.

Suggested labels: area: e2e, integration: hermes

Suggested reviewers: cv, prekshivyas, aasthajh

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately reflects the main change: Hermes is restarted before MCP rediscovery is asserted.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/fix-hermes-mcp-tool-search

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 06406a3 in the codex/fix-hermes-mcp... branch remains at 96%, unchanged from commit 4dcb89e in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 06406a3 in the codex/fix-hermes-mcp... branch remains at 81%, unchanged from commit 4dcb89e in the main branch.

Show a code coverage summary of the most impacted files.

File	main 4dcb89e	codex/fix-hermes-mcp... 06406a3	+/-
src/lib/sandbox...rce-identity.ts	88%	88%	0%
src/lib/domain/.../connect-env.ts	89%	97%	+8%

---

_{Updated July 30, 2026 13:05 UTC}

[github-actions]

PR Review Advisor — Informational

Advisor assessment: Informational / low confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: PR review advisor failed: PR review advisor SDK execution failed: session: scope-risk-map-analysis omitted required analysis; turn: scope-risk-map-analysis: scope-risk-map-analysis omitted required analysis

Model lanes

• GPT-5.6 Terra (primary): Failed
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-inference, cloud-onboard, security-posture, mcp-bridge, mcp-bridge-dev

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/live/mcp-bridge-tool-discovery.ts`:
- Around line 17-27: Strengthen assertAuthenticatedMcpRediscovery to validate a
fresh, valid MCP session after requestOffset, requiring an ordered initialize
followed by tools/list exchange and successful responses tied to the same
negotiated session. Update or replace the delegated
assertAuthenticatedMcpDiscovery behavior so recorded requests alone cannot
satisfy the teardown check.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 795eaceb-87ce-425c-8b61-a93e715814f1

📥 Commits

Reviewing files that changed from the base of the PR and between 376beb5 and aac4db7.

📒 Files selected for processing (3)

• test/e2e/live/mcp-bridge-tool-discovery.ts
• test/e2e/live/mcp-bridge.test.ts
• test/e2e/support/mcp-bridge-sandbox.test.ts

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/support/mcp-bridge-tool-discovery.test.ts`:
- Around line 69-80: Refactor the parameterized test around failedMethod so it
no longer needs the runtime missing-request guard. Split the initialize and
tools/list failure scenarios into separate linear tests, or provide keyed
fixtures that select a guaranteed-defined request while preserving each existing
response override and assertion behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e6921bd3-973b-43f6-a3cb-8228328a26f3

📥 Commits

Reviewing files that changed from the base of the PR and between aac4db7 and 4f4c046.

📒 Files selected for processing (3)

• test/e2e/live/mcp-bridge-servers.ts
• test/e2e/live/mcp-bridge-tool-discovery.ts
• test/e2e/support/mcp-bridge-tool-discovery.test.ts

[cv]

Investigation update: Hermes MCP rediscovery

Scope: PR SHA e0cf9ee19. The failing job is mcp-bridge (hermes).

CI evidence

• Initial authenticated MCP discovery passed.
• mcp status --tools --json listed fake_echo and fake_status.
• The Shields restoration and gateway restart phase passed.
• The DNS rebinding lifecycle then removed the temporary MCP bridge.
• The next real Hermes tool call returned mock protocol error: Hermes tool_search did not return the deferred target.
• The lifecycle phase failed after 174.616 seconds.
• DNS rebinding bridge removal used 34.477 seconds.
• The failed tool call used 30.919 seconds.
• The OpenClaw and Deep Agents Code MCP bridge jobs passed in the same child run.

The failing call is therefore after successful initial discovery. It is also after a lifecycle operation that can change the Hermes MCP registry.

Source analysis

I inspected the Hermes v2026.7.20 runtime in the pinned base image. Hermes reports semver 0.19.0.

Hermes calls discover_mcp_tools() at these lifecycle boundaries:

• Gateway startup, before runner.start.
• The explicit /reload-mcp operation.

Hermes does not call discover_mcp_tools() from tool_search. The tool_search path searches the live in-memory registry. Repeating the call in one message history does not rebuild that registry.

The PR mock can issue 15 same-turn tool_search calls with two-second delays. Each call observes the same registry. The retries add latency but do not create a rediscovery event.

I also ran an exact-image probe with mcp__fake__fake_echo registered in the Hermes registry. One tool_search request returned the target. This result excludes the query, tool name, and search ranking as the cause.

The test already runs mcp restart fake, but it runs after the failing tool call. That restart is the lifecycle event that can rebuild the Hermes MCP registry. The MCP configuration transaction confirms an acknowledged reload and gateway health. It does not confirm that every remote MCP connection joined the registry.

Candidate correction

A local test-only candidate makes these changes:

• Move the existing mcp restart fake immediately after removal of the DNS rebinding bridge.
• Run one tool_search → tool_describe → tool_call sequence after that restart.
• Require fresh authenticated discovery after the restart.
• Require exactly one new tools/call request.
• Remove the 15 same-turn search retries.
• Remove the later duplicate restart and tool call.
• Add the missing negotiated-session and negotiated-protocol cases from PRA-1.

This candidate does not change NemoClaw product behavior. It aligns the live E2E test with the Hermes registry lifecycle.

Local evidence

The local candidate has these passing results:

• E2E-support tests: 25 passed.
• MCP fixture integration tests: 10 passed.
• npm run test:changed.
• npm run checks:repository.
• npm run typecheck:cli.
• npm run test-conditionals:scan -- --top 25.

These results verify the fixture sequence and repository contracts. They do not establish a passing live E2E result.

The latest local live attempt stopped during onboarding because the host could not allocate a dashboard port. The sandbox was destroyed during cleanup. No MCP assertion ran in that attempt. A prior isolated run reproduced the same missing-discovery failure family locally.

Remaining verification

• Complete one local Hermes live shard through the post-removal restart and tool call.
• Commit and push the test-only candidate.
• Repeat the documentation writer review for the new PR SHA.
• Launch one trusted E2E controller run for the pushed PR SHA.

I have not launched another trusted controller run for the local candidate.