fix(mcp): surface the real cause on MCP loss and recover parked servers faster - #80815
Open
Revelation-Agency wants to merge 1 commit into
Open
fix(mcp): surface the real cause on MCP loss and recover parked servers faster#80815Revelation-Agency wants to merge 1 commit into
Revelation-Agency wants to merge 1 commit into
Conversation
…rs faster When an MCP server drops, the agent silently loses its tools and the operator is not told. A client-facing agent ran 53 hours with zero tools and nothing alerted. 1. _flatten_exc() unwraps ExceptionGroup/TaskGroup sub-exceptions (bounded depth 4, first 5) so the real failure reaches the log instead of "unhandled errors in a TaskGroup (1 sub-exception)". 2. Parking is logged at ERROR naming the server and the number of tools lost; _deregister_tools(reason=...) emits ERROR on the transition to ZERO tools, suppressed on ordinary shutdown so clean exits stay quiet. 3. _PARKED_RETRY_INTERVAL 300s -> 30s so recovery is not itself an outage.
leewenjie
pushed a commit
to leewenjie/hermes-agent
that referenced
this pull request
Aug 7, 2026
…esearch#25437) Followup to PR NousResearch#24182 — caught when scanning OpenClaw for recent codex fixes we hadn't considered. OpenClaw learned the hard way (NousResearch#80815) that migrating plugins which codex itself reports as unavailable produces config that fails at activation time. Our /codex-runtime codex_app_server enable path queries codex's plugin/list and migrates everything where installed=true. We were trusting codex's installation state and ignoring its availability field. So a plugin that's installed=true but availability=UNAVAILABLE (broken local install) or REQUIRES_AUTH (OAuth expired or never completed) would get an [plugins."<n>@openai-curated"] entry in ~/.codex/config.toml — and the user's first codex turn after enabling the runtime would fail because codex refuses to activate it. Fix: filter on availability in _query_codex_plugins(). Only emit plugins where availability is empty (older codex versions without the field — preserve backward compat) or explicitly AVAILABLE. Tests: test_plugin_discovery_skips_unavailable_plugins — verifies 4 cases: - good-plugin (installed=True, availability=AVAILABLE) → migrated - broken-plugin (installed=True, availability=UNAVAILABLE) → skipped - auth-pending (installed=True, availability=REQUIRES_AUTH) → skipped - legacy-plugin (installed=True, no availability field) → migrated (older codex versions; preserve backward compat) Docs: Added bullet to 'What's NOT migrated' list in the docs page calling out the availability filter and why. Other OpenClaw codex PRs I reviewed but did NOT apply (with reasoning): - NousResearch#81591 (load Codex for selectable models): we resolve runtime per-call already, no startup-time gating to fix - NousResearch#81510 (cron compatibility): we documented cron as untested; their fix is for OpenClaw-specific cron orchestration shape - NousResearch#81223 (rotate incompatible context-engine threads): we don't have a Lossless context engine equivalent - NousResearch#80688 (constrain sandbox): we don't have an outer-sandbox concept - NousResearch#80616 (release on turn_aborted): we already handle status= interrupted in turn/completed correctly - NousResearch#80278 (expose activeModel in plugin SDK): not our surface - NousResearch#80792 (default destructive_actions on): we don't expose that knob 56 codex-runtime migration tests still green (+1 new).
Contributor
|
This was generated by AI during triage. Summary: Problems:
Solution: Checked against |
igangz
pushed a commit
to igangz/hermes-agent
that referenced
this pull request
Aug 10, 2026
…esearch#25437) Followup to PR NousResearch#24182 — caught when scanning OpenClaw for recent codex fixes we hadn't considered. OpenClaw learned the hard way (NousResearch#80815) that migrating plugins which codex itself reports as unavailable produces config that fails at activation time. Our /codex-runtime codex_app_server enable path queries codex's plugin/list and migrates everything where installed=true. We were trusting codex's installation state and ignoring its availability field. So a plugin that's installed=true but availability=UNAVAILABLE (broken local install) or REQUIRES_AUTH (OAuth expired or never completed) would get an [plugins."<n>@openai-curated"] entry in ~/.codex/config.toml — and the user's first codex turn after enabling the runtime would fail because codex refuses to activate it. Fix: filter on availability in _query_codex_plugins(). Only emit plugins where availability is empty (older codex versions without the field — preserve backward compat) or explicitly AVAILABLE. Tests: test_plugin_discovery_skips_unavailable_plugins — verifies 4 cases: - good-plugin (installed=True, availability=AVAILABLE) → migrated - broken-plugin (installed=True, availability=UNAVAILABLE) → skipped - auth-pending (installed=True, availability=REQUIRES_AUTH) → skipped - legacy-plugin (installed=True, no availability field) → migrated (older codex versions; preserve backward compat) Docs: Added bullet to 'What's NOT migrated' list in the docs page calling out the availability filter and why. Other OpenClaw codex PRs I reviewed but did NOT apply (with reasoning): - NousResearch#81591 (load Codex for selectable models): we resolve runtime per-call already, no startup-time gating to fix - NousResearch#81510 (cron compatibility): we documented cron as untested; their fix is for OpenClaw-specific cron orchestration shape - NousResearch#81223 (rotate incompatible context-engine threads): we don't have a Lossless context engine equivalent - NousResearch#80688 (constrain sandbox): we don't have an outer-sandbox concept - NousResearch#80616 (release on turn_aborted): we already handle status= interrupted in turn/completed correctly - NousResearch#80278 (expose activeModel in plugin SDK): not our surface - NousResearch#80792 (default destructive_actions on): we don't expose that knob 56 codex-runtime migration tests still green (+1 new).
randlee
pushed a commit
to randlee/hermes-agent
that referenced
this pull request
Aug 11, 2026
…esearch#25437) Followup to PR NousResearch#24182 — caught when scanning OpenClaw for recent codex fixes we hadn't considered. OpenClaw learned the hard way (NousResearch#80815) that migrating plugins which codex itself reports as unavailable produces config that fails at activation time. Our /codex-runtime codex_app_server enable path queries codex's plugin/list and migrates everything where installed=true. We were trusting codex's installation state and ignoring its availability field. So a plugin that's installed=true but availability=UNAVAILABLE (broken local install) or REQUIRES_AUTH (OAuth expired or never completed) would get an [plugins."<n>@openai-curated"] entry in ~/.codex/config.toml — and the user's first codex turn after enabling the runtime would fail because codex refuses to activate it. Fix: filter on availability in _query_codex_plugins(). Only emit plugins where availability is empty (older codex versions without the field — preserve backward compat) or explicitly AVAILABLE. Tests: test_plugin_discovery_skips_unavailable_plugins — verifies 4 cases: - good-plugin (installed=True, availability=AVAILABLE) → migrated - broken-plugin (installed=True, availability=UNAVAILABLE) → skipped - auth-pending (installed=True, availability=REQUIRES_AUTH) → skipped - legacy-plugin (installed=True, no availability field) → migrated (older codex versions; preserve backward compat) Docs: Added bullet to 'What's NOT migrated' list in the docs page calling out the availability filter and why. Other OpenClaw codex PRs I reviewed but did NOT apply (with reasoning): - NousResearch#81591 (load Codex for selectable models): we resolve runtime per-call already, no startup-time gating to fix - NousResearch#81510 (cron compatibility): we documented cron as untested; their fix is for OpenClaw-specific cron orchestration shape - NousResearch#81223 (rotate incompatible context-engine threads): we don't have a Lossless context engine equivalent - NousResearch#80688 (constrain sandbox): we don't have an outer-sandbox concept - NousResearch#80616 (release on turn_aborted): we already handle status= interrupted in turn/completed correctly - NousResearch#80278 (expose activeModel in plugin SDK): not our surface - NousResearch#80792 (default destructive_actions on): we don't expose that knob 56 codex-runtime migration tests still green (+1 new).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi — a small reliability patch to
tools/mcp_tool.py, offered from production experience.What happened to us. A client-facing agent ran ~53 hours with zero MCP tools and nothing alerted. The MCP server had dropped, its tools were deregistered, and the agent carried on answering from memory rather than saying it had lost the ability to act. When we went looking, the log said
unhandled errors in a TaskGroup (1 sub-exception), which didn't tell us what actually broke.What this changes (all in
tools/mcp_tool.py):Surface the real cause.
_flatten_exc()unwraps ExceptionGroup / TaskGroup sub-exceptions — bounded at depth 4 and the first 5 sub-exceptions — so the underlying failure reaches the log instead of the wrapper message.Make total tool loss loud. Parking now logs at ERROR and names the server plus how many tools were lost, and
_deregister_tools(reason=...)emits an ERROR on the transition to zero tools. It stays quiet on ordinary shutdown, so clean exits don't get noisy. An agent that can no longer act felt worth an ERROR rather than a WARNING.Recover faster.
_PARKED_RETRY_INTERVAL300s → 30s. At five minutes a parked server turns a blip into an outage; 30s took our recovery from ~9 minutes down to ~60–90s.Beyond log levels/content and that retry cadence there's no behaviour change, and no public API change.
Happy to adjust any of this to your conventions — split it into separate commits, reword the log lines or dial the levels back, make the retry interval configurable instead of a constant, or drop any part you'd rather not carry. Just say which and I'll rework it.