Multiplexed gateway housekeeping runs per profile scope; parked MCP servers stop flooding the error log (#115713, #116700, salvage #115775) - #116744
Merged
Conversation
A parked server's timed self-probe wakes the run task and, when the dependency is still down, re-parks — re-emitting the same WARNING as the first park on every interval with no bound. On a long-lived gateway this flooded gateway.error.log with identical lines (10,548 for one server in a month) and drowned the genuinely-new errors. Route all park lines through _log_park: the first park (a real state transition) still warns; re-parks while the server never revived are DEBUG-only (hermes mcp list already surfaces the parked state). A session that proves healthy clears _was_parked, so the next outage warns again.
… again `_log_park` demoted every park after the first to DEBUG for as long as `_was_parked` held. That also hid a park for a DIFFERENT reason (a server parked on connection-refused that re-parks on a revoked token), which is new information an operator needs at WARNING. Remember the last park line per server and demote only an identical repeat; fold the two unit checks into one contract test (first warns, identical repeat DEBUG, new reason warns, its repeat DEBUG).
…erved profile's scope
The housekeeping thread has no turn on the stack, so nothing bound a profile
for the 60-tick chores. Under gateway.multiplex_profiles the skills-sync
pull (`tools/skills_sync_client.py::resolve_identity` ->
`resolve_nous_runtime_credentials`) and its org-sync sibling read Nous
credentials through the fail-closed reader and logged
`nous: NOUS_INFERENCE_BASE_URL unreadable - no profile secret scope on a
multiplexed call` (and the portal-URL twin) every hourly tick: 150 lines in
24h on one two-profile gateway, in bursts of six. The curator tick read the
LAUNCH profile's skills tree and curator state for every served profile.
Lift the MCP reconciler's per-served-profile iteration into
`_for_each_served_profile` and run the curator, sync pull and org sync pull
through it, so each served profile's tick reads ITS OWN home, config and
credentials (A reads A's NOUS_INFERENCE_BASE_URL, B reads B's, B never sees
A's). Single-profile gateways still run each chore once against the process
home; a standalone gateway that a hosted room flipped into multi-profile
hosting binds the launch profile's own scope, as turns do.
Probe (two temp homes, multiplex active, 60 housekeeping ticks):
base: per-chore (home, override) = [(A, None)] x3, scope warnings 3
head: [(A, a-url), (B, b-url)] per chore, scope warnings 0; single-profile
control unchanged ([(A, a-url)] per chore, 0 warnings).
Contributor
૮ >ﻌ< ა ci reviewran on d506461 — fix(gateway): run curator/skills-sync housekeeping ticks und debug infoCI timingsCI timings · View report · View jobWall time 4m52s vs 6m29s (-24.9%). 4 job(s) slower, 7 faster, 1 unchanged.
|
alt-glitch
added
type/bug
13 tasks
9 tasks
14 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On a multiplexed gateway the hourly curator / skills-sync / org-sync housekeeping ticks now run once per served profile under that profile's runtime scope, and a parked MCP server's identical re-park lines no longer flood the error log.
Closes #115713
Supersedes #115775
Closes #116700
Changes
gateway/run_profile_reconcile.py::_for_each_served_profile,profile_scoped_chore;gateway/run.py::_start_gateway_housekeeping): the MCP reconciler's per-served-profile iteration is lifted into a helper andCurator tick,Sync pull tickandOrg sync pull tickride it. Each served profile's tick reads ITS OWN home, config and Nous credentials; single-profile gateways run each chore once against the process home (a standalone gateway flipped into multi-profile hosting binds the launch profile's own scope, as turns do — [Bug]: Hosted-room activation breaks subsequent standalone gateway model resolution #112878).MCPServerTask._log_park; first park = WARNING (with thehermes mcp login <name>remedy, state still visible inhermes mcp list), identical re-park while never revived = DEBUG, healthy session clears the latch so the next outage warns again.Root cause (scope):
_start_gateway_housekeepingruns on a bare thread with no turn on the stack;tools/skills_sync_client.py::resolve_identity→hermes_cli.auth.resolve_nous_runtime_credentialsreads through the fail-closed multiplex reader and logsnous: NOUS_INFERENCE_BASE_URL unreadable — no profile secret scope on a multiplexed call(plus the portal-URL twin) every tick. Live journal on one two-profile gateway: 150 lines / 24h in 29 bursts, 23 of them exactly 6 lines at the top of the hour — the 60-tick cadence, not the MCP re-park timer they were first attributed to (probe: MCP parked-retry runs under the connect-site scope, 0 warnings).Validation
Probe: two temp homes A (launch = multiplex
default) and B (served), differentNOUS_INFERENCE_BASE_URLin each.env,set_multiplex_active(True), 60 ticks driven through_start_gateway_housekeeping.[(A, None)]×3 chores[(A, a-url), (B, b-url)]×3 choresno profile secret scopewarnings, multiplex[(A, a-url)], 0 warningsTests:
tests/gateway/test_housekeeping_profile_scope.py(2 invariants, red on base:[('.hermes', None)] != [('.hermes', a-url), ('b', b-url)]),tests/tools/test_mcp_park_log_dedupe.py(3). Suite:tests/tools tests/gateway tests/hermes_cli— 32253 passed; reds are the known env-red files plustest_dashboard_auth_gate(port 9119 held by the host backend — red on pristine origin/main too) and onetest_cmd_updateuv-not-found flake that passes on re-run.Sibling:
b9-auth-credentials-mcp-connect-scope(fb87caa, d51429e) covers the MCP connect scope; it does not touch the housekeeping sync chores. No overlap.Not covered:
Auto-archive tick(launchstate.dbonly) andCheckpoint prune tick(launchcheckpoints/only) stay launch-scoped — whether the gateway should archive/prune served profiles' stores is a separate decision, listed in #116700's sweep table.Infographic