From c95910ff5ae9b5e9a98f30dc9a9eb7f49cc73104 Mon Sep 17 00:00:00 2001 From: MahdiHedhli <16087011+MahdiHedhli@users.noreply.github.com> Date: Fri, 7 Aug 2026 13:39:03 -0400 Subject: [PATCH 1/8] fix(plugin): derive the IPC idle timeout from the health-check cadence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An "external relay" style plugin — one whose real inference traffic bypasses this IPC channel entirely, going through its own HTTP proxy instead — generates no application-level IPC traffic of its own. The supervisor's periodic HealthRequest (health::HEALTH_CHECK_INTERVAL_SECS) is the *only* guaranteed traffic such a plugin's connection ever sees, so this timeout must comfortably outlast that cadence — the previous fixed 10s value is shorter than the 15s health-check interval, so it will always eventually disconnect an idle-but-healthy relay plugin as falsely idle, race-losing against its own supervisor's probe cadence, well before any genuinely idle timeout should apply. Derive it from HEALTH_CHECK_INTERVAL_SECS directly (4x plus a fixed floor, to absorb a slow/jittery tick) rather than a separate hardcoded constant, so the relationship this timeout depends on stays enforced if that interval ever changes. Found and confirmed via live testing of an external-relay plugin against a real OpenAI-compatible server. Signed-off-by: MahdiHedhli <16087011+MahdiHedhli@users.noreply.github.com> --- .../mesh-llm-host-runtime/src/plugin/transport.rs | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/crates/mesh-llm-host-runtime/src/plugin/transport.rs b/crates/mesh-llm-host-runtime/src/plugin/transport.rs index 6c6f752699..ff52eaa8c1 100644 --- a/crates/mesh-llm-host-runtime/src/plugin/transport.rs +++ b/crates/mesh-llm-host-runtime/src/plugin/transport.rs @@ -13,7 +13,20 @@ use std::time::Duration; use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tokio::sync::{Mutex, mpsc, oneshot}; -const PLUGIN_ENVELOPE_PREFIX_READ_TIMEOUT: Duration = Duration::from_secs(10); +// An "external relay" style plugin (one whose real inference traffic +// bypasses this IPC channel entirely, going through its own HTTP proxy +// instead — see `openai_http(...).managed_by_plugin(false)` plugins) +// generates no application-level IPC traffic of its own. The supervisor's +// periodic `HealthRequest` (see `health::HEALTH_CHECK_INTERVAL_SECS`) is the +// *only* guaranteed traffic such a plugin's connection ever sees, so this +// timeout must comfortably outlast that cadence — a timeout shorter than +// the health-check interval would disconnect every idle-but-healthy relay +// plugin as falsely idle before its next probe ever arrives. 4x the health +// interval plus a fixed floor covers a slow/jittery tick without staying +// pinned to an arbitrarily large value. +const PLUGIN_ENVELOPE_PREFIX_READ_TIMEOUT: Duration = Duration::from_secs( + (super::health::HEALTH_CHECK_INTERVAL_SECS * 4) + 30, +); const PLUGIN_ENVELOPE_BODY_READ_TIMEOUT: Duration = Duration::from_secs(30); const PLUGIN_MESH_STREAM_RESPONSE_TIMEOUT: Duration = Duration::from_secs(super::REQUEST_TIMEOUT_SECS); From 360491b3a30349a3ac55331d9e62be08b8428c93 Mon Sep 17 00:00:00 2001 From: MahdiHedhli <16087011+MahdiHedhli@users.noreply.github.com> Date: Fri, 7 Aug 2026 14:10:42 -0400 Subject: [PATCH 2/8] feat(plugin): make plugin-only nodes discoverable/reachable by mesh peers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A node whose only inference capacity comes from a plugin (e.g. one relaying to an external OpenAI-compatible server — no local model ever loads) is currently invisible to every other mesh peer, for two independent reasons, even though its own local ingress and gossip advertisement of its plugin's models already work correctly: 1. `NodeRole::Host { http_port }` — required for any peer's `hosts_for_model()` to consider a node as a route candidate at all (`mesh/peer_state.rs`) — is set in exactly one place in the whole crate, as a side effect of a local model finishing load (`startup_handles.rs`). A node with zero local models never becomes `Host`, so it's filtered out of every peer's routing consideration regardless of what its gossip payload advertises. Fixed with `spawn_plugin_host_role_watcher`: promotes to `Host` once the plugin manager reports at least one inference model, demotes back to `Worker` when it stops. Reads the plugin manager's already-debounced health state rather than probing anything itself, so a short poll interval doesn't add flapping risk. Only ever touches a `Host` role it set itself, never one a local model set. 2. `tunnel::Manager`'s http_port — which an inbound QUIC-tunneled request from a peer gets relayed to — is likewise only ever set as a side effect of a local model finishing load (`startup_handles.rs`, three call sites). Since `run_auto()` already unconditionally binds and starts the real API proxy for every node before any model resolution happens, the fix is simply to set it right there too, immediately after the tunnel manager is constructed — the three existing call sites in `startup_handles.rs` become redundant-but-harmless (same node, same `api_port`, for the process's lifetime) rather than conflicting with this. Both changes are small and land at a single call site each in `run_auto()`'s already-unconditional startup path, rather than requiring any of the passive-listener-specific machinery an earlier version of this fix (against an older base) needed — that machinery is dead code on current `main` (see `run_auto_model_path_or_shutdown`'s `dead_code` `expect`), superseded by this crate's own "daemon model lifecycle reconciliation" work, which already generalized the node's own local ingress and tunnel manager construction independent of local model selection. This fix closes the one piece that generalization didn't cover: role eligibility for plugin-only nodes. See docs/plugins/plugin-hosted-mesh-serving-design.md for the full analysis. Signed-off-by: MahdiHedhli <16087011+MahdiHedhli@users.noreply.github.com> --- .../src/runtime/run_auto.rs | 69 +++++++++++++++++++ 1 file changed, 69 insertions(+) diff --git a/crates/mesh-llm-host-runtime/src/runtime/run_auto.rs b/crates/mesh-llm-host-runtime/src/runtime/run_auto.rs index 07bc1b6506..cf1f0bad1a 100644 --- a/crates/mesh-llm-host-runtime/src/runtime/run_auto.rs +++ b/crates/mesh-llm-host-runtime/src/runtime/run_auto.rs @@ -1167,6 +1167,59 @@ pub(super) struct RunAutoContext { Option>, } +const PLUGIN_HOST_ROLE_WATCH_INTERVAL: std::time::Duration = std::time::Duration::from_secs(5); + +/// Promotes this node to `NodeRole::Host` whenever a loaded plugin is +/// advertising at least one inference model, and demotes it back to +/// `Worker` when that stops being true. +/// +/// `NodeRole::Host { http_port }` is required for any *other* peer's +/// `hosts_for_model()` to consider this node as a route candidate at all +/// (`mesh/peer_state.rs`) — but it's otherwise only ever set once, as a +/// side effect of a local model finishing load (`startup_handles.rs`). A +/// node whose only inference capacity comes from a plugin (no local model +/// ever loads) would otherwise never become `Host`, so it would be +/// filtered out of every peer's routing consideration regardless of what +/// its gossip payload advertises — gossip already correctly includes +/// plugin-provided models (`mesh/gossip.rs`'s `plugin_inference_models`), +/// so this was purely a role-eligibility gap, not a discovery gap. +/// +/// Only promotes/demotes a role this watcher itself set (tracked via +/// `plugin_promoted_role`, not the node's live role) — it never touches a +/// `Host` role a local model is responsible for. A node running both a +/// local model and a plugin isn't exercised by this watcher; a caller +/// mixing both would need real "why is this node Host" tracking instead +/// of this single-writer assumption. +/// +/// `inference_models()` reads the plugin manager's already-debounced +/// health state (see `plugin::health`'s startup-grace/failure-threshold +/// logic) rather than probing anything itself, so polling it on a short +/// interval doesn't introduce new flapping risk. +fn spawn_plugin_host_role_watcher(node: mesh::Node, plugin_manager: plugin::PluginManager, http_port: u16) { + tokio::spawn(async move { + let mut plugin_promoted_role = false; + loop { + tokio::time::sleep(PLUGIN_HOST_ROLE_WATCH_INTERVAL).await; + let has_plugin_models = plugin_manager + .inference_models() + .await + .map(|models| !models.is_empty()) + .unwrap_or(false); + if has_plugin_models && !plugin_promoted_role { + if node.role().await == NodeRole::Worker { + node.set_role(NodeRole::Host { http_port }).await; + plugin_promoted_role = true; + } + } else if !has_plugin_models && plugin_promoted_role { + if node.role().await == (NodeRole::Host { http_port }) { + node.set_role(NodeRole::Worker).await; + } + plugin_promoted_role = false; + } + } + }); +} + #[expect( clippy::cognitive_complexity, reason = "run_auto is the top-level runtime orchestration path and preserves startup/shutdown ordering" @@ -1243,6 +1296,22 @@ pub(super) async fn run_auto(ctx: RunAutoContext) -> Result<()> { let tunnel_mgr = tunnel::Manager::start(node.clone(), channels.rpc, channels.http, channels.stage).await?; + // Set unconditionally, not only as a side effect of a local model + // finishing load (see `startup_handles.rs`'s three call sites, which + // remain and are now redundant-but-harmless — same node, same + // `api_port`, for the lifetime of the process). The api proxy this + // points at is already bound and already answers correctly with no + // models loaded, so an inbound tunneled request arriving before any + // model is ready gets a normal "not available" response instead of + // being silently dropped (the previous behavior whenever this was + // still 0 — see `network/tunnel.rs`'s `port == 0` early-return). This + // is what lets a plugin-only node (no local model ever loads) accept + // inbound requests at all; see `spawn_plugin_host_role_watcher` below + // for the other half — whether peers actually route here. + tunnel_mgr.set_http_port(api_port); + if !is_client { + spawn_plugin_host_role_watcher(node.clone(), plugin_manager.clone(), api_port); + } // Election publishes per-model targets let (target_tx, target_rx) = tokio::sync::watch::channel(election::ModelTargets::default()); From 0f97965e7d17cc71be8dda0284b0f59182095203 Mon Sep 17 00:00:00 2001 From: MahdiHedhli <16087011+MahdiHedhli@users.noreply.github.com> Date: Fri, 7 Aug 2026 14:11:31 -0400 Subject: [PATCH 3/8] docs(plugins): design doc for mesh-wide plugin-hosted model serving MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents the root cause (with file:line references) and fix shape for the previous commit: a plugin-only node's own local ingress and gossip advertisement already worked correctly, but two independent, narrower gaps — host-role eligibility and inbound tunnel routing — meant no other mesh peer could ever discover or route a request to it. Also documents what's deliberately deferred (capacity/VRAM host ranking, model descriptors, tunnel-path streaming verification, live multi-peer verification) and the history of an earlier, now-reverted version of this fix that targeted a code path a since-landed upstream refactor made unreachable. Signed-off-by: MahdiHedhli <16087011+MahdiHedhli@users.noreply.github.com> --- .../plugin-hosted-mesh-serving-design.md | 140 ++++++++++++++++++ 1 file changed, 140 insertions(+) create mode 100644 docs/plugins/plugin-hosted-mesh-serving-design.md diff --git a/docs/plugins/plugin-hosted-mesh-serving-design.md b/docs/plugins/plugin-hosted-mesh-serving-design.md new file mode 100644 index 0000000000..ddfeb96d29 --- /dev/null +++ b/docs/plugins/plugin-hosted-mesh-serving-design.md @@ -0,0 +1,140 @@ +# Design: mesh-wide serving for plugin-hosted models + +## Status + +Implemented in the commit immediately preceding this document on this +branch. Written after live end-to-end testing (an external-relay plugin +serving an LM Studio model through the mesh) surfaced a gap in how a +plugin-only node — one with zero local (downloaded) models, whose only +inference capacity comes from a plugin endpoint (e.g. +`inference: [openai_http(...)]`) — is treated by the rest of the mesh. + +## What already worked + +Two things were already correct by the time this investigation started, +and are not touched by this change: + +- **This node's own local ingress.** The unified API proxy every node now + binds unconditionally at startup (`run_auto()` → `api_proxy` → + `network/openai/ingress.rs`) already merges `plugin_manager + .inference_models()` into `/v1/models` and already has full + external-endpoint forwarding for plugin-hosted models. This has been + true since before this investigation — an earlier draft of this + document (and an earlier commit on this branch, since dropped) mistook + a *different*, no-longer-primary code path + (`network/openai/transport.rs::handle_mesh_request`, reached only via + the now-`#[expect(dead_code)]` "passive listener" lane — + `run_auto_model_path_or_shutdown` — that a large "daemon model + lifecycle reconciliation" refactor superseded between this fork's base + and current `main`) for the node's primary listener. It isn't; that + patch was reverted as a no-op once the actual current architecture was + understood. See "History" below. +- **Gossip advertisement.** `mesh/gossip.rs`'s + `plugin_inference_models()` already correctly merges plugin-provided + models into what this node tells the mesh it serves. + +## The actual gap + +Even with both of those correct, no *other* peer could ever route a +request to a plugin-only node, for two structural reasons — both +independent of the local-ingress/gossip machinery above, and both still +present on current `main`: + +### 1. Host-role eligibility + +Other peers only consider a node as a candidate host for model X if +`PeerInfo::routes_http_model` returns true, which requires +`accepts_http_inference()` — `matches!(self.role, NodeRole::Host { .. })` +(`mesh/peer_state.rs`). + +`NodeRole::Host { http_port: u16 }` is set in exactly one place in the +whole crate: `startup_handles.rs`, as a side effect of a local model +finishing load. A plugin-only node's role never leaves the default +`Worker`, so **every other peer's `hosts_for_model()` filters this node +out completely**, independent of what its gossip payload advertises. + +### 2. Inbound tunnel routing + +A request another peer *does* decide to route here arrives as a raw byte +relay: an inbound QUIC stream gets forwarded to a local TCP connection at +`tunnel::Manager`'s configured `http_port`. That port defaults to `0` and +is — like the role above — only ever set as a side effect of a local +model finishing load (`startup_handles.rs`, three call sites). If it's +still `0`, an inbound tunnel is dropped with a warning +(`network/tunnel.rs`). + +## Fix + +Both gaps have the same shape: a value that should be a property of "the +node's API surface is up," generalized everywhere else in `run_auto()` +since the daemon-lifecycle-reconciliation refactor, but that these two +specific call sites still only set as a side effect of local-model load +completing. + +- **Tunnel port**: since `run_auto()` already unconditionally constructs + `tunnel::Manager` and binds the real API proxy before any model + resolution happens, set `tunnel_mgr.set_http_port(api_port)` + immediately after construction, unconditionally. The three call sites + in `startup_handles.rs` remain and become redundant-but-harmless (same + node, same `api_port`, for the process's lifetime) rather than + conflicting. +- **Host role**: add `spawn_plugin_host_role_watcher`, a small background + task started alongside the tunnel manager (skipped for `--client` + nodes) that polls `plugin_manager.inference_models()` on a short + interval and promotes to `NodeRole::Host { http_port: api_port }` when + it's non-empty, demoting back to `Worker` when it becomes empty again. + It only ever touches a `Host` role it set itself (tracked via a local + flag, not the node's live role) — never one a local model is + responsible for, so a node that has both isn't affected by this + watcher's demotion path. `inference_models()` reads the plugin + manager's own already-debounced health state (see `plugin::health`'s + startup-grace/failure-threshold logic) rather than probing anything + itself, so the short poll interval doesn't add flapping risk beyond + what that debouncing already provides. + +Both changes are two small, unconditional additions at a single call site +in `run_auto()`'s existing startup path — no new machinery, no threading +through the passive-listener/model-selection context that an earlier +version of this fix (written against an older fork base, before the +daemon-lifecycle-reconciliation refactor landed upstream) needed and that +would have been dead weight added to an already-dead code path. + +## Deliberately deferred + +- **Capacity/VRAM-based host ordering.** `order_remote_hosts_by_context` + and related host-ranking logic (used when multiple peers can serve the + same model) is built around real local VRAM/context-window numbers. A + plugin-relayed model has no meaningful local VRAM figure — needs a + policy decision (advertise an unbounded/synthetic capacity, or exempt + plugin-sourced models from VRAM-based ranking and use a simpler + round-robin/health-only ordering among plugin hosts). +- **Model descriptors.** `all_served_model_descriptors` / + `all_model_runtime_descriptors` assume a real local runtime descriptor + (loaded quant, context length, etc.). Plugin-sourced models need either + synthetic descriptors or these call sites need to become Option-aware. +- **Streaming verification through the tunnel path specifically.** Local + streaming completions through the plugin's proxy are verified working; + the same request routed to this node over an inbound QUIC tunnel from a + peer has not been separately verified. +- **Live multi-peer verification.** This fix compiles cleanly and doesn't + regress the existing single-node acceptance test, but the actual + mesh-wide behavior it targets — another peer discovering and + successfully routing a request to a plugin-only node — has not yet been + exercised by a live multi-peer test. + +## History + +An earlier version of this branch, written against this fork's original +base (before a large "daemon model lifecycle reconciliation" feature and +related refactors landed on upstream `main`), additionally patched +`network/openai/transport.rs::handle_mesh_request` to merge plugin models +into its own `/v1/models` response and to route matching requests +directly to a plugin's loopback endpoint. That function is reached via +`run_passive`, itself reached only through +`run_auto_model_path_or_shutdown` — a function upstream has since marked +`#[expect(dead_code, reason = "bridges the retained advertised-model and +passive runtime compatibility lanes")]`. The refactor made every node's +own local ingress go through the already-plugin-aware +`network/openai/ingress.rs` path unconditionally instead, which made that +patch a no-op on current `main`. It was reverted once this was confirmed, +rather than landing a fix for an unreachable code path. From dc3568eb9859bf218b72c69a6e0e123140938c38 Mon Sep 17 00:00:00 2001 From: MahdiHedhli <16087011+MahdiHedhli@users.noreply.github.com> Date: Fri, 7 Aug 2026 14:42:50 -0400 Subject: [PATCH 4/8] fix(plugin): don't conflate a read error with an empty model list; fix a role-ownership race MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two issues from review: 1. `plugin_manager.inference_models()` returning `Err` was collapsed into `has_plugin_models = false` via `.unwrap_or(false)`, treating a read failure identically to "the plugin genuinely has no models right now." A transient error could therefore demote an otherwise-healthy `Host` role (or silently block a legitimate promotion). Now logs a warning and skips the tick entirely on error, re-evaluating fresh next time rather than acting on an indeterminate read. 2. `http_port` is identical whether a local model or this watcher's plugin-driven promotion is the reason a node is `Host` — both derive it from the same `api_port` — so the role value alone can't distinguish who's responsible for it. If a local model started being served *after* this watcher's promotion, the watcher's stale `plugin_promoted_role` flag could still fire once the plugin's models later disappeared, demoting a role the local model may now depend on. Added a `node.models_being_served().is_empty()` check before actually changing the role on the demotion path — the flag still resets either way (the watcher's original promotion claim is stale regardless), but the role itself is only touched when no local model is active. This isn't full ownership tracking — `node.set_role()` has no concept of "who claimed this and why," and building that is a larger, more invasive change than this fix's scope. Documented as a deliberately-deferred follow-up, and softened the design doc's earlier claim that this watcher "never touches a Host role a local model is responsible for" — that's now accurate for the one race this patch closes, not an enforced guarantee for every possible sequencing. Signed-off-by: MahdiHedhli <16087011+MahdiHedhli@users.noreply.github.com> --- .../src/runtime/run_auto.rs | 35 +++++++++++++--- .../plugin-hosted-mesh-serving-design.md | 42 +++++++++++++++---- 2 files changed, 63 insertions(+), 14 deletions(-) diff --git a/crates/mesh-llm-host-runtime/src/runtime/run_auto.rs b/crates/mesh-llm-host-runtime/src/runtime/run_auto.rs index cf1f0bad1a..0c8d99013a 100644 --- a/crates/mesh-llm-host-runtime/src/runtime/run_auto.rs +++ b/crates/mesh-llm-host-runtime/src/runtime/run_auto.rs @@ -1200,18 +1200,41 @@ fn spawn_plugin_host_role_watcher(node: mesh::Node, plugin_manager: plugin::Plug let mut plugin_promoted_role = false; loop { tokio::time::sleep(PLUGIN_HOST_ROLE_WATCH_INTERVAL).await; - let has_plugin_models = plugin_manager - .inference_models() - .await - .map(|models| !models.is_empty()) - .unwrap_or(false); + let has_plugin_models = match plugin_manager.inference_models().await { + Ok(models) => !models.is_empty(), + Err(error) => { + // A read failure isn't the same as "no models" — treating + // it as such would demote a healthy Host role (or block + // a legitimate promotion) on what's likely a transient + // hiccup. Skip this tick; the next one re-evaluates from + // scratch. + tracing::warn!( + %error, + "plugin host-role watcher: failed to read plugin inference models, skipping this tick" + ); + continue; + } + }; if has_plugin_models && !plugin_promoted_role { if node.role().await == NodeRole::Worker { node.set_role(NodeRole::Host { http_port }).await; plugin_promoted_role = true; } } else if !has_plugin_models && plugin_promoted_role { - if node.role().await == (NodeRole::Host { http_port }) { + // `http_port` is identical whether a local model or a + // plugin is the reason this node is `Host` (both derive it + // from the same `api_port`), so the role value alone can't + // tell the two apart. If a local model has since started + // being served, it may now be the one relying on `Host` + // status — don't pull the role out from under it just + // because this watcher's own earlier promotion is now + // stale. `plugin_promoted_role` still resets either way: + // whatever happens, an "I own this promotion" claim from + // when plugin models were present is no longer accurate + // once they're gone. + let role_is_still_ours = node.role().await == (NodeRole::Host { http_port }) + && node.models_being_served().await.is_empty(); + if role_is_still_ours { node.set_role(NodeRole::Worker).await; } plugin_promoted_role = false; diff --git a/docs/plugins/plugin-hosted-mesh-serving-design.md b/docs/plugins/plugin-hosted-mesh-serving-design.md index ddfeb96d29..eae1f552ee 100644 --- a/docs/plugins/plugin-hosted-mesh-serving-design.md +++ b/docs/plugins/plugin-hosted-mesh-serving-design.md @@ -83,14 +83,30 @@ completing. nodes) that polls `plugin_manager.inference_models()` on a short interval and promotes to `NodeRole::Host { http_port: api_port }` when it's non-empty, demoting back to `Worker` when it becomes empty again. - It only ever touches a `Host` role it set itself (tracked via a local - flag, not the node's live role) — never one a local model is - responsible for, so a node that has both isn't affected by this - watcher's demotion path. `inference_models()` reads the plugin - manager's own already-debounced health state (see `plugin::health`'s - startup-grace/failure-threshold logic) rather than probing anything - itself, so the short poll interval doesn't add flapping risk beyond - what that debouncing already provides. + It tracks whether *it* performed the last promotion via a task-local + flag and only demotes when that flag is set, but this is a soft + safeguard, not real ownership tracking — `node.set_role()` is a plain, + unconditional overwrite with no concept of "who claimed this role and + why," and a local model's `Host { http_port }` (`startup_handles.rs`) + uses the identical `http_port` value this watcher does (both derive it + from the same `api_port`), so the role value alone can't distinguish + the two callers. The demotion path additionally checks + `node.models_being_served()` is empty before actually changing the + role, so a node that starts serving a local model *after* this watcher + promoted it won't have that model's `Host` status pulled out from under + it once the plugin's models later disappear — but this is a targeted + patch for the one race that's currently reachable, not a general + ownership mechanism. A node with more complex overlapping + local-model/plugin lifecycles than "plugin promotes first, local model + starts being served later" isn't fully covered by this. Buzz's relay + feature (the motivating use case) never sets a local `model_id`, so it + doesn't exercise this at all; a caller that does mix both should treat + this watcher's demotion behavior as best-effort until a real + ownership-aware transition mechanism exists. `inference_models()` reads + the plugin manager's own already-debounced health state (see + `plugin::health`'s startup-grace/failure-threshold logic) rather than + probing anything itself, so the short poll interval doesn't add + flapping risk beyond what that debouncing already provides. Both changes are two small, unconditional additions at a single call site in `run_auto()`'s existing startup path — no new machinery, no threading @@ -101,6 +117,16 @@ would have been dead weight added to an already-dead code path. ## Deliberately deferred +- **Real ownership tracking for `NodeRole::Host`.** Nothing in `mesh::Node` + currently tracks *why* a node is `Host` — `set_role()` is a plain + overwrite. The plugin-host-role watcher's task-local flag plus a + `models_being_served()` check (see above) closes the one race that's + reachable today, but a real fix would give `NodeRole::Host` (or a + parallel piece of state) an explicit set of claimants — local model + load, plugin health, potentially others later — and only clear the role + when the last claimant releases it. That's a bigger, more invasive + change than this fix's scope; worth doing if a node mixing local models + and plugins becomes a real use case rather than a theoretical one. - **Capacity/VRAM-based host ordering.** `order_remote_hosts_by_context` and related host-ranking logic (used when multiple peers can serve the same model) is built around real local VRAM/context-window numbers. A From e932d025a034ed2e71f81c0ca53ba59d9006261d Mon Sep 17 00:00:00 2001 From: MahdiHedhli <16087011+MahdiHedhli@users.noreply.github.com> Date: Fri, 7 Aug 2026 15:12:27 -0400 Subject: [PATCH 5/8] docs(plugins): tighten wording accuracy in the design doc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two imprecisions caught by review: - The "Status" intro described the live test as "serving an LM Studio model through the mesh," which reads as claiming mesh-wide (multi-peer) serving was exercised — contradicted a few paragraphs later by "Deliberately deferred"'s explicit "not yet exercised by a live multi-peer test." Reworded to be precise: a single node's own local proxy, through the tunnel path, not yet through an actual inbound request from another peer. - "Both changes are two small, unconditional additions ... no new machinery" undersold the host-role watcher, which is new background machinery and is started conditionally (skipped for --client nodes) — only the tunnel-port fix is truly "unconditional, no new machinery." Reworded to describe each accurately. Signed-off-by: MahdiHedhli <16087011+MahdiHedhli@users.noreply.github.com> --- .../plugin-hosted-mesh-serving-design.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/docs/plugins/plugin-hosted-mesh-serving-design.md b/docs/plugins/plugin-hosted-mesh-serving-design.md index eae1f552ee..b87dc0e953 100644 --- a/docs/plugins/plugin-hosted-mesh-serving-design.md +++ b/docs/plugins/plugin-hosted-mesh-serving-design.md @@ -3,9 +3,11 @@ ## Status Implemented in the commit immediately preceding this document on this -branch. Written after live end-to-end testing (an external-relay plugin -serving an LM Studio model through the mesh) surfaced a gap in how a -plugin-only node — one with zero local (downloaded) models, whose only +branch. Written after live end-to-end testing — a single node running an +external-relay plugin, serving an LM Studio model through that node's own +local proxy (not yet exercised through an actual inbound mesh tunnel from +another peer; see "Deliberately deferred" below) — surfaced a gap in how +a plugin-only node — one with zero local (downloaded) models, whose only inference capacity comes from a plugin endpoint (e.g. `inference: [openai_http(...)]`) — is treated by the rest of the mesh. @@ -108,10 +110,13 @@ completing. probing anything itself, so the short poll interval doesn't add flapping risk beyond what that debouncing already provides. -Both changes are two small, unconditional additions at a single call site -in `run_auto()`'s existing startup path — no new machinery, no threading -through the passive-listener/model-selection context that an earlier -version of this fix (written against an older fork base, before the +Both changes land at a single call site in `run_auto()`'s existing startup +path: the tunnel-port fix is an unconditional one-line addition; the +host-role watcher is new (if small) background machinery, started +conditionally (skipped for `--client` nodes, which have no compute to +offer regardless of plugin state). Neither threads through the +passive-listener/model-selection context that an earlier version of this +fix (written against an older fork base, before the daemon-lifecycle-reconciliation refactor landed upstream) needed and that would have been dead weight added to an already-dead code path. From 74ab7ad9f6d3fb0cf1f1b0f5ef6a1e22133d1348 Mon Sep 17 00:00:00 2001 From: James Dumay Date: Sat, 8 Aug 2026 17:22:55 +1000 Subject: [PATCH 6/8] fix(plugin): make host-role ownership explicit --- crates/mesh-llm-host-runtime/src/mesh/mod.rs | 2 +- crates/mesh-llm-host-runtime/src/mesh/node.rs | 58 ++++++++++++++ .../src/mesh/tests/connections.rs | 3 +- .../src/plugin/transport.rs | 41 ++++++---- .../mesh-llm-host-runtime/src/runtime/mod.rs | 1 + .../src/runtime/plugin_host_role.rs | 35 ++++++++ .../src/runtime/run_auto.rs | 79 +------------------ .../src/runtime/startup_handles.rs | 9 ++- 8 files changed, 130 insertions(+), 98 deletions(-) create mode 100644 crates/mesh-llm-host-runtime/src/runtime/plugin_host_role.rs diff --git a/crates/mesh-llm-host-runtime/src/mesh/mod.rs b/crates/mesh-llm-host-runtime/src/mesh/mod.rs index 67bf98877a..5040ad5758 100644 --- a/crates/mesh-llm-host-runtime/src/mesh/mod.rs +++ b/crates/mesh-llm-host-runtime/src/mesh/mod.rs @@ -133,6 +133,7 @@ pub use identity_persistence::{ load_node_key_from_path, mark_was_public, save_last_mesh_id, save_node_key_to_path, was_previously_public, }; +pub(crate) use node::{HostRoleClaim, PeerDownReport, peer_down_endpoint_id}; #[expect( unused_imports, reason = "public compatibility re-export for existing mesh node callers" @@ -140,7 +141,6 @@ pub use identity_persistence::{ pub use node::{ LocalRequestMetricsSnapshot, Node, RouteEntry, RoutingTable, detect_vram_bytes_capped, }; -pub(crate) use node::{PeerDownReport, peer_down_endpoint_id}; pub(crate) use peer_state::{ ControlListenerLifecycle, DEAD_PEER_TTL, MeshState, PEER_DOWN_REPORTER_COOLDOWN_SECS, PEER_STALE_SECS, resolve_peer_leaving, diff --git a/crates/mesh-llm-host-runtime/src/mesh/node.rs b/crates/mesh-llm-host-runtime/src/mesh/node.rs index f871aeae85..96682883da 100644 --- a/crates/mesh-llm-host-runtime/src/mesh/node.rs +++ b/crates/mesh-llm-host-runtime/src/mesh/node.rs @@ -1,6 +1,7 @@ use super::*; use mesh_llm_types::mesh::{DEMAND_TTL_SECS, merge_demand}; use serde_json::json; +use std::collections::BTreeMap; use std::net::SocketAddr; mod startup; @@ -11,6 +12,12 @@ use startup::{ }; pub(crate) use startup::{default_plugin_event_source, hardware_snapshot_for_start}; +#[derive(Debug, Clone, Copy, Eq, Hash, Ord, PartialEq, PartialOrd)] +pub(crate) enum HostRoleClaim { + LocalModel, + PluginInference, +} + /// Lightweight routing table for passive nodes (clients + standby GPU). /// Contains just enough info to route requests to the right host. #[derive(Debug, Clone, Serialize, Deserialize)] @@ -40,6 +47,7 @@ pub struct Node { pub(crate) local_mesh_requirements: crate::MeshRequirements, pub(crate) state: Arc>, pub(crate) role: Arc>, + pub(crate) host_role_claims: Arc>>, pub(crate) models: Arc>>, pub(crate) model_source: Arc>>, pub(crate) serving_models: Arc>>, @@ -775,6 +783,7 @@ impl Node { recent_mesh_rejections: VecDeque::new(), })), role: Arc::new(Mutex::new(role)), + host_role_claims: Arc::new(Mutex::new(BTreeMap::new())), models: Arc::new(Mutex::new(Vec::new())), model_source: Arc::new(Mutex::new(None)), serving_models: Arc::new(Mutex::new(Vec::new())), @@ -950,6 +959,7 @@ impl Node { recent_mesh_rejections: VecDeque::new(), })), role: Arc::new(Mutex::new(role)), + host_role_claims: Arc::new(Mutex::new(BTreeMap::new())), models: Arc::new(Mutex::new(Vec::new())), model_source: Arc::new(Mutex::new(None)), serving_models: Arc::new(Mutex::new(Vec::new())), @@ -1154,10 +1164,38 @@ impl Node { self.role.lock().await.clone() } + #[cfg(test)] pub async fn set_role(&self, role: NodeRole) { *self.role.lock().await = role; } + pub async fn claim_host_role(&self, claim: HostRoleClaim, http_port: u16) { + let mut claims = self.host_role_claims.lock().await; + *claims.entry(claim).or_default() += 1; + let mut role = self.role.lock().await; + if matches!(*role, NodeRole::Worker) { + *role = NodeRole::Host { http_port }; + } + } + + pub async fn release_host_role(&self, claim: HostRoleClaim) { + let mut claims = self.host_role_claims.lock().await; + let Some(count) = claims.get_mut(&claim) else { + return; + }; + if *count > 1 { + *count -= 1; + } else { + claims.remove(&claim); + } + if claims.is_empty() { + let mut role = self.role.lock().await; + if matches!(*role, NodeRole::Host { .. }) { + *role = NodeRole::Worker; + } + } + } + pub async fn set_release_attestation_report( &self, summary: crate::ReleaseAttestationSummary, @@ -1816,6 +1854,26 @@ mod node_tests { } } + #[tokio::test] + async fn host_role_claims_are_reference_counted_across_sources() { + let node = Node::new_for_tests(NodeRole::Worker).await.unwrap(); + + node.claim_host_role(HostRoleClaim::LocalModel, 9337).await; + node.claim_host_role(HostRoleClaim::PluginInference, 9337) + .await; + assert_eq!(node.role().await, NodeRole::Host { http_port: 9337 }); + + node.release_host_role(HostRoleClaim::PluginInference).await; + assert_eq!(node.role().await, NodeRole::Host { http_port: 9337 }); + + node.claim_host_role(HostRoleClaim::LocalModel, 9337).await; + node.release_host_role(HostRoleClaim::LocalModel).await; + assert_eq!(node.role().await, NodeRole::Host { http_port: 9337 }); + + node.release_host_role(HostRoleClaim::LocalModel).await; + assert_eq!(node.role().await, NodeRole::Worker); + } + #[tokio::test] async fn update_peer_rtt_notifies_when_first_sample_is_split_eligible() { let node = Node::new_for_tests(NodeRole::Worker).await.unwrap(); diff --git a/crates/mesh-llm-host-runtime/src/mesh/tests/connections.rs b/crates/mesh-llm-host-runtime/src/mesh/tests/connections.rs index b202dc87df..aca3058c07 100644 --- a/crates/mesh-llm-host-runtime/src/mesh/tests/connections.rs +++ b/crates/mesh-llm-host-runtime/src/mesh/tests/connections.rs @@ -11,7 +11,7 @@ use crate::plugin; use crate::proto::node::{GossipFrame, NodeRole, PeerAnnouncement, RouteTableRequest}; use serial_test::serial; use skippy_protocol::proto::stage as skippy_stage_proto; -use std::collections::{HashMap, HashSet}; +use std::collections::{BTreeMap, HashMap, HashSet}; use tokio::sync::{mpsc, watch}; mod direct_path; @@ -418,6 +418,7 @@ async fn make_test_node_with_requirements( recent_mesh_rejections: VecDeque::new(), })), role: Arc::new(Mutex::new(role)), + host_role_claims: Arc::new(Mutex::new(BTreeMap::new())), models: Arc::new(Mutex::new(Vec::new())), model_source: Arc::new(Mutex::new(None)), serving_models: Arc::new(Mutex::new(Vec::new())), diff --git a/crates/mesh-llm-host-runtime/src/plugin/transport.rs b/crates/mesh-llm-host-runtime/src/plugin/transport.rs index ff52eaa8c1..9a92fdfa20 100644 --- a/crates/mesh-llm-host-runtime/src/plugin/transport.rs +++ b/crates/mesh-llm-host-runtime/src/plugin/transport.rs @@ -24,10 +24,11 @@ use tokio::sync::{Mutex, mpsc, oneshot}; // plugin as falsely idle before its next probe ever arrives. 4x the health // interval plus a fixed floor covers a slow/jittery tick without staying // pinned to an arbitrarily large value. -const PLUGIN_ENVELOPE_PREFIX_READ_TIMEOUT: Duration = Duration::from_secs( - (super::health::HEALTH_CHECK_INTERVAL_SECS * 4) + 30, -); +const PLUGIN_ENVELOPE_PREFIX_READ_TIMEOUT: Duration = + Duration::from_secs((super::health::HEALTH_CHECK_INTERVAL_SECS * 4) + 30); const PLUGIN_ENVELOPE_BODY_READ_TIMEOUT: Duration = Duration::from_secs(30); +#[cfg(test)] +const PLUGIN_ENVELOPE_TEST_READ_TIMEOUT: Duration = Duration::from_secs(1); const PLUGIN_MESH_STREAM_RESPONSE_TIMEOUT: Duration = Duration::from_secs(super::REQUEST_TIMEOUT_SECS); @@ -420,31 +421,41 @@ pub(crate) async fn read_envelope(stream: &mut LocalStream) -> Result(stream: &mut R) -> Result +where + R: tokio::io::AsyncRead + Unpin, +{ + read_envelope_from_reader_with_timeouts( + stream, + PLUGIN_ENVELOPE_TEST_READ_TIMEOUT, + PLUGIN_ENVELOPE_TEST_READ_TIMEOUT, + ) + .await +} + +#[cfg(test)] +async fn read_envelope_from_reader_with_timeouts( + stream: &mut R, + prefix_timeout: Duration, + body_timeout: Duration, +) -> Result where R: tokio::io::AsyncRead + Unpin, { let mut len_buf = [0u8; 4]; tokio::time::timeout( - PLUGIN_ENVELOPE_PREFIX_READ_TIMEOUT, + prefix_timeout, AsyncReadExt::read_exact(stream, &mut len_buf), ) .await - .map_err(|_| { - anyhow!("timeout reading plugin frame prefix after {PLUGIN_ENVELOPE_PREFIX_READ_TIMEOUT:?}") - })??; + .map_err(|_| anyhow!("timeout reading plugin frame prefix after {prefix_timeout:?}"))??; let len = u32::from_le_bytes(len_buf) as usize; if len > 16 * 1024 * 1024 { bail!("Plugin frame too large"); } let mut body = vec![0u8; len]; - tokio::time::timeout( - PLUGIN_ENVELOPE_BODY_READ_TIMEOUT, - AsyncReadExt::read_exact(stream, &mut body), - ) - .await - .map_err(|_| { - anyhow!("timeout reading plugin frame body after {PLUGIN_ENVELOPE_BODY_READ_TIMEOUT:?}") - })??; + tokio::time::timeout(body_timeout, AsyncReadExt::read_exact(stream, &mut body)) + .await + .map_err(|_| anyhow!("timeout reading plugin frame body after {body_timeout:?}"))??; Ok(prost::Message::decode(body.as_slice())?) } diff --git a/crates/mesh-llm-host-runtime/src/runtime/mod.rs b/crates/mesh-llm-host-runtime/src/runtime/mod.rs index 6fd8e6f031..5bb4a9dbf6 100644 --- a/crates/mesh-llm-host-runtime/src/runtime/mod.rs +++ b/crates/mesh-llm-host-runtime/src/runtime/mod.rs @@ -17,6 +17,7 @@ mod local_split; mod model_lifecycle; pub(crate) mod model_reconciliation; mod options; +mod plugin_host_role; mod proxy; mod publication; mod release_attestation; diff --git a/crates/mesh-llm-host-runtime/src/runtime/plugin_host_role.rs b/crates/mesh-llm-host-runtime/src/runtime/plugin_host_role.rs new file mode 100644 index 0000000000..8dd0f38bb8 --- /dev/null +++ b/crates/mesh-llm-host-runtime/src/runtime/plugin_host_role.rs @@ -0,0 +1,35 @@ +use crate::{mesh, plugin}; +use std::time::Duration; + +const WATCH_INTERVAL: Duration = Duration::from_secs(5); + +/// Promotes this node to `NodeRole::Host` whenever a loaded plugin is +/// advertising at least one inference model, and releases the plugin's host +/// claim when that stops being true. +pub(super) fn spawn(node: mesh::Node, plugin_manager: plugin::PluginManager, http_port: u16) { + tokio::spawn(async move { + let mut plugin_claimed = false; + loop { + tokio::time::sleep(WATCH_INTERVAL).await; + let has_plugin_models = match plugin_manager.inference_models().await { + Ok(models) => !models.is_empty(), + Err(error) => { + tracing::warn!( + %error, + "plugin host-role watcher: failed to read plugin inference models, skipping this tick" + ); + continue; + } + }; + if has_plugin_models && !plugin_claimed { + node.claim_host_role(mesh::HostRoleClaim::PluginInference, http_port) + .await; + plugin_claimed = true; + } else if !has_plugin_models && plugin_claimed { + node.release_host_role(mesh::HostRoleClaim::PluginInference) + .await; + plugin_claimed = false; + } + } + }); +} diff --git a/crates/mesh-llm-host-runtime/src/runtime/run_auto.rs b/crates/mesh-llm-host-runtime/src/runtime/run_auto.rs index 0c8d99013a..8b35952e1b 100644 --- a/crates/mesh-llm-host-runtime/src/runtime/run_auto.rs +++ b/crates/mesh-llm-host-runtime/src/runtime/run_auto.rs @@ -1,4 +1,5 @@ use super::daemon_startup::{check_mode_conflicts, resolve_effective_mode}; +use super::plugin_host_role; use super::startup_identity::{emit_private_mesh_name_warning, handle_public_identity_transition}; use super::status::mesh_guardrail_mode_to_openai; use super::{ @@ -1167,82 +1168,6 @@ pub(super) struct RunAutoContext { Option>, } -const PLUGIN_HOST_ROLE_WATCH_INTERVAL: std::time::Duration = std::time::Duration::from_secs(5); - -/// Promotes this node to `NodeRole::Host` whenever a loaded plugin is -/// advertising at least one inference model, and demotes it back to -/// `Worker` when that stops being true. -/// -/// `NodeRole::Host { http_port }` is required for any *other* peer's -/// `hosts_for_model()` to consider this node as a route candidate at all -/// (`mesh/peer_state.rs`) — but it's otherwise only ever set once, as a -/// side effect of a local model finishing load (`startup_handles.rs`). A -/// node whose only inference capacity comes from a plugin (no local model -/// ever loads) would otherwise never become `Host`, so it would be -/// filtered out of every peer's routing consideration regardless of what -/// its gossip payload advertises — gossip already correctly includes -/// plugin-provided models (`mesh/gossip.rs`'s `plugin_inference_models`), -/// so this was purely a role-eligibility gap, not a discovery gap. -/// -/// Only promotes/demotes a role this watcher itself set (tracked via -/// `plugin_promoted_role`, not the node's live role) — it never touches a -/// `Host` role a local model is responsible for. A node running both a -/// local model and a plugin isn't exercised by this watcher; a caller -/// mixing both would need real "why is this node Host" tracking instead -/// of this single-writer assumption. -/// -/// `inference_models()` reads the plugin manager's already-debounced -/// health state (see `plugin::health`'s startup-grace/failure-threshold -/// logic) rather than probing anything itself, so polling it on a short -/// interval doesn't introduce new flapping risk. -fn spawn_plugin_host_role_watcher(node: mesh::Node, plugin_manager: plugin::PluginManager, http_port: u16) { - tokio::spawn(async move { - let mut plugin_promoted_role = false; - loop { - tokio::time::sleep(PLUGIN_HOST_ROLE_WATCH_INTERVAL).await; - let has_plugin_models = match plugin_manager.inference_models().await { - Ok(models) => !models.is_empty(), - Err(error) => { - // A read failure isn't the same as "no models" — treating - // it as such would demote a healthy Host role (or block - // a legitimate promotion) on what's likely a transient - // hiccup. Skip this tick; the next one re-evaluates from - // scratch. - tracing::warn!( - %error, - "plugin host-role watcher: failed to read plugin inference models, skipping this tick" - ); - continue; - } - }; - if has_plugin_models && !plugin_promoted_role { - if node.role().await == NodeRole::Worker { - node.set_role(NodeRole::Host { http_port }).await; - plugin_promoted_role = true; - } - } else if !has_plugin_models && plugin_promoted_role { - // `http_port` is identical whether a local model or a - // plugin is the reason this node is `Host` (both derive it - // from the same `api_port`), so the role value alone can't - // tell the two apart. If a local model has since started - // being served, it may now be the one relying on `Host` - // status — don't pull the role out from under it just - // because this watcher's own earlier promotion is now - // stale. `plugin_promoted_role` still resets either way: - // whatever happens, an "I own this promotion" claim from - // when plugin models were present is no longer accurate - // once they're gone. - let role_is_still_ours = node.role().await == (NodeRole::Host { http_port }) - && node.models_being_served().await.is_empty(); - if role_is_still_ours { - node.set_role(NodeRole::Worker).await; - } - plugin_promoted_role = false; - } - } - }); -} - #[expect( clippy::cognitive_complexity, reason = "run_auto is the top-level runtime orchestration path and preserves startup/shutdown ordering" @@ -1333,7 +1258,7 @@ pub(super) async fn run_auto(ctx: RunAutoContext) -> Result<()> { // for the other half — whether peers actually route here. tunnel_mgr.set_http_port(api_port); if !is_client { - spawn_plugin_host_role_watcher(node.clone(), plugin_manager.clone(), api_port); + plugin_host_role::spawn(node.clone(), plugin_manager.clone(), api_port); } // Election publishes per-model targets diff --git a/crates/mesh-llm-host-runtime/src/runtime/startup_handles.rs b/crates/mesh-llm-host-runtime/src/runtime/startup_handles.rs index e3e540e3bd..3efb7a5898 100644 --- a/crates/mesh-llm-host-runtime/src/runtime/startup_handles.rs +++ b/crates/mesh-llm-host-runtime/src/runtime/startup_handles.rs @@ -18,7 +18,7 @@ use super::{ }; use crate::api; use crate::inference::{election, skippy}; -use crate::mesh::{self, NodeRole}; +use crate::mesh; use crate::network::tunnel; use crate::plugin; use crate::runtime::interactive; @@ -951,6 +951,9 @@ pub(super) async fn startup_shutdown_local_model_loop( task.abort(); let _ = task.await; } + ctx.node + .release_host_role(mesh::HostRoleClaim::LocalModel) + .await; if !state.survey_exited_unexpectedly { ctx.survey_telemetry .record_unload(&state.survey_loaded_model); @@ -1517,9 +1520,7 @@ pub(super) async fn startup_publish_loaded_runtime( ) { let payload = startup_register_loaded_runtime(ctx, loaded_name, handle).await; ctx.node - .set_role(NodeRole::Host { - http_port: ctx.api_port, - }) + .claim_host_role(mesh::HostRoleClaim::LocalModel, ctx.api_port) .await; refresh_dashboard_context_usage(ctx.dashboard_context_usage, loaded_name, handle).await; publish_runtime_llama_slots( From 8944bc43a59967166708cdb3289347fa0e00a38d Mon Sep 17 00:00:00 2001 From: James Dumay Date: Tue, 11 Aug 2026 07:36:35 +1000 Subject: [PATCH 7/8] fix: address plugin serving review feedback --- .../src/mesh/host_role_claims.rs | 98 +++++++++++++++++++ crates/mesh-llm-host-runtime/src/mesh/mod.rs | 4 +- crates/mesh-llm-host-runtime/src/mesh/node.rs | 60 +----------- .../src/mesh/tests/connections.rs | 63 +++++++++++- .../src/runtime/startup_handles.rs | 5 + .../plugin-hosted-mesh-serving-design.md | 70 +++++-------- 6 files changed, 194 insertions(+), 106 deletions(-) create mode 100644 crates/mesh-llm-host-runtime/src/mesh/host_role_claims.rs diff --git a/crates/mesh-llm-host-runtime/src/mesh/host_role_claims.rs b/crates/mesh-llm-host-runtime/src/mesh/host_role_claims.rs new file mode 100644 index 0000000000..b3b50e36fa --- /dev/null +++ b/crates/mesh-llm-host-runtime/src/mesh/host_role_claims.rs @@ -0,0 +1,98 @@ +use super::node::Node; +use super::peer_state::NodeRole; +use std::collections::BTreeMap; + +#[derive(Debug, Clone, Copy, Eq, Hash, Ord, PartialEq, PartialOrd)] +pub(crate) enum HostRoleClaim { + LocalModel, + PluginInference, +} + +#[derive(Default)] +pub(crate) struct HostRoleClaims(BTreeMap); + +impl HostRoleClaims { + fn claim(&mut self, claim: HostRoleClaim) { + *self.0.entry(claim).or_default() += 1; + } + + fn release(&mut self, claim: HostRoleClaim) -> bool { + let Some(count) = self.0.get_mut(&claim) else { + return false; + }; + if *count > 1 { + *count -= 1; + } else { + self.0.remove(&claim); + } + true + } + + fn is_empty(&self) -> bool { + self.0.is_empty() + } +} + +impl Node { + pub async fn claim_host_role(&self, claim: HostRoleClaim, http_port: u16) { + let transitioned = { + let mut claims = self.host_role_claims.lock().await; + claims.claim(claim); + let mut role = self.role.lock().await; + if matches!(*role, NodeRole::Worker) { + *role = NodeRole::Host { http_port }; + true + } else { + false + } + }; + if transitioned { + self.regossip().await; + } + } + + pub async fn release_host_role(&self, claim: HostRoleClaim) { + let transitioned = { + let mut claims = self.host_role_claims.lock().await; + if !claims.release(claim) || !claims.is_empty() { + false + } else { + let mut role = self.role.lock().await; + if matches!(*role, NodeRole::Host { .. }) { + *role = NodeRole::Worker; + true + } else { + false + } + } + }; + if transitioned { + self.regossip().await; + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[tokio::test] + async fn host_role_claims_are_reference_counted_across_sources() { + let node = Node::new_for_tests(NodeRole::Worker).await.unwrap(); + + node.claim_host_role(HostRoleClaim::LocalModel, 9337).await; + node.claim_host_role(HostRoleClaim::PluginInference, 9337) + .await; + assert_eq!(node.role().await, NodeRole::Host { http_port: 9337 }); + + node.release_host_role(HostRoleClaim::PluginInference).await; + assert_eq!(node.role().await, NodeRole::Host { http_port: 9337 }); + + node.claim_host_role(HostRoleClaim::LocalModel, 9337).await; + node.release_host_role(HostRoleClaim::LocalModel).await; + assert_eq!(node.role().await, NodeRole::Host { http_port: 9337 }); + + node.release_host_role(HostRoleClaim::LocalModel).await; + assert_eq!(node.role().await, NodeRole::Worker); + } +} diff --git a/crates/mesh-llm-host-runtime/src/mesh/mod.rs b/crates/mesh-llm-host-runtime/src/mesh/mod.rs index 5040ad5758..36584b9d8c 100644 --- a/crates/mesh-llm-host-runtime/src/mesh/mod.rs +++ b/crates/mesh-llm-host-runtime/src/mesh/mod.rs @@ -85,6 +85,7 @@ mod connections; mod direct_path; mod gossip; mod heartbeat; +mod host_role_claims; mod identity_persistence; mod lan_bootstrap; mod model_identity; @@ -107,6 +108,7 @@ mod stun; use connection_reservation::*; use connections::*; +pub(crate) use host_role_claims::{HostRoleClaim, HostRoleClaims}; use model_identity::*; use node_identity::*; use owner_control::*; @@ -133,7 +135,6 @@ pub use identity_persistence::{ load_node_key_from_path, mark_was_public, save_last_mesh_id, save_node_key_to_path, was_previously_public, }; -pub(crate) use node::{HostRoleClaim, PeerDownReport, peer_down_endpoint_id}; #[expect( unused_imports, reason = "public compatibility re-export for existing mesh node callers" @@ -141,6 +142,7 @@ pub(crate) use node::{HostRoleClaim, PeerDownReport, peer_down_endpoint_id}; pub use node::{ LocalRequestMetricsSnapshot, Node, RouteEntry, RoutingTable, detect_vram_bytes_capped, }; +pub(crate) use node::{PeerDownReport, peer_down_endpoint_id}; pub(crate) use peer_state::{ ControlListenerLifecycle, DEAD_PEER_TTL, MeshState, PEER_DOWN_REPORTER_COOLDOWN_SECS, PEER_STALE_SECS, resolve_peer_leaving, diff --git a/crates/mesh-llm-host-runtime/src/mesh/node.rs b/crates/mesh-llm-host-runtime/src/mesh/node.rs index 96682883da..00b17bd414 100644 --- a/crates/mesh-llm-host-runtime/src/mesh/node.rs +++ b/crates/mesh-llm-host-runtime/src/mesh/node.rs @@ -1,7 +1,6 @@ use super::*; use mesh_llm_types::mesh::{DEMAND_TTL_SECS, merge_demand}; use serde_json::json; -use std::collections::BTreeMap; use std::net::SocketAddr; mod startup; @@ -12,12 +11,6 @@ use startup::{ }; pub(crate) use startup::{default_plugin_event_source, hardware_snapshot_for_start}; -#[derive(Debug, Clone, Copy, Eq, Hash, Ord, PartialEq, PartialOrd)] -pub(crate) enum HostRoleClaim { - LocalModel, - PluginInference, -} - /// Lightweight routing table for passive nodes (clients + standby GPU). /// Contains just enough info to route requests to the right host. #[derive(Debug, Clone, Serialize, Deserialize)] @@ -47,7 +40,7 @@ pub struct Node { pub(crate) local_mesh_requirements: crate::MeshRequirements, pub(crate) state: Arc>, pub(crate) role: Arc>, - pub(crate) host_role_claims: Arc>>, + pub(crate) host_role_claims: Arc>, pub(crate) models: Arc>>, pub(crate) model_source: Arc>>, pub(crate) serving_models: Arc>>, @@ -783,7 +776,7 @@ impl Node { recent_mesh_rejections: VecDeque::new(), })), role: Arc::new(Mutex::new(role)), - host_role_claims: Arc::new(Mutex::new(BTreeMap::new())), + host_role_claims: Arc::new(Mutex::new(HostRoleClaims::default())), models: Arc::new(Mutex::new(Vec::new())), model_source: Arc::new(Mutex::new(None)), serving_models: Arc::new(Mutex::new(Vec::new())), @@ -959,7 +952,7 @@ impl Node { recent_mesh_rejections: VecDeque::new(), })), role: Arc::new(Mutex::new(role)), - host_role_claims: Arc::new(Mutex::new(BTreeMap::new())), + host_role_claims: Arc::new(Mutex::new(HostRoleClaims::default())), models: Arc::new(Mutex::new(Vec::new())), model_source: Arc::new(Mutex::new(None)), serving_models: Arc::new(Mutex::new(Vec::new())), @@ -1169,33 +1162,6 @@ impl Node { *self.role.lock().await = role; } - pub async fn claim_host_role(&self, claim: HostRoleClaim, http_port: u16) { - let mut claims = self.host_role_claims.lock().await; - *claims.entry(claim).or_default() += 1; - let mut role = self.role.lock().await; - if matches!(*role, NodeRole::Worker) { - *role = NodeRole::Host { http_port }; - } - } - - pub async fn release_host_role(&self, claim: HostRoleClaim) { - let mut claims = self.host_role_claims.lock().await; - let Some(count) = claims.get_mut(&claim) else { - return; - }; - if *count > 1 { - *count -= 1; - } else { - claims.remove(&claim); - } - if claims.is_empty() { - let mut role = self.role.lock().await; - if matches!(*role, NodeRole::Host { .. }) { - *role = NodeRole::Worker; - } - } - } - pub async fn set_release_attestation_report( &self, summary: crate::ReleaseAttestationSummary, @@ -1854,26 +1820,6 @@ mod node_tests { } } - #[tokio::test] - async fn host_role_claims_are_reference_counted_across_sources() { - let node = Node::new_for_tests(NodeRole::Worker).await.unwrap(); - - node.claim_host_role(HostRoleClaim::LocalModel, 9337).await; - node.claim_host_role(HostRoleClaim::PluginInference, 9337) - .await; - assert_eq!(node.role().await, NodeRole::Host { http_port: 9337 }); - - node.release_host_role(HostRoleClaim::PluginInference).await; - assert_eq!(node.role().await, NodeRole::Host { http_port: 9337 }); - - node.claim_host_role(HostRoleClaim::LocalModel, 9337).await; - node.release_host_role(HostRoleClaim::LocalModel).await; - assert_eq!(node.role().await, NodeRole::Host { http_port: 9337 }); - - node.release_host_role(HostRoleClaim::LocalModel).await; - assert_eq!(node.role().await, NodeRole::Worker); - } - #[tokio::test] async fn update_peer_rtt_notifies_when_first_sample_is_split_eligible() { let node = Node::new_for_tests(NodeRole::Worker).await.unwrap(); diff --git a/crates/mesh-llm-host-runtime/src/mesh/tests/connections.rs b/crates/mesh-llm-host-runtime/src/mesh/tests/connections.rs index aca3058c07..111e361b5c 100644 --- a/crates/mesh-llm-host-runtime/src/mesh/tests/connections.rs +++ b/crates/mesh-llm-host-runtime/src/mesh/tests/connections.rs @@ -11,7 +11,7 @@ use crate::plugin; use crate::proto::node::{GossipFrame, NodeRole, PeerAnnouncement, RouteTableRequest}; use serial_test::serial; use skippy_protocol::proto::stage as skippy_stage_proto; -use std::collections::{BTreeMap, HashMap, HashSet}; +use std::collections::{HashMap, HashSet}; use tokio::sync::{mpsc, watch}; mod direct_path; @@ -418,7 +418,7 @@ async fn make_test_node_with_requirements( recent_mesh_rejections: VecDeque::new(), })), role: Arc::new(Mutex::new(role)), - host_role_claims: Arc::new(Mutex::new(BTreeMap::new())), + host_role_claims: Arc::new(Mutex::new(HostRoleClaims::default())), models: Arc::new(Mutex::new(Vec::new())), model_source: Arc::new(Mutex::new(None)), serving_models: Arc::new(Mutex::new(Vec::new())), @@ -506,6 +506,65 @@ async fn make_test_node_with_requirements( Ok(node) } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn host_role_claim_transitions_regossip_to_connected_peer() -> Result<()> { + let host = make_test_node(super::NodeRole::Worker).await?; + let peer = make_test_node(super::NodeRole::Worker).await?; + host.set_mesh_id("host-role-claim-regossip-test".to_string()) + .await; + peer.set_mesh_id("host-role-claim-regossip-test".to_string()) + .await; + host.start_accepting(); + peer.start_accepting(); + + let host_id = host.id(); + peer.join(&host.invite_token().await).await?; + wait_for_peer(&peer, host_id).await; + wait_for_peer(&host, peer.id()).await; + + host.claim_host_role(super::HostRoleClaim::PluginInference, 9337) + .await; + tokio::time::timeout(std::time::Duration::from_secs(5), async { + loop { + if peer + .peers() + .await + .into_iter() + .find(|candidate| candidate.id == host_id) + .is_some_and(|candidate| { + candidate.role == super::NodeRole::Host { http_port: 9337 } + }) + { + break; + } + tokio::time::sleep(std::time::Duration::from_millis(25)).await; + } + }) + .await + .expect("connected peer should receive host promotion gossip"); + + host.release_host_role(super::HostRoleClaim::PluginInference) + .await; + tokio::time::timeout(std::time::Duration::from_secs(5), async { + loop { + if peer + .peers() + .await + .into_iter() + .find(|candidate| candidate.id == host_id) + .is_some_and(|candidate| candidate.role == super::NodeRole::Worker) + { + break; + } + tokio::time::sleep(std::time::Duration::from_millis(25)).await; + } + }) + .await + .expect("connected peer should receive host demotion gossip"); + + Ok(()) +} + #[tokio::test] async fn set_serving_models_preserves_existing_known_descriptor_capabilities_when_adding_model() -> Result<()> { diff --git a/crates/mesh-llm-host-runtime/src/runtime/startup_handles.rs b/crates/mesh-llm-host-runtime/src/runtime/startup_handles.rs index 3efb7a5898..2f20517cb7 100644 --- a/crates/mesh-llm-host-runtime/src/runtime/startup_handles.rs +++ b/crates/mesh-llm-host-runtime/src/runtime/startup_handles.rs @@ -664,6 +664,11 @@ pub(super) async fn startup_handle_fallback_failure( event.reason, event.generation, unavailable_stage_nodes )), }); + // The failed fallback exits without the shared shutdown path, so release + // the local-model host claim explicitly before returning to the runtime. + ctx.node + .release_host_role(mesh::HostRoleClaim::LocalModel) + .await; startup_remove_runtime_instance_artifacts(ctx, model_name).await; StartupLoopControl::Return } diff --git a/docs/plugins/plugin-hosted-mesh-serving-design.md b/docs/plugins/plugin-hosted-mesh-serving-design.md index b87dc0e953..ee1361c90e 100644 --- a/docs/plugins/plugin-hosted-mesh-serving-design.md +++ b/docs/plugins/plugin-hosted-mesh-serving-design.md @@ -2,11 +2,11 @@ ## Status -Implemented in the commit immediately preceding this document on this -branch. Written after live end-to-end testing — a single node running an -external-relay plugin, serving an LM Studio model through that node's own -local proxy (not yet exercised through an actual inbound mesh tunnel from -another peer; see "Deliberately deferred" below) — surfaced a gap in how +Implemented in this branch. A live end-to-end test covered a single node +running an external-relay plugin and serving an LM Studio model through that +node's own local proxy; it did not exercise an inbound mesh tunnel from +another peer or mesh-wide discovery and routing (see "Deliberately deferred" +below). That test surfaced a gap in how a plugin-only node — one with zero local (downloaded) models, whose only inference capacity comes from a plugin endpoint (e.g. `inference: [openai_http(...)]`) — is treated by the rest of the mesh. @@ -83,37 +83,21 @@ completing. - **Host role**: add `spawn_plugin_host_role_watcher`, a small background task started alongside the tunnel manager (skipped for `--client` nodes) that polls `plugin_manager.inference_models()` on a short - interval and promotes to `NodeRole::Host { http_port: api_port }` when - it's non-empty, demoting back to `Worker` when it becomes empty again. - It tracks whether *it* performed the last promotion via a task-local - flag and only demotes when that flag is set, but this is a soft - safeguard, not real ownership tracking — `node.set_role()` is a plain, - unconditional overwrite with no concept of "who claimed this role and - why," and a local model's `Host { http_port }` (`startup_handles.rs`) - uses the identical `http_port` value this watcher does (both derive it - from the same `api_port`), so the role value alone can't distinguish - the two callers. The demotion path additionally checks - `node.models_being_served()` is empty before actually changing the - role, so a node that starts serving a local model *after* this watcher - promoted it won't have that model's `Host` status pulled out from under - it once the plugin's models later disappear — but this is a targeted - patch for the one race that's currently reachable, not a general - ownership mechanism. A node with more complex overlapping - local-model/plugin lifecycles than "plugin promotes first, local model - starts being served later" isn't fully covered by this. Buzz's relay - feature (the motivating use case) never sets a local `model_id`, so it - doesn't exercise this at all; a caller that does mix both should treat - this watcher's demotion behavior as best-effort until a real - ownership-aware transition mechanism exists. `inference_models()` reads - the plugin manager's own already-debounced health state (see - `plugin::health`'s startup-grace/failure-threshold logic) rather than - probing anything itself, so the short poll interval doesn't add - flapping risk beyond what that debouncing already provides. - -Both changes land at a single call site in `run_auto()`'s existing startup -path: the tunnel-port fix is an unconditional one-line addition; the -host-role watcher is new (if small) background machinery, started -conditionally (skipped for `--client` nodes, which have no compute to + interval and claims `NodeRole::Host { http_port: api_port }` when the + successful result is non-empty, releasing that claim when it becomes + empty again. Errors are logged and leave the previous role state intact + until a successful sample arrives. `mesh::Node` reference-counts local + model and plugin claims, demotes only after the final claimant releases, + and re-gossips actual role transitions. The local-model `Exit` fallback + path also releases its claim explicitly because it bypasses the shared + shutdown teardown. `inference_models()` reads the plugin manager's own + already-debounced health state (see `plugin::health`'s startup-grace/ + failure-threshold logic) rather than probing anything itself, so the + short poll interval doesn't add flapping risk beyond that debouncing. + +The tunnel-port fix is an unconditional addition in `run_auto()`'s existing +startup path. The host-role watcher is new (if small) background machinery, +started conditionally (skipped for `--client` nodes, which have no compute to offer regardless of plugin state). Neither threads through the passive-listener/model-selection context that an earlier version of this fix (written against an older fork base, before the @@ -122,16 +106,10 @@ would have been dead weight added to an already-dead code path. ## Deliberately deferred -- **Real ownership tracking for `NodeRole::Host`.** Nothing in `mesh::Node` - currently tracks *why* a node is `Host` — `set_role()` is a plain - overwrite. The plugin-host-role watcher's task-local flag plus a - `models_being_served()` check (see above) closes the one race that's - reachable today, but a real fix would give `NodeRole::Host` (or a - parallel piece of state) an explicit set of claimants — local model - load, plugin health, potentially others later — and only clear the role - when the last claimant releases it. That's a bigger, more invasive - change than this fix's scope; worth doing if a node mixing local models - and plugins becomes a real use case rather than a theoretical one. +- **Additional host-role claim sources.** Local model load and plugin health + now have explicit claims, but future host capabilities may need their own + claim type and lifecycle. Any new claimant must use the same ownership and + re-gossip path so it cannot demote a node still serving for another source. - **Capacity/VRAM-based host ordering.** `order_remote_hosts_by_context` and related host-ranking logic (used when multiple peers can serve the same model) is built around real local VRAM/context-window numbers. A From 31775ec0594fffe5a8ec2e7d21f752e89150989b Mon Sep 17 00:00:00 2001 From: Michael Neale <14976+michaelneale@users.noreply.github.com> Date: Wed, 12 Aug 2026 20:09:14 +1000 Subject: [PATCH 8/8] fix: keep client nodes off the inbound HTTP tunnel path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A --client node has no compute to offer and never advertises Host, so setting its tunnel http_port only had one effect: an inbound QUIC HTTP tunnel from any admitted peer would relay into the local API proxy, find no local model, and route back out to another peer — making the client a mesh-internal request relay. Nothing selects a client that way, but any peer that deliberately dials one could use it. That ingress also carries /mesh/load and /mesh/drop, dispatched ahead of the inference admission check, so this widened exactly the surface issue #1190 exists to narrow. Gate set_http_port with the host-role watcher it already sits beside, so a client stays as reachable as it was before this change — not at all. Plugin-only nodes are never --client, so the case this PR targets is unaffected. Also refresh the design doc: it still named the pre-rename spawn_plugin_host_role_watcher and described the tunnel port as unconditional. Co-authored-by: Mahdi Hedhli <16087011+MahdiHedhli@users.noreply.github.com> Co-authored-by: Michael Neale <14976+michaelneale@users.noreply.github.com> Signed-off-by: Michael Neale <14976+michaelneale@users.noreply.github.com> --- .../src/runtime/run_auto.rs | 37 ++++++++----- .../plugin-hosted-mesh-serving-design.md | 53 +++++++++++++++---- 2 files changed, 66 insertions(+), 24 deletions(-) diff --git a/crates/mesh-llm-host-runtime/src/runtime/run_auto.rs b/crates/mesh-llm-host-runtime/src/runtime/run_auto.rs index 8b35952e1b..793e274ae7 100644 --- a/crates/mesh-llm-host-runtime/src/runtime/run_auto.rs +++ b/crates/mesh-llm-host-runtime/src/runtime/run_auto.rs @@ -1244,20 +1244,31 @@ pub(super) async fn run_auto(ctx: RunAutoContext) -> Result<()> { let tunnel_mgr = tunnel::Manager::start(node.clone(), channels.rpc, channels.http, channels.stage).await?; - // Set unconditionally, not only as a side effect of a local model - // finishing load (see `startup_handles.rs`'s three call sites, which - // remain and are now redundant-but-harmless — same node, same - // `api_port`, for the lifetime of the process). The api proxy this - // points at is already bound and already answers correctly with no - // models loaded, so an inbound tunneled request arriving before any - // model is ready gets a normal "not available" response instead of - // being silently dropped (the previous behavior whenever this was - // still 0 — see `network/tunnel.rs`'s `port == 0` early-return). This - // is what lets a plugin-only node (no local model ever loads) accept - // inbound requests at all; see `spawn_plugin_host_role_watcher` below - // for the other half — whether peers actually route here. - tunnel_mgr.set_http_port(api_port); + // Both halves of inbound reachability are established here for any node + // that can serve, rather than only as a side effect of a local model + // finishing load. + // + // `set_http_port` is what lets a plugin-only node (no local model ever + // loads) accept inbound requests at all: the api proxy it points at is + // already bound and already answers correctly with no models loaded, so a + // tunneled request arriving before any model is ready gets a normal "not + // available" response instead of being silently dropped (the previous + // behavior whenever this was still 0 — see `network/tunnel.rs`'s + // `port == 0` early-return). The three call sites in `startup_handles.rs` + // remain and are now redundant-but-harmless — same node, same `api_port`, + // for the lifetime of the process. + // + // `plugin_host_role::spawn` is the other half: whether peers actually + // route here. + // + // Both are gated on `!is_client`. A client node has no compute to offer + // and never advertises `Host`, so nothing selects it as a route target; + // leaving its inbound HTTP tunnel terminated at the `port == 0` check + // keeps it exactly as reachable as it was before this change — not at + // all — instead of turning it into a mesh-internal request relay for any + // admitted peer that dials it. if !is_client { + tunnel_mgr.set_http_port(api_port); plugin_host_role::spawn(node.clone(), plugin_manager.clone(), api_port); } diff --git a/docs/plugins/plugin-hosted-mesh-serving-design.md b/docs/plugins/plugin-hosted-mesh-serving-design.md index ee1361c90e..108cf32dc6 100644 --- a/docs/plugins/plugin-hosted-mesh-serving-design.md +++ b/docs/plugins/plugin-hosted-mesh-serving-design.md @@ -76,13 +76,13 @@ completing. - **Tunnel port**: since `run_auto()` already unconditionally constructs `tunnel::Manager` and binds the real API proxy before any model resolution happens, set `tunnel_mgr.set_http_port(api_port)` - immediately after construction, unconditionally. The three call sites - in `startup_handles.rs` remain and become redundant-but-harmless (same - node, same `api_port`, for the process's lifetime) rather than - conflicting. -- **Host role**: add `spawn_plugin_host_role_watcher`, a small background - task started alongside the tunnel manager (skipped for `--client` - nodes) that polls `plugin_manager.inference_models()` on a short + immediately after construction, for any node that can serve. The three + call sites in `startup_handles.rs` remain and become + redundant-but-harmless (same node, same `api_port`, for the process's + lifetime) rather than conflicting. +- **Host role**: add `plugin_host_role::spawn`, a small background + task started alongside the tunnel manager that polls + `plugin_manager.inference_models()` on a short interval and claims `NodeRole::Host { http_port: api_port }` when the successful result is non-empty, releasing that claim when it becomes empty again. Errors are logged and leave the previous role state intact @@ -95,15 +95,46 @@ completing. failure-threshold logic) rather than probing anything itself, so the short poll interval doesn't add flapping risk beyond that debouncing. -The tunnel-port fix is an unconditional addition in `run_auto()`'s existing -startup path. The host-role watcher is new (if small) background machinery, -started conditionally (skipped for `--client` nodes, which have no compute to -offer regardless of plugin state). Neither threads through the +The tunnel-port fix is a small addition in `run_auto()`'s existing startup +path. The host-role watcher is new (if small) background machinery. Neither +threads through the passive-listener/model-selection context that an earlier version of this fix (written against an older fork base, before the daemon-lifecycle-reconciliation refactor landed upstream) needed and that would have been dead weight added to an already-dead code path. +## Client nodes + +Both changes are gated behind `!is_client`, together, at the same call site: + +```rust +if !is_client { + tunnel_mgr.set_http_port(api_port); + plugin_host_role::spawn(node.clone(), plugin_manager.clone(), api_port); +} +``` + +A `--client` node has no compute to offer regardless of plugin state, so the +host-role watcher is pointless there. The tunnel port is gated for a separate +and more important reason: setting it is what makes a node's inbound QUIC HTTP +tunnel terminate at the local API proxy instead of at `tunnel.rs`'s +`port == 0` early-return. + +If a client set it, an inbound tunnel from any admitted peer would relay into +the local proxy, find no local model, and route back out to another peer +through `hosts_for_model()` — making the client a mesh-internal request relay. +Nothing would *select* a client that way (clients never advertise `Host`), but +any peer that deliberately dialed one could use it, which on a public mesh +means any member. The same ingress also carries the `/mesh/load` and +`/mesh/drop` control paths, which are dispatched ahead of the inference +admission check; `/mesh/load` is refused for clients by the runtime control +loop, and `/mesh/drop` is inert only because a client has nothing loaded. + +Gating keeps a client exactly as reachable as it was before this change — not +at all — and avoids widening a surface that issue #1190 exists to narrow. +Plugin-only nodes, the case this document is about, are never `--client`, so +the gate does not affect them. + ## Deliberately deferred - **Additional host-role claim sources.** Local model load and plugin health