Add mode-aware management health endpoint

[i386]

Summary

• add a lightweight GET /health management endpoint that always returns HTTP 200/status: ok when the management API can answer
• expose advisory mesh connectivity (standalone, connected, or disconnected) plus Worker, Client, and Serving mode health without active probes or network refreshes
• report healthy local models/stages, cached plugin models, and starting/degraded/unhealthy serving states
• document the stable response contract and cover headless, connectivity, mode, model, plugin, and failure cases

Simulated responses

These are representative examples. Both return HTTP 200 OK because /health is a management-process liveness endpoint; the nested fields carry mesh and model-serving readiness.

Healthy serving node

{
  "status": "ok",
  "mode": "serving",
  "mesh": {
    "status": "connected",
    "admitted_peer_count": 2,
    "connected_peer_count": 2
  },
  "serving": {
    "status": "healthy",
    "models": [
      "Qwen3-8B-Q4_K_M",
      "Llama-3.2-3B-Instruct-Q4_K_M"
    ]
  }
}

Non-healthy worker node

This simulates an admitted mesh member whose live control connection has dropped and whose local split stage has failed.

{
  "status": "ok",
  "mode": "worker",
  "mesh": {
    "status": "disconnected",
    "admitted_peer_count": 1,
    "connected_peer_count": 0
  },
  "serving": {
    "status": "unhealthy",
    "models": []
  }
}

Validation

• just build
• just website-build
• just with-lld cargo fmt --all -- --check
• just with-lld cargo test -p mesh-llm-host-runtime health --lib (50 passed)
• just with-lld cargo test -p mesh-llm-host-runtime --lib (2,483 passed, 8 ignored)
• just with-lld cargo clippy -p mesh-llm-host-runtime --lib

Closes #1349

Summary by CodeRabbit

• New Features

  • Added a lightweight GET /health endpoint returning HTTP 200 when the management process is reachable.
  • Reports node mode, mesh admission and connection counts, serving readiness, and available models.
  • Supports client, serving, and worker modes.

• Documentation

  • Documented endpoint behavior, response fields, readiness semantics, and distinction from the full status API.

• Tests

  • Added coverage for health states, model readiness, mesh connectivity, and failure scenarios.

[github-actions]

This pull request is currently a draft. Reviews will not take place until the PR is marked as ready for review.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: b4632305-cbd9-4e2f-80e0-b08ded881e45

📥 Commits

Reviewing files that changed from the base of the PR and between ba70e72 and 2f8649a.

📒 Files selected for processing (1)

• crates/mesh-llm-host-runtime/src/api/routes/health.rs

🚧 Files skipped from review as they are similar to previous changes (1)

• crates/mesh-llm-host-runtime/src/api/routes/health.rs

Included review availability: Your plan includes up to 8 reviews per rolling hour; 5 remain after this review.

---

📝 Walkthrough

Walkthrough

Added a cache-based GET /health management endpoint. It returns HTTP 200 with liveness, node mode, mesh connectivity, and serving status data. Tests cover headless operation, runtime states, failures, model filtering, and cached plugin models. Documentation describes the endpoint.

Changes

Management health endpoint

Layer / File(s)	Summary
Health data and connectivity crates/mesh-llm-host-runtime/src/mesh/connectivity.rs, crates/mesh-llm-host-runtime/src/mesh/mod.rs, crates/mesh-llm-host-runtime/src/runtime/dashboard.rs, crates/mesh-llm-host-runtime/src/runtime/mod.rs, crates/mesh-llm-host-runtime/src/mesh/tests/peer_state.rs, crates/mesh-llm-host-runtime/src/api/routes/health.rs	Added health response types, cached mesh connectivity counts, and crate-visible runtime status mapping.
Health route and status classification crates/mesh-llm-host-runtime/src/api/routes/mod.rs, crates/mesh-llm-host-runtime/src/api/routes/health.rs, crates/mesh-llm-host-runtime/src/api/tests/management_http.rs	Added GET /health dispatch and cache-based mode and serving-state computation. Tests cover liveness, headless mode, node roles, readiness, failures, filtering, and plugin models.
Health endpoint documentation crates/mesh-llm-host-runtime/src/api/mod.rs, docs/MESHES.md, docs/design/DESIGN.md, website/src/docs/pages/api-reference.md	Documented the endpoint response, status values, liveness behavior, and distinction from /api/status.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: ⚪ Minimal · up to 2f864

The change adds a management health endpoint with documented behavior and reported passing validation; no actionable merge-blocking risk remains beyond normal checks and review.

Sequence Diagram(s)

sequenceDiagram
  participant ManagementClient
  participant APIRouter
  participant health_handle as health::handle
  participant Node
  participant RuntimeCaches
  ManagementClient->>APIRouter: GET /health
  APIRouter->>health_handle: dispatch request
  health_handle->>Node: read connectivity_snapshot
  Node-->>health_handle: peer counts
  health_handle->>RuntimeCaches: read cached runtime data
  RuntimeCaches-->>health_handle: mode and serving data
  health_handle-->>ManagementClient: HTTP 200 JSON response

Loading

Possibly related PRs

• Mesh-LLM/mesh-llm#871: Adds a related GET /health endpoint in another service.
• Mesh-LLM/mesh-llm#1148: Provides runtime serving-status infrastructure consumed by this endpoint.
• Mesh-LLM/mesh-llm#1192: Changes mesh node role and plugin-serving state handling reported by this endpoint.

Suggested reviewers: ndizazzo

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the new mode-aware management health endpoint, which is the pull request's primary change.
Linked Issues check	✅ Passed	The implementation, tests, and documentation satisfy the linked issue's requirements for a lightweight, headless-capable liveness endpoint.
Out of Scope Changes check	✅ Passed	All changes support the health endpoint, including routing, status data, tests, runtime mapping, connectivity support, and documentation.
Docstring Coverage	✅ Passed	Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent/management-health-endpoint

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[michaelneale]

Review: #1351 — mode-aware management health endpoint

Overall: the shape is right — GET /health as a liveness endpoint that always
200s, with advisory nested readiness, cache-only (no probes, no network
refresh), available in headless mode, and documented in three places. Routing,
docs, and the connectivity split (membership ≠ connectivity) all look correct.
All six CI checks are green.

My concerns are concentrated in one place: the process-state strings in
local_serving_state were not derived from what the runtime actually emits.

1. The has_failure string set is partly unreachable, and partly wrong (blocking-ish)

health.rs:150,157:

.filter(|process| matches!(process.state.as_str(), "ready" | "serving"))
...
matches!(process.state.as_str(), "error" | "exited" | "failed" | "stopped")

I grepped every non-test writer of RuntimeProcessSnapshot.state /
RuntimeProcessPayload.status in the workspace. The complete production set is:

string	writer
ready	runtime/local.rs:937 (local_process_snapshot)
shutting down	model_lifecycle.rs:272, model_lifecycle/unload.rs:212,221, startup_handles.rs:981
stopped	model_lifecycle.rs:291 (shutdown_runtime_loaded_models only)
exited	model_lifecycle/unload.rs:328,332 (run_auto_handle_runtime_exit)

So:

• "failed" and "error" are never emitted for a local process. The only
  status: "error" writers in the tree are plugin/mod.rs:321 and
  plugin/installed.rs:223 (plugin summaries, different field). "failed" for
  a process only exists as InstanceLifecycleState::Failed.as_str()
  (instance_lifecycle.rs:82), which populates lifecycle_instances, not
  local_processes. That means
  management_health_reports_failed_host_process_as_unhealthy asserts on a
  string the runtime never produces — the test passes without covering the real
  failure path. The real one to assert is "exited".
• "serving" in the healthy filter is likewise dead — same reason
  (InstanceLifecycleState::Serving, not a process state).
• "stopped" is a clean state, not a failure. It is emitted only by
  shutdown_runtime_loaded_models (control_loop.rs:387), which upserts
  stopped and does not remove the row. Combined with has_work being true
  for any non-empty local_processes, a node in normal shutdown reports
  serving.status: "unhealthy". For a probe wired to alerting that is a false
  alarm on every graceful stop.
• "shutting down" matches neither arm, so a draining single-model node
  reports starting. Probably harmless, but it's an accident rather than a
  decision.

Suggested fix: don't hand-maintain a string set here. runtime/dashboard.rs:349
already has runtime_status_from_process_status mapping exactly these strings to
the RuntimeStatus enum, and it is currently #[allow(dead_code)]. Match on
that enum instead, and the compiler starts carrying the invariant. If that's too
much surface for this PR, at minimum: drop "failed"/"error"/"serving",
drop "stopped" from the failure set, decide explicitly about
"shutting down", and retarget the failed-host-process test at "exited".

2. Mode is exclusive, so a serving node's split stages are invisible

health_mode returns one of three, and local_stage_statuses is only fetched
if matches!(mode, HealthMode::Worker) (health.rs:~110). A NodeRole::Host
node that also owns a local split stage reports mode: serving and never looks
at stage_runtime_statuses(), so a StageRuntimeState::Failed stage on a
serving host yields serving.status: healthy. If that combination can't happen,
say so in the doc comment; otherwise consider collecting both sources
regardless of mode and letting mode be presentation only.

3. cached_plugin_models degrades a serving host to idle, not degraded

inference_models().await.unwrap_or_default() — the comment says "fail closed",
but the effect is that an unavailable cached plugin inventory on a
plugin-only serving host produces models: [] → idle. That reads as "nothing
configured" rather than "I can't tell". A probe cannot distinguish the two.

4. Minor

• mesh.status: "disconnected" for a healthy standalone single-node mesh is
  technically true but will read as a fault in dashboards. standalone (or
  documenting it more loudly) would be kinder. It is documented as advisory,
  so not blocking.
• /health is not in requires_trusted_local_access (api/access.rs:3), so
  under --listen-all (server.rs:146) it discloses model names and peer
  counts to the LAN unauthenticated. This matches the existing posture of
  /api/status, so it's not a regression — but it's a new, deliberately
  cheap-to-poll surface, so worth a conscious yes.
• openai-frontend already serves /health and /healthz
  (router.rs:206-207) on the serving port. Two different /health endpoints
  on two ports will confuse operators; consider a /healthz alias here for
  symmetry, and a line in the docs distinguishing the two.
• connectivity_snapshot walks state.peers twice; one pass computing both
  counters would be equivalent and cheaper. Cosmetic.
• eligible_management_route (management_lifecycle.rs:51) excludes /health
  because it isn't under /api/, so it won't pollute the workload ledger —
  correct and probably intentional; logging/request_metadata.rs:57 already has
  the health label. Worth one sentence in the module doc so a future move under
  /api/ doesn't silently start recording probe traffic.

Verification I did / did not do

Source read at PR head 2bf0fb3d, fetched as pr1351. I did not build or
run the tests locally, and I did not curl a live node. All claims above are
source-grep claims scoped to crates/ in this checkout; the "never emitted"
claims in §1 come from grepping every non-test writer of those two fields and
could be wrong if a state string is constructed dynamically somewhere I didn't
match on a literal.

[i386]

@michaelneale Thanks for the detailed source-level review — addressed in 9063d3acb.

Changes made:

• Reused the runtime's canonical runtime_status_from_process_status mapping instead of maintaining a second string vocabulary.
• Retargeted the failure coverage to the real unexpected-exit state, exited.
• Treat shutting down and stopped as graceful inactive states, so they report idle rather than starting or unhealthy.
• Collect cached local stage state for serving hosts as well as workers; a failed host-owned stage now contributes degraded/unhealthy.
• Preserve cached plugin inventory read failures so plugin-only hosts cannot silently fall back to idle.
• Added mesh.status: "standalone" for nodes with no admitted peers; disconnected now specifically means admitted membership without a current connection.
• Reduced the connectivity snapshot to one pass over peer state.
• Documented that the management endpoint is remotely readable like /api/status, discloses model names/peer counts, and is excluded from the management workload ledger.
• Documented the management-port endpoint separately from the OpenAI serving port's existing /health, /healthz, and /readyz. I kept the management contract at the issue-requested /health rather than adding another alias.

Validation:

• just with-lld cargo fmt --all -- --check
• just with-lld cargo test -p mesh-llm-host-runtime health --lib — 50 passed
• just with-lld cargo test -p mesh-llm-host-runtime --lib — 2,483 passed, 8 ignored
• just with-lld cargo clippy -p mesh-llm-host-runtime --lib
• just build
• just website-build

[i386]

Follow-up verification after the branch was synchronized with latest main:

• Exact PR head: ba70e7256ca60a00b804f2c0b5c860b0004903d6
• Review-fix commit remains included: 9063d3acb
• just with-lld cargo fmt --all -- --check passed
• just with-lld cargo test -p mesh-llm-host-runtime health --lib passed: 50 tests
• just with-lld cargo test -p mesh-llm-host-runtime --lib passed: 2,509 passed, 8 ignored, 0 failed
• GitHub CI is fully terminal and green for all selected jobs; only expected planner skips remain
• CodeRabbit reports no new actionable comments, and there are no unresolved review threads

The local worktree is clean and matches the pushed PR head.

[michaelneale]

good to get in when green