fix(health): backport #8294 to release/1.1.0 — sglang decode canary instance_id-not-found

[nnshah1]

Backport #8294 to release/1.1.0 — fix sglang decode worker canary instance_id not found

Cherry-picks upstream commit c2ec33597f (PR #8294, fix(health): use local endpoint registry for canary health checks, merged 2026-04-17) onto release/1.1.0.

Fixes DYN-2913 (Urgent, SLA breach 2026-04-29) — sglang PD decode worker canary fails with instance_id not found for endpoint <ns>/backend/generate, decode /health returns 503, liveness probe restarts the pod.

See #8294 for the full upstream rationale and design discussion. The change replaces the canary's etcd-discovery path in health_check.rs with a direct in-process lookup in DistributedRuntime::local_endpoint_registry() — no etcd, no NATS, no watch propagation, so the discovery race is structurally eliminated.

Backport adaptation

Dropped the fail-fast guard upstream added to EndpointConfigBuilder in component/endpoint.rs. It referenced SystemHealth::health_check_enabled(), which doesn't exist on release/1.1.0. The guard is informational only — Python serve_endpoint calls register_local_engine unconditionally for every endpoint (lib/bindings/python/rust/lib.rs:882), so the precondition the guard checks is satisfied without it.

Net diff vs. upstream cherry-pick: just the endpoint.rs guard hunk dropped. health_check.rs matches upstream byte-for-byte.

Diff

lib/runtime/src/health_check.rs only — 1 file, +21 / −111.

Verification

• cargo check -p dynamo-runtime --locked ✅
• cargo clippy -p dynamo-runtime --locked --no-deps -- -D warnings ✅
• cargo fmt --check ✅
• Local sglang PD disagg sanity check (Qwen/Qwen3-0.6B, 1×A6000) confirms post-fix decode /health returns 200 with generate: ready. The K8s repro from DYN-2913 is the authoritative bug evidence; the discovery race doesn't surface on a single host.
• cargo test --features integration health_check:: has pre-existing breakage on release/1.1.0 (Instance struct missing device_type initializers in test code, unrelated to this fix). Verified by checking out origin/release/1.1.0 HEAD without the cherry-pick — same compile errors.

Risk

Low. register_local_engine was already merged onto release/1.1.0 and is invoked unconditionally from the Python serve_endpoint binding, so the new code path is reachable for every existing caller. The change is a substitution at one call site within the canary loop; no API surface or behavior shifts for non-canary code.