fix(health): skip canary health check for trtllm/vllm disagg decode workers

[nnshah1]

Summary

• TRT-LLM: Skip canary health check payload registration for disagg decode workers
• vLLM: Skip canary health check payload for multimodal disagg decode workers (text-only decode still gets canary)
• SGLang: Already correct (uses FAKE_BOOTSTRAP_HOST for disagg decode health checks)

Motivation

Enabling DYN_HEALTH_CHECK_ENABLED=true on disagg decode workers causes a crash-loop (DIS-1737). The canary sends a standard inference request, but decode workers reject it with "Disaggregated params are required for decode mode." The canary has no disagg topology awareness.

SGLang solved this with FAKE_BOOTSTRAP_HOST (fake-transfer mode). TRT-LLM and vLLM have no equivalent — TRT-LLM decode requires real opaque_state from prefill, vLLM multimodal decode needs embedding_params.

This PR fixes what we can now. TRT-LLM decode fake-transfer mode is being discussed with the TRT-LLM team separately.

Changes

• trtllm/workers/llm_worker.py: Gate health_check_payload on disaggregation_mode != DECODE
• vllm/worker_factory.py: Gate on not (DECODE and MULTIMODAL)
• trtllm/tests/test_health_check_disagg.py: Test verifying standard payload has no disagg params

Test plan

• TRT-LLM decode: no health check payload registered (no crash)
• TRT-LLM prefill/agg: standard health check payload (unchanged)
• vLLM multimodal decode: no health check payload (no crash)
• vLLM text-only decode: standard health check payload (unchanged)
• SGLang: validated existing disagg path is correct
• e2e validation with real backends in k8s

Related

• DIS-1737, DIS-1738, DYN-2670
• PR fix(health): make canary sole authority on endpoint readiness when enabled #8165 (DIS-1185 canary race condition fix — prerequisite)

🤖 Generated with Claude Code

Summary by CodeRabbit

• Tests

  • Added validation test for health check payload structure in specific scenarios

• Improvements

  • Optimized health check behavior for decode workers in disaggregation mode, conditionally skipping canary checks based on model type and runtime configuration

---

[coderabbitai]

Walkthrough

The changes implement conditional health check payload handling for decode workers across TrtLLM and vLLM frameworks. When disaggregation mode is DECODE, health check payloads are conditionally skipped (and logged). An additional condition for vLLM checks for MULTIMODAL model type. A test validates the health check payload structure excludes certain fields.

Changes

Cohort / File(s)	Summary
Health Check Payload Tests components/src/dynamo/trtllm/tests/test_health_check_disagg.py	Added test for TrtllmHealthCheckPayload verifying that converted payload excludes "disaggregated_params" and "prefill_result" while including "token_ids".
TrtLLM Worker components/src/dynamo/trtllm/workers/llm_worker.py	Conditionally sets health_check_payload to None during init_llm_worker when config.disaggregation_mode == DisaggregationMode.DECODE, with logging; otherwise constructs TrtllmHealthCheckPayload(tokenizer=tokenizer).to_dict(). Updated payload is passed to endpoint.serve_endpoint(...).
vLLM Worker Factory components/src/dynamo/vllm/worker_factory.py	Conditionally sets health_check_payload to None for decode workers when both config.disaggregation_mode == DECODE and model_type == MULTIMODAL, with logging; otherwise computes payload via VllmHealthCheckPayload(...).to_dict(). Updated payload is passed to generate_endpoint.serve_endpoint(...).

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Title check	✅ Passed	The title accurately and concisely summarizes the primary change: skipping canary health checks for disaggregated decode workers in both trtllm and vllm frameworks.
Description check	✅ Passed	The pull request description provides comprehensive details covering overview, motivation, changes, and test plan, with clear section structure matching the template requirements.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[nnshah1]

Update — significant rework since initial submission

The fix evolved substantially over the last round of iteration. Summary of the current state (HEAD 98452dfd3f) for reviewers:

Per-backend canary policy for disagg decode:

Backend	Decode canary mechanism
vllm	Standard VllmHealthCheckPayload — DecodeWorkerHandler._generate_token_mode already handles prefill_result=None by running agg-style locally.
sglang	SglangDisaggHealthCheckPayload with FAKE_BOOTSTRAP_HOST (pre-existing, prod-proven).
trtllm (new)	TrtllmDisaggDecodeHealthCheckPayload sets a _canary_probe marker; _setup_disaggregated_params_for_mode short-circuits to LlmDisaggregatedParams(request_type="context_and_generation"). Engine runs the probe as full local prefill+decode; cache transceiver activates only for "generation_only" and stays out of this path.

Rust-side fix in lib/runtime/src/system_health.rs:

• set_endpoint_registered now auto-Readies endpoints with no registered canary target (DIS-1185 contract preserved for endpoints that opt in).
• get_health_status path ci: update container registry location #3 consults endpoint_health when no targets are registered.
• 5 new Rust unit tests.

Local end-to-end verification (GPU on an A6000):

• Launched disagg_same_gpu.sh with DYN_HEALTH_CHECK_ENABLED=true; both prefill and decode workers became Ready via their canary probes.
• SIGSTOP on decode's engine rank (mpi4py.futures.server) → /health flipped 503 in 12 s (canary detected the hang).
• SIGCONT → /health recovered to 200 in 8 s.

This is the correctness gate — proves the new probe actually reaches the engine (not a Python-layer bypass) and that canary detects real engine-level hangs for disagg decode, matching sglang.

New fault-tolerance test harness at tests/fault_tolerance/test_canary_rank_pause.py:

• Parameterized across {trtllm, vllm, sglang} × {agg, disagg-prefill, disagg-decode} — 9 positive scenarios (expected detect).
• One negative control (trtllm-agg-canary-off, expected miss) to prove the harness distinguishes both outcomes rather than always reporting success.
• @nightly + @gpu_1; doesn't gate pre-merge.

CI status: trtllm-runtime / Test cuda13.1, amd64 is now green (was the original blocker). Remaining failure is sglang-runtime / Test cuda13.0, amd64 :: test_sql_tool_arguments_schema_valid, unrelated to this PR.

Happy to walk through any piece in detail.

[nnshah1]

Superseded by #8521, which ships a cleaner shape of the same DIS-1737 fix:

• The Rust system_health.rs changes are split off (future separate PR)
• The canary payload carries explicit top-level disaggregated_params instead of a _canary_probe marker — self-describing, no magic sentinel
• Reworked test harness with port-aware rank discovery and @pytest.mark.model (saves ~30 GB of unnecessary model downloads)
• Local E2E verified 6/6 cells: trtllm × {agg, disagg-prefill, disagg-decode} + vllm × same matrix