fix(trtllm): export KV event hosts for MPI endpoints - #402
Merged
ishandhanani merged 1 commit intoSep 10, 2026
Merged
Conversation
GuanLuo
requested review from
alec-flowers,
csahithi,
ishandhanani and
nlevin-ui
as code owners
September 7, 2026 06:25
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #402 +/- ##
=======================================
Coverage ? 73.49%
=======================================
Files ? 101
Lines ? 14053
Branches ? 0
=======================================
Hits ? 10328
Misses ? 3725
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
ishandhanani
approved these changes
Sep 10, 2026
ishandhanani
added a commit
that referenced
this pull request
Sep 13, 2026
Nine conflicting files, resolved by hand: - The process exporter (#413) and the wider node-exporter collector set (#415) arrive as tachometer-stage exporter launches; on this branch the exporters are services. process-exporter is now a service kind (`type: process-exporter`, implied while tachometer runs, `placement.node: all`), host-native from the `configs/process-exporter` binary `make setup` installs (no container, no mounts), skipped with a warning when the binary is missing, container launch when a recipe declares a `container`. The group file is written by the kind's `prepare` hook; `ServiceKind` gains `host_native`, `prepare`, and `skip_reason`. node-exporter's built command carries the stat/vmstat/pressure/meminfo_numa/processes collectors and the widened vmstat field filter. The templates, the group YAML, and the host binary resolver move to `srtctl.services.exporters`; their tests follow. - Load-window tachometer (#359): start/stop inside run_benchmark is kept; `stop_tachometer` now terminates through `ManagedProcess.terminate`, which signals the Slurm step (SIGTERM to the srun client would abort the step and SIGKILL the scraper). The scraper's `terminate_timeout` is the recipe's `shutdown_grace_secs`; the 90 s module constant is gone. - Ingest timestamp fallback (#414) is taken from main wholesale (it also offers `--start-ns`); this branch's own fallback is dropped, its dedup module and docstring edits re-applied. - CPU power telemetry (#410, #422): taken as is (power is frozen); the energy report runs before the S3 upload, which returns the URL only. - Makefile: the `examples`/`golden-check` targets plus the cpu-power targets; the `recipes/`-based runner targets stay deleted. SUMMARY: both new pages, `analyzing.md` stays deleted. submit.py: the direct-host renderer import stays deleted; the arch helpers the cpu-power preflight uses are kept. telemetry.py: IPv6-safe host and the SGLang gateway metrics port together. 2355 tests on Python 3.10 and 3.13, lint, schema docs, 21 examples validated, golden 574 identical / 0 mismatched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DYN_TRTLLM_KV_EVENT_HOSTSfrom the endpoint node list for multi-node TRT-LLM endpoints that enable native KV-event publishingWhy
Dynamo native TRT-LLM KV-event subscribers require either
DYN_TRTLLM_KV_EVENT_HOSTSorSLURM_STEP_NODELIST. The latter was absent in the Dynamo process in a Pyxis container launch, so multi-node workers could not start their direct subscribers.Validation
uv run pytest tests/test_slurm.py -quv run ruff check src/srtctl/cli/mixins/worker_stage.py tests/test_slurm.py