refactor(frontends): worker endpoint, metrics, and profiling ports on the protocol - #488
Merged
ishandhanani merged 1 commit intoSep 22, 2026
Conversation
… the protocol Four consumers decided which rank serves what by comparing the frontend name, each with slightly different rules: tachometer scrape targets (core/telemetry.py, 9 sites), the benchmark's logical worker endpoints, the AIPerf server metrics URLs, and the profiling control endpoints in benchmark_stage.py, plus the sequential-start readiness port in worker_stage.py. Those become frontend members: metrics_path, worker_metrics_port, worker_endpoint_port, profiling_control_port, profiling_control_is_leader_only, direct_endpoint_nodes, and worker_ready_port, implemented per class from the rules each site encoded. No frontend-name comparison remains in telemetry.py or the benchmark stage; BenchmarkStageMixin.frontend is None for a services-only job. Behavior changes, all corrections of dead ports the old per-site rules produced: the trtllm_serve aggregate worker's logical endpoint is the public port it binds rather than its unbound http_port; the direct sglang nsys control endpoint is the leader's public port rather than http_port; built-in AIPerf metrics for sglang-router target the worker leaders' HTTP ports rather than Dynamo system ports no sglang worker serves; and direct frontends no longer return early from the metrics env, so KVBM and CPU power exporter URLs are appended for them like every other frontend. The vLLM direct telemetry test fake gains the frontend_port it relies on. Tests: a per-frontend contract test over a four-process topology for every new member, and the Dynamo sidecar case. Dry-run output for all 26 example recipes is unchanged. Signed-off-by: Ishan Dhanani <idhanani@nvidia.com>
ishandhanani
requested review from
alec-flowers,
csahithi and
nlevin-ui
as code owners
September 22, 2026 00:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #487. Four consumers decided which rank serves what by comparing the frontend name, each with slightly different rules: the tachometer scrape targets in
core/telemetry.py(9 sites), the benchmark's logical worker endpoints, the AIPerf server metrics URLs, and the profiling control endpoints inbenchmark_stage.py, plus the sequential-start readiness port inworker_stage.py. Those questions move onto the frontend protocol asmetrics_path,worker_metrics_port,worker_endpoint_port,profiling_control_port,profiling_control_is_leader_only,direct_endpoint_nodes, andworker_ready_port, implemented per class from the rules each site encoded. No frontend-name comparison remains intelemetry.pyor the benchmark stage.BenchmarkStageMixin.frontendisNonefor a services-only job.Behavior changes
All four are corrections of dead ports the old per-site rules produced; none affects a shipped example (dry-run parity below).
http_port.http_port.Validation
tests/test_frontends.py(a four-process topology per frontend covering every new member, and the Dynamo sidecar case).srtctl dry-runfor all 26 example recipes is byte-identical to main after normalizing generated timestamps.frontend_portthe direct vLLM worker actually binds.Review map
frontends/base.py: the seven members andagg_leader_nodes.frontends/*.py: per-class rules.vllm_router.pyoverrides the two per-pool ports.core/telemetry.py: the per-process loop and the frontend-node selection read the frontend.cli/mixins/benchmark_stage.py:frontendproperty,_public_api_node,_logical_worker_endpoints,_profiling_worker_endpoints,_get_aiperf_server_metrics_env.cli/mixins/worker_stage.py:_wait_for_worker_ready.CLAUDE.md: Frontends section lists the members.Still keyed on the frontend name after this PR, for the next one:
_get_health_expectationsand the special cases inwait_for_model, the dry-run panels insubmit.py, the sglang-router metrics listener intelemetry_stage.py, and the Dynamo request-tracing checks.