Skip to content

refactor(frontends): worker endpoint, metrics, and profiling ports on the protocol - #488

Merged
ishandhanani merged 1 commit into
idhanani/frontend-protocolfrom
idhanani/frontend-endpoints
Sep 22, 2026
Merged

ishandhanani merged 1 commit into
idhanani/frontend-protocolfrom
idhanani/frontend-endpoints

Conversation

@ishandhanani

Copy link
Copy Markdown
Collaborator

Summary

Stacked on #487. Four consumers decided which rank serves what by comparing the frontend name, each with slightly different rules: the tachometer scrape targets in core/telemetry.py (9 sites), the benchmark's logical worker endpoints, the AIPerf server metrics URLs, and the profiling control endpoints in benchmark_stage.py, plus the sequential-start readiness port in worker_stage.py. Those questions move onto the frontend protocol as metrics_path, worker_metrics_port, worker_endpoint_port, profiling_control_port, profiling_control_is_leader_only, direct_endpoint_nodes, and worker_ready_port, implemented per class from the rules each site encoded. No frontend-name comparison remains in telemetry.py or the benchmark stage. BenchmarkStageMixin.frontend is None for a services-only job.

Behavior changes

All four are corrections of dead ports the old per-site rules produced; none affects a shipped example (dry-run parity below).

  • The trtllm_serve aggregate worker's logical endpoint is the public port it binds, not its unbound http_port.
  • The direct sglang nsys control endpoint is the leader's public port, not http_port.
  • Built-in AIPerf metrics for sglang-router target the worker leaders' HTTP ports; before, they targeted Dynamo system ports that no sglang worker serves.
  • Direct frontends no longer return early from the metrics env, so KVBM and CPU power exporter URLs are appended for them like every other frontend.

Validation

  • Full suite: 2972 passed on Python 3.13 and 3.10, plus 8 new contract tests in tests/test_frontends.py (a four-process topology per frontend covering every new member, and the Dynamo sidecar case).
  • srtctl dry-run for all 26 example recipes is byte-identical to main after normalizing generated timestamps.
  • One test fake gained the frontend_port the direct vLLM worker actually binds.

Review map

  • frontends/base.py: the seven members and agg_leader_nodes.
  • frontends/*.py: per-class rules. vllm_router.py overrides the two per-pool ports.
  • core/telemetry.py: the per-process loop and the frontend-node selection read the frontend.
  • cli/mixins/benchmark_stage.py: frontend property, _public_api_node, _logical_worker_endpoints, _profiling_worker_endpoints, _get_aiperf_server_metrics_env.
  • cli/mixins/worker_stage.py: _wait_for_worker_ready.
  • CLAUDE.md: Frontends section lists the members.

Still keyed on the frontend name after this PR, for the next one: _get_health_expectations and the special cases in wait_for_model, the dry-run panels in submit.py, the sglang-router metrics listener in telemetry_stage.py, and the Dynamo request-tracing checks.

… the protocol

Four consumers decided which rank serves what by comparing the frontend
name, each with slightly different rules: tachometer scrape targets
(core/telemetry.py, 9 sites), the benchmark's logical worker endpoints,
the AIPerf server metrics URLs, and the profiling control endpoints in
benchmark_stage.py, plus the sequential-start readiness port in
worker_stage.py. Those become frontend members: metrics_path,
worker_metrics_port, worker_endpoint_port, profiling_control_port,
profiling_control_is_leader_only, direct_endpoint_nodes, and
worker_ready_port, implemented per class from the rules each site
encoded. No frontend-name comparison remains in telemetry.py or the
benchmark stage; BenchmarkStageMixin.frontend is None for a
services-only job.

Behavior changes, all corrections of dead ports the old per-site rules
produced: the trtllm_serve aggregate worker's logical endpoint is the
public port it binds rather than its unbound http_port; the direct sglang
nsys control endpoint is the leader's public port rather than http_port;
built-in AIPerf metrics for sglang-router target the worker leaders' HTTP
ports rather than Dynamo system ports no sglang worker serves; and direct
frontends no longer return early from the metrics env, so KVBM and CPU
power exporter URLs are appended for them like every other frontend.
The vLLM direct telemetry test fake gains the frontend_port it relies on.

Tests: a per-frontend contract test over a four-process topology for
every new member, and the Dynamo sidecar case. Dry-run output for all 26
example recipes is unchanged.

Signed-off-by: Ishan Dhanani <idhanani@nvidia.com>
@ishandhanani
ishandhanani merged commit 6cbe13f into idhanani/frontend-protocol Sep 22, 2026
3 checks passed
@ishandhanani
ishandhanani deleted the idhanani/frontend-endpoints branch September 22, 2026 00:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant