Skip to content

refactor(frontends): the frontend class owns registration, validation, worker shape, and endpoints - #487

Merged
ishandhanani merged 7 commits into
mainfrom
idhanani/frontend-protocol
Sep 22, 2026
Merged

ishandhanani merged 7 commits into
mainfrom
idhanani/frontend-protocol

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adding a frontend meant editing a Literal, an if-chain, a schema pairing map, four per-frontend schema validators, eleven frontend_type == "..." comparisons across the three backends, nine more in the telemetry target builder, a dozen in the benchmark stage, and three special cases inside the readiness loop. This PR makes the frontend class the owner of every one of those decisions, so a new router is one registered module. It is the groundwork #432 (vLLM MoRI-IO) will be rewritten onto.

Seven commits, each behavior-preserving except where noted:

  1. Registry, required_backend, validate(config). @register_frontend("<type>") on each implementation, imported from the package __init__, replaces the FrontendType literal and the if-chain in get_frontend. Each frontend declares required_backend and its recipe rules in validate(config). SrtConfig has one _validate_frontend: type registered, backend pairing generic, the frontend's ValueError reported as a load-time ValidationError. The four per-frontend schema validators move into their classes with their messages unchanged.

  2. Worker shape. worker_launch (dynamo or direct), worker_api_port(mode) (public or allocated), and expands_node_local_dp on the protocol. Backends resolve frontend_type through get_frontend and read those; no frontend-name comparison remains in backends/vllm.py, sglang.py, trtllm.py, or the nsys prefix builder.

  3. Endpoint, metrics, and profiling ports. metrics_path, worker_metrics_port, worker_endpoint_port, profiling_control_port, profiling_control_is_leader_only, direct_endpoint_nodes, worker_ready_port. Tachometer scrape targets, the benchmark's logical endpoints and public API node, AIPerf metrics URLs, profiling control, and the sequential-start readiness port read them. Four behavior changes, each replacing a dead port the old per-site rules produced: the trtllm_serve aggregate endpoint is the public port it binds rather than its unbound http_port; the direct sglang nsys control port is the leader's public port rather than http_port; built-in AIPerf metrics for sglang-router target the worker leaders' HTTP ports rather than Dynamo system ports no sglang worker serves; direct frontends no longer return early from the metrics env, so KVBM and CPU power exporter URLs are appended for them too.

  4. DynamicFrontend base. The counterpart of StaticRouterFrontend for frontends whose workers register themselves over a discovery plane. Dynamo is its only implementation; vLLM Router's ZMQ discovery mode stays a mode of a static router.

  5. Readiness on the protocol. probe_ready(host, port, n_prefill, n_decode) performs one readiness check and returns a WorkerHealthResult; wait_for_model keeps only timing, abort, and logging, with no frontend names. core/health.py offers the three probe shapes: probe_json_health (a JSON count through parse_health), probe_http_ok (a bare 200, trtllm_serve), probe_direct_server (/health then /v1/models, direct vllm and sglang). health_expectations(config, processes) returns the counts in the frontend's own units; Dynamo's vLLM DP registration logic moves into frontends/dynamo.py, vLLM Router's expansion sum into frontends/vllm_router.py, and _get_health_expectations delegates with its signature unchanged. A router in a discovery mode overrides probe_ready and touches nothing else.

  6. No del of unused arguments. Review comment: ruff's unused-argument rules are off here, so the 49 del name lines in the frontends were noise. Removed, with the rule noted in CLAUDE.md.

  7. Protocol slimmed to what consumers call; implied_services and frontend_metrics_port. health_endpoint, parse_health, and get_frontend_args_list leave the protocol (the two bases keep them as their own hooks; the direct frontends drop never-called stubs; the dead check_trtllm_serve_health goes; one verbatim-args helper frontend_args_to_cli). implied_services(config) moves Dynamo's etcd/NATS rule from services/implicit.py onto DynamoFrontend, so a future registration-based frontend declares its own discovery plane. frontend_metrics_port(frontend_args) moves the SGLang gateway's separate Prometheus listener out of the telemetry stage. Two schema checks that re-derived frontend facts by name (profiler control on the leader only, Dynamo system ports) read profiling_control_is_leader_only and worker_launch. docs/architecture.md describes the protocol as it is now.

Validation

  • Full suite: 2988 passed on Python 3.13 and 3.10 after the final commit (2972 on main plus 16 new tests across tests/test_frontends.py and tests/test_health.py: registry contents, registering a toy frontend, unknown type rejected at load, generic pairing per frontend, contract tests for the worker-shape and port members, the Dynamo sidecar case, the DynamicFrontend defaults, every frontend's probe against mocked HTTP, and the wait_for_model loop).
  • srtctl dry-run for all 26 example recipes is byte-identical to main after normalizing generated timestamps, after every commit.

Review map

  • frontends/base.py: the protocol members, the registry, logical_health_expectations, agg_leader_nodes.
  • frontends/dynamic_frontend.py, frontends/static_router.py: the two bases.
  • frontends/*.py: decorator, required_backend, validate, worker shape, port rules, probe, and expectations per class.
  • core/health.py: wait_for_model reduced to the loop; the three probe helpers.
  • core/schema.py: _validate_frontend replaces four validators; the nsys prefix reads worker_launch.
  • backends/vllm.py, sglang.py, trtllm.py: name comparisons become attribute reads.
  • core/telemetry.py, cli/mixins/benchmark_stage.py, cli/mixins/worker_stage.py: consumers read the frontend.
  • CLAUDE.md: Frontends section and the mode-versus-type checklist describe the registry, the two bases, and the members.

The frontend-name checks left outside frontends/ are Dynamo-config and SGLang-gateway feature switches (sidecar, failover, and worker_selection validators, installs_dynamo, request tracing, slow_down, dry-run panels): each asks whether the recipe uses that product's feature block, not a topology or readiness question.

… on the protocol

frontend.type now resolves through a registry populated by
@register_frontend on each implementation, imported from the package
__init__; the FrontendType literal and the if-chain in get_frontend are
gone. Each frontend declares required_backend and its recipe rules in
validate(config), so SrtConfig has one _validate_frontend that checks the
type is registered, enforces the pairing generically, and reports the
frontend's ValueError as a load-time ValidationError. The four
per-frontend schema validators (trtllm_serve, vllm, sglang, static
routers) move into their classes with their messages unchanged; the
direct sglang pairing message becomes the shared form.

Adding a frontend is now one module under srtctl/frontends plus one
import. Dry-run output for all 26 example recipes is unchanged.

Signed-off-by: Ishan Dhanani <idhanani@nvidia.com>
…aring frontend names

Every backend decided the worker shape by comparing the frontend name:
Dynamo registration versus a direct server, whether the server binds the
public port or its allocated http_port, and whether a router expands
per-node hybrid-LB DP pools. Those are frontend properties, so they move
onto the protocol as worker_launch, worker_api_port(mode), and
expands_node_local_dp. build_worker_command and endpoints_to_processes
still take frontend_type and resolve it through get_frontend; the eleven
name comparisons in the vLLM, SGLang, and TRT-LLM backends and the one in
the nsys prefix builder read those members instead. Two messages now name
the frontend generically.

Tests: a per-frontend contract test for the three members; the nsys
prefix tests use the registered router name instead of a legacy spelling.
Dry-run output for all 26 example recipes is unchanged.

Signed-off-by: Ishan Dhanani <idhanani@nvidia.com>
… the protocol

Four consumers decided which rank serves what by comparing the frontend
name, each with slightly different rules: tachometer scrape targets
(core/telemetry.py, 9 sites), the benchmark's logical worker endpoints,
the AIPerf server metrics URLs, and the profiling control endpoints in
benchmark_stage.py, plus the sequential-start readiness port in
worker_stage.py. Those become frontend members: metrics_path,
worker_metrics_port, worker_endpoint_port, profiling_control_port,
profiling_control_is_leader_only, direct_endpoint_nodes, and
worker_ready_port, implemented per class from the rules each site
encoded. No frontend-name comparison remains in telemetry.py or the
benchmark stage; BenchmarkStageMixin.frontend is None for a
services-only job.

Behavior changes, all corrections of dead ports the old per-site rules
produced: the trtllm_serve aggregate worker's logical endpoint is the
public port it binds rather than its unbound http_port; the direct sglang
nsys control endpoint is the leader's public port rather than http_port;
built-in AIPerf metrics for sglang-router target the worker leaders' HTTP
ports rather than Dynamo system ports no sglang worker serves; and direct
frontends no longer return early from the metrics env, so KVBM and CPU
power exporter URLs are appended for them like every other frontend.
The vLLM direct telemetry test fake gains the frontend_port it relies on.

Tests: a per-frontend contract test over a four-process topology for
every new member, and the Dynamo sidecar case. Dry-run output for all 26
example recipes is unchanged.

Signed-off-by: Ishan Dhanani <idhanani@nvidia.com>
…tends

The counterpart of StaticRouterFrontend. A static router is launched with
its worker URLs on the command line; a dynamic frontend is launched with no
worker list, workers announce themselves over a discovery plane, and
readiness is the frontend's own registration count. What every such
frontend shares moves into the base: it fronts any engine
(required_backend None), needs no per-worker URL gate, no worker is the
public endpoint, workers bind allocated ports, /health is the registration
endpoint, and frontend.args pass through with keys verbatim.

DynamoFrontend subclasses it and keeps only what is Dynamo's: the
dynamo.frontend launch, /health parsing, and the system-port rules. A
router binary that can also take static URLs (vLLM Router's ZMQ discovery
mode) remains a mode of a static router.

Signed-off-by: Ishan Dhanani <idhanani@nvidia.com>
@ishandhanani ishandhanani changed the title refactor(frontends): registry, validate(config), and worker shape on the frontend protocol refactor(frontends): the frontend class owns registration, validation, worker shape, and endpoints Sep 22, 2026
…rotocol

wait_for_model special-cased three frontends by name: trtllm_serve was
ready on a bare 200, direct vllm and sglang went through /health plus
/v1/models, everything else parsed a JSON worker count. And
_get_health_expectations in the benchmark stage knew that Dynamo counts
vLLM DP registrations and that vLLM Router counts expanded ranks. Both are
frontend knowledge.

probe_ready(host, port, n_prefill, n_decode) performs one readiness check
and returns a WorkerHealthResult, raising requests.RequestException while
the endpoint is down; wait_for_model keeps only timing, abort, and progress
logging. core/health.py offers the three probe shapes to build on:
probe_json_health (a JSON count through parse_health, the DynamicFrontend
and StaticRouterFrontend default), probe_http_ok (a bare 200,
trtllm_serve), and probe_direct_server (/health then /v1/models, direct
vllm and sglang). A frontend with an unusual readiness contract, such as a
router in a discovery mode, overrides probe_ready and touches nothing
else.

health_expectations(config, processes) returns the expected counts in the
units the frontend reports plus a description; Dynamo's vLLM DP
registration logic moves with it into frontends/dynamo.py, vLLM Router's
expansion sum into frontends/vllm_router.py, and every other frontend
counts logical workers through logical_health_expectations.
_get_health_expectations now delegates and keeps its signature.

Tests: probes for every frontend against mocked HTTP (bare 200, non-200,
JSON registry, direct server with and without a listed model, connection
errors propagating) and the wait_for_model loop (ready after a retry,
timeout while refused, stop event). The non-Dynamo expectations test names
a real frontend instead of none. Dry-run output for all 26 example recipes
is unchanged.

Signed-off-by: Ishan Dhanani <idhanani@nvidia.com>
Comment thread src/srtctl/frontends/dynamo.py
Review comment on #487. Ruff's unused-argument rules are off in this repo,
so 'del name' for a parameter a hook ignores adds nothing; the pattern had
spread from a few existing sites to every new protocol method. Remove all
49 in the frontends package and note the rule in CLAUDE.md.

Signed-off-by: Ishan Dhanani <idhanani@nvidia.com>
…d services and metrics listener on the frontend

With probe_ready owning readiness, health_endpoint and parse_health were
consulted only by the two bases' JSON probes and by a log line, and
get_frontend_args_list only by the three start_frontends that launch a
process. They leave the protocol: the bases keep them as their own hooks,
the direct frontends drop their never-called stubs, the dead
check_trtllm_serve_health parser goes, and the one verbatim-args
implementation becomes frontend_args_to_cli.

Two more decisions move onto the frontend. implied_services(config)
returns the services a frontend brings along; Dynamo's etcd and NATS rule
moves from services/implicit.py into frontends/dynamo.py, so a future
registration-based frontend declares its own discovery plane.
frontend_metrics_port(frontend_args) names a Prometheus listener separate
from the routing port; the SGLang gateway's rule moves out of the
telemetry stage. Two schema checks that re-derived frontend facts by name
(profiler control on the leader only, Dynamo system ports) read
profiling_control_is_leader_only and worker_launch instead.

CLAUDE.md and docs/architecture.md describe the protocol as it is now.
Dry-run output for all 26 example recipes is unchanged.

Signed-off-by: Ishan Dhanani <idhanani@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant