refactor(frontends): the frontend class owns registration, validation, worker shape, and endpoints - #487
Merged
Conversation
… on the protocol frontend.type now resolves through a registry populated by @register_frontend on each implementation, imported from the package __init__; the FrontendType literal and the if-chain in get_frontend are gone. Each frontend declares required_backend and its recipe rules in validate(config), so SrtConfig has one _validate_frontend that checks the type is registered, enforces the pairing generically, and reports the frontend's ValueError as a load-time ValidationError. The four per-frontend schema validators (trtllm_serve, vllm, sglang, static routers) move into their classes with their messages unchanged; the direct sglang pairing message becomes the shared form. Adding a frontend is now one module under srtctl/frontends plus one import. Dry-run output for all 26 example recipes is unchanged. Signed-off-by: Ishan Dhanani <idhanani@nvidia.com>
…aring frontend names Every backend decided the worker shape by comparing the frontend name: Dynamo registration versus a direct server, whether the server binds the public port or its allocated http_port, and whether a router expands per-node hybrid-LB DP pools. Those are frontend properties, so they move onto the protocol as worker_launch, worker_api_port(mode), and expands_node_local_dp. build_worker_command and endpoints_to_processes still take frontend_type and resolve it through get_frontend; the eleven name comparisons in the vLLM, SGLang, and TRT-LLM backends and the one in the nsys prefix builder read those members instead. Two messages now name the frontend generically. Tests: a per-frontend contract test for the three members; the nsys prefix tests use the registered router name instead of a legacy spelling. Dry-run output for all 26 example recipes is unchanged. Signed-off-by: Ishan Dhanani <idhanani@nvidia.com>
ishandhanani
requested review from
alec-flowers,
csahithi and
nlevin-ui
as code owners
September 22, 2026 00:06
… the protocol Four consumers decided which rank serves what by comparing the frontend name, each with slightly different rules: tachometer scrape targets (core/telemetry.py, 9 sites), the benchmark's logical worker endpoints, the AIPerf server metrics URLs, and the profiling control endpoints in benchmark_stage.py, plus the sequential-start readiness port in worker_stage.py. Those become frontend members: metrics_path, worker_metrics_port, worker_endpoint_port, profiling_control_port, profiling_control_is_leader_only, direct_endpoint_nodes, and worker_ready_port, implemented per class from the rules each site encoded. No frontend-name comparison remains in telemetry.py or the benchmark stage; BenchmarkStageMixin.frontend is None for a services-only job. Behavior changes, all corrections of dead ports the old per-site rules produced: the trtllm_serve aggregate worker's logical endpoint is the public port it binds rather than its unbound http_port; the direct sglang nsys control endpoint is the leader's public port rather than http_port; built-in AIPerf metrics for sglang-router target the worker leaders' HTTP ports rather than Dynamo system ports no sglang worker serves; and direct frontends no longer return early from the metrics env, so KVBM and CPU power exporter URLs are appended for them like every other frontend. The vLLM direct telemetry test fake gains the frontend_port it relies on. Tests: a per-frontend contract test over a four-process topology for every new member, and the Dynamo sidecar case. Dry-run output for all 26 example recipes is unchanged. Signed-off-by: Ishan Dhanani <idhanani@nvidia.com>
…tends The counterpart of StaticRouterFrontend. A static router is launched with its worker URLs on the command line; a dynamic frontend is launched with no worker list, workers announce themselves over a discovery plane, and readiness is the frontend's own registration count. What every such frontend shares moves into the base: it fronts any engine (required_backend None), needs no per-worker URL gate, no worker is the public endpoint, workers bind allocated ports, /health is the registration endpoint, and frontend.args pass through with keys verbatim. DynamoFrontend subclasses it and keeps only what is Dynamo's: the dynamo.frontend launch, /health parsing, and the system-port rules. A router binary that can also take static URLs (vLLM Router's ZMQ discovery mode) remains a mode of a static router. Signed-off-by: Ishan Dhanani <idhanani@nvidia.com>
…rotocol wait_for_model special-cased three frontends by name: trtllm_serve was ready on a bare 200, direct vllm and sglang went through /health plus /v1/models, everything else parsed a JSON worker count. And _get_health_expectations in the benchmark stage knew that Dynamo counts vLLM DP registrations and that vLLM Router counts expanded ranks. Both are frontend knowledge. probe_ready(host, port, n_prefill, n_decode) performs one readiness check and returns a WorkerHealthResult, raising requests.RequestException while the endpoint is down; wait_for_model keeps only timing, abort, and progress logging. core/health.py offers the three probe shapes to build on: probe_json_health (a JSON count through parse_health, the DynamicFrontend and StaticRouterFrontend default), probe_http_ok (a bare 200, trtllm_serve), and probe_direct_server (/health then /v1/models, direct vllm and sglang). A frontend with an unusual readiness contract, such as a router in a discovery mode, overrides probe_ready and touches nothing else. health_expectations(config, processes) returns the expected counts in the units the frontend reports plus a description; Dynamo's vLLM DP registration logic moves with it into frontends/dynamo.py, vLLM Router's expansion sum into frontends/vllm_router.py, and every other frontend counts logical workers through logical_health_expectations. _get_health_expectations now delegates and keeps its signature. Tests: probes for every frontend against mocked HTTP (bare 200, non-200, JSON registry, direct server with and without a listed model, connection errors propagating) and the wait_for_model loop (ready after a retry, timeout while refused, stop event). The non-Dynamo expectations test names a real frontend instead of none. Dry-run output for all 26 example recipes is unchanged. Signed-off-by: Ishan Dhanani <idhanani@nvidia.com>
ishandhanani
commented
Sep 22, 2026
Review comment on #487. Ruff's unused-argument rules are off in this repo, so 'del name' for a parameter a hook ignores adds nothing; the pattern had spread from a few existing sites to every new protocol method. Remove all 49 in the frontends package and note the rule in CLAUDE.md. Signed-off-by: Ishan Dhanani <idhanani@nvidia.com>
…d services and metrics listener on the frontend With probe_ready owning readiness, health_endpoint and parse_health were consulted only by the two bases' JSON probes and by a log line, and get_frontend_args_list only by the three start_frontends that launch a process. They leave the protocol: the bases keep them as their own hooks, the direct frontends drop their never-called stubs, the dead check_trtllm_serve_health parser goes, and the one verbatim-args implementation becomes frontend_args_to_cli. Two more decisions move onto the frontend. implied_services(config) returns the services a frontend brings along; Dynamo's etcd and NATS rule moves from services/implicit.py into frontends/dynamo.py, so a future registration-based frontend declares its own discovery plane. frontend_metrics_port(frontend_args) names a Prometheus listener separate from the routing port; the SGLang gateway's rule moves out of the telemetry stage. Two schema checks that re-derived frontend facts by name (profiler control on the leader only, Dynamo system ports) read profiling_control_is_leader_only and worker_launch instead. CLAUDE.md and docs/architecture.md describe the protocol as it is now. Dry-run output for all 26 example recipes is unchanged. Signed-off-by: Ishan Dhanani <idhanani@nvidia.com>
ishandhanani
added this pull request to stack #490
September 22, 2026 01:44
This was referenced Sep 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adding a frontend meant editing a
Literal, an if-chain, a schema pairing map, four per-frontend schema validators, elevenfrontend_type == "..."comparisons across the three backends, nine more in the telemetry target builder, a dozen in the benchmark stage, and three special cases inside the readiness loop. This PR makes the frontend class the owner of every one of those decisions, so a new router is one registered module. It is the groundwork #432 (vLLM MoRI-IO) will be rewritten onto.Seven commits, each behavior-preserving except where noted:
Registry,
required_backend,validate(config).@register_frontend("<type>")on each implementation, imported from the package__init__, replaces theFrontendTypeliteral and the if-chain inget_frontend. Each frontend declaresrequired_backendand its recipe rules invalidate(config).SrtConfighas one_validate_frontend: type registered, backend pairing generic, the frontend'sValueErrorreported as a load-timeValidationError. The four per-frontend schema validators move into their classes with their messages unchanged.Worker shape.
worker_launch(dynamoordirect),worker_api_port(mode)(publicorallocated), andexpands_node_local_dpon the protocol. Backends resolvefrontend_typethroughget_frontendand read those; no frontend-name comparison remains inbackends/vllm.py,sglang.py,trtllm.py, or the nsys prefix builder.Endpoint, metrics, and profiling ports.
metrics_path,worker_metrics_port,worker_endpoint_port,profiling_control_port,profiling_control_is_leader_only,direct_endpoint_nodes,worker_ready_port. Tachometer scrape targets, the benchmark's logical endpoints and public API node, AIPerf metrics URLs, profiling control, and the sequential-start readiness port read them. Four behavior changes, each replacing a dead port the old per-site rules produced: the trtllm_serve aggregate endpoint is the public port it binds rather than its unboundhttp_port; the direct sglang nsys control port is the leader's public port rather thanhttp_port; built-in AIPerf metrics for sglang-router target the worker leaders' HTTP ports rather than Dynamo system ports no sglang worker serves; direct frontends no longer return early from the metrics env, so KVBM and CPU power exporter URLs are appended for them too.DynamicFrontendbase. The counterpart ofStaticRouterFrontendfor frontends whose workers register themselves over a discovery plane. Dynamo is its only implementation; vLLM Router's ZMQ discovery mode stays a mode of a static router.Readiness on the protocol.
probe_ready(host, port, n_prefill, n_decode)performs one readiness check and returns aWorkerHealthResult;wait_for_modelkeeps only timing, abort, and logging, with no frontend names.core/health.pyoffers the three probe shapes:probe_json_health(a JSON count throughparse_health),probe_http_ok(a bare 200, trtllm_serve),probe_direct_server(/healththen/v1/models, direct vllm and sglang).health_expectations(config, processes)returns the counts in the frontend's own units; Dynamo's vLLM DP registration logic moves intofrontends/dynamo.py, vLLM Router's expansion sum intofrontends/vllm_router.py, and_get_health_expectationsdelegates with its signature unchanged. A router in a discovery mode overridesprobe_readyand touches nothing else.No
delof unused arguments. Review comment: ruff's unused-argument rules are off here, so the 49del namelines in the frontends were noise. Removed, with the rule noted inCLAUDE.md.Protocol slimmed to what consumers call;
implied_servicesandfrontend_metrics_port.health_endpoint,parse_health, andget_frontend_args_listleave the protocol (the two bases keep them as their own hooks; the direct frontends drop never-called stubs; the deadcheck_trtllm_serve_healthgoes; one verbatim-args helperfrontend_args_to_cli).implied_services(config)moves Dynamo's etcd/NATS rule fromservices/implicit.pyontoDynamoFrontend, so a future registration-based frontend declares its own discovery plane.frontend_metrics_port(frontend_args)moves the SGLang gateway's separate Prometheus listener out of the telemetry stage. Two schema checks that re-derived frontend facts by name (profiler control on the leader only, Dynamo system ports) readprofiling_control_is_leader_onlyandworker_launch.docs/architecture.mddescribes the protocol as it is now.Validation
tests/test_frontends.pyandtests/test_health.py: registry contents, registering a toy frontend, unknown type rejected at load, generic pairing per frontend, contract tests for the worker-shape and port members, the Dynamo sidecar case, theDynamicFrontenddefaults, every frontend's probe against mocked HTTP, and thewait_for_modelloop).srtctl dry-runfor all 26 example recipes is byte-identical to main after normalizing generated timestamps, after every commit.Review map
frontends/base.py: the protocol members, the registry,logical_health_expectations,agg_leader_nodes.frontends/dynamic_frontend.py,frontends/static_router.py: the two bases.frontends/*.py: decorator,required_backend,validate, worker shape, port rules, probe, and expectations per class.core/health.py:wait_for_modelreduced to the loop; the three probe helpers.core/schema.py:_validate_frontendreplaces four validators; the nsys prefix readsworker_launch.backends/vllm.py,sglang.py,trtllm.py: name comparisons become attribute reads.core/telemetry.py,cli/mixins/benchmark_stage.py,cli/mixins/worker_stage.py: consumers read the frontend.CLAUDE.md: Frontends section and the mode-versus-type checklist describe the registry, the two bases, and the members.The frontend-name checks left outside
frontends/are Dynamo-config and SGLang-gateway feature switches (sidecar, failover, and worker_selection validators,installs_dynamo, request tracing,slow_down, dry-run panels): each asks whether the recipe uses that product's feature block, not a topology or readiness question.