Repository navigation
[TRTLLM-17267][refactor] Move the operational endpoints into serving classes and routers - #21725
Draft
JunyiXu-nv wants to merge 85 commits into
Draft
JunyiXu-nv wants to merge 85 commits into
JunyiXu-nv wants to merge 85 commits into
Conversation
…ce per server OpenAIServer decided nine times, in four handlers, whether a response's post-processed value comes from a postproc worker (read the value the worker stored on outputs[0]) or must be computed in the server process (call the post-processor inline). The flag never changes after construction, so the server now chooses a strategy object once: select_postproc_strategy picks WorkerPostprocStrategy or InlinePostprocStrategy from num_postprocess_workers; __init__ pre-caches it and a cached property keeps server instances that unit tests build without __init__ working. The nine decision points become one strategy call each, and the conditional post-processor unpacks they fed are gone. CompletionOutput gains the read-only public accessor postprocess_result so the server no longer reads the executor's private field; the api-stability reference records the new property. The submission-side uses of postproc_worker_enabled (what generate_async receives, filling postproc_args for in-server post-processing, the Responses store decision) are deliberately unchanged. No behavior change. With more than one candidate the two modes still differ in what the frontend records (worker mode: only outputs[0] carries the result and metrics arrive merged; inline mode records metrics per candidate); that known difference is documented on the strategy class and left for a separate change. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ss lint for the serving frontends The HTTP server, the SMG gRPC frontend and the OpenEngine gRPC frontend learned engine health by reading private members (_check_health, _executor, _fatal_error, _engine_dead). Add the public health API on the classes that own those members and switch all three frontends to it, with no behavior change: - BaseLLM.check_health() (public rename of _check_health; the private name stays for one release as an alias for out-of-tree callers that look it up), BaseLLM.fatal_error, BaseLLM.shutdown_started, BaseLLM.engine_dead. - GenerationExecutor.fatal_error and engine_dead; GenerationExecutorProxy.engine_dead also reads the sticky worker-death flag set before the fatal error is recorded. Add a private-access lint (tests/unittest/serve/test_private_access_lint.py) that counts cross-package private accesses in tensorrt_llm/serve, tensorrt_llm/commands and tensorrt_llm/grpc against a checked-in baseline (tests/unittest/serve/private_access_baseline.yaml) that only shrinks, with companion test files under tests/unittest/serve/api/visual_gen/ and tests/unittest/grpc/openengine/ so VisualGen-only and OpenEngine-only pull requests keep running the check. Replace the three frontend health mirror tests with tests of the production code, rename the _check_health test doubles to the public name, and register the new CPU test files in l0_cpu.yml. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…lope Pin the exact 422 body at the route-level entry point (FastAPI RequestValidationError on /v1/images/generations, which declares its request model) and at the handler-level entry points (pydantic ValidationError raised while building the request model on /v1/videos/sync, /v1/videos and /v1/images/edits, JSON and multipart). All entry points must render byte-identical bodies; these tests pin the full body so a later refactor of the rendering code has exact coverage. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…derers OpenAIServer._create_visual_gen_validation_error_response (for FastAPI's RequestValidationError) and _VideoRoutesMixin._render_pydantic_validation_error (for pydantic's ValidationError) were statement-for-statement identical. Replace both with one module-level function, render_visual_gen_validation_error in tensorrt_llm/serve/_visual_gen_errors.py, that accepts either exception and the server's error-response factory, and point all six call sites at it. The factory argument keeps the error envelope and the context-length code rule defined in one place and avoids an import cycle with openai_server. This also removes the only call OpenAIServer makes to a method defined by the video-routes mixin (openai_image_edit called the mixin's renderer). No behavior change: both entry points return the same status and bytes as before; the characterization tests added in the previous commit pass unchanged on both sides of this refactor. The new module joins the mypy file list. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…erving-frontend private accesses The serving frontends (tensorrt_llm/serve, tensorrt_llm/commands, the OpenEngine gRPC frontend) read private members of LLM, the executors, CompletionOutput, SamplingParams, MultimodalDataTracker, OpenAIDisaggServer and PyExecutor. Give each remaining read a thin public entry point on the class that owns the member, switch the frontends to it with no behavior change, and shrink the private-access baseline from 50 recorded accesses to 19. The reads of the two private LlmArgs attributes stay (exposing them is an LLM-args change for its own PR); the remaining baseline entries belong to other in-flight work. New public members (status prototype where api-stability tracks them): LLM.record_fatal_error / multi_frontend_attach_info / multi_frontend_ipc_dir / engine_in_worker_process / hf_model_dir / hf_model_config / resource_governor_queue / encode_limits / get_kv_cache_load / get_openengine_discovery; GenerationExecutor.record_fatal_error and multi_frontend_ipc_dir; CompletionOutput.spec_dec_counters and text_diff_offset (and the SpecDecCounters rename); SamplingParams.set_request_provided_fields and stop_word_ids; MultimodalDataTracker.multimodal_server_config; OpenAIDisaggServer.address; PyExecutor.set_openengine_discovery and set_openengine_node_server; registered_model_type() in inputs/registry. Also adds tensorrt_llm/llmapi/engine_client.py: the EngineClient protocol naming the engine operations every frontend shares (generate, health, load), checked by tests rather than at runtime. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…a bare OpenAIServer The shut-down-LLM case of the launcher-lost watchdog test now calls the production method unbound on a stand-in object whose generator double defines the public members the method reads, the form the file's other stand-in tests use. New tests must not build servers with object.__new__. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…apshots and a CPU test seam for trtllm-serve Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… baselines at the merge base The route reference is generated from the live apps; the object.__new__ baseline and the gRPC import allowance record what exists on main today and only shrink. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…n the snapshot matrix An attached frontend of a multi-frontend server registers ten launcher forwarding routes before the local table; seven share a method and path with local handlers and three use a path parameter that matches every batch id, so registration order decides which handler serves them. The new configuration records all of it, a live invariant pins the forwarding-before-local order independently of the reference, and the route-class failure message now says why the msgpack route class matters. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
- AGENTS.md: the CPU test command sets LLM_MODELS_ROOT, which the runner requires, and the stage-mapping note says the mapper matches list-entry text, so a file listed through its directory maps by the directory. - check_contracts matches each from-import as <module>.<name>, so 'from pkg import sub' and 'from .. import sub' count as imports of the submodule; new test for both forms and for the allowance entry. - The VisualGen /v1/models test asserts the 200 status. - The AGENTS.md command check also recognizes commands that start with LLM_MODELS_ROOT=. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…on the current main The disaggregated frontend now registers its routes with KvHintsRoute (NVIDIA#20269), which strips kv_hints from request bodies unless the server trusts its ingress. Regenerate the route reference: the six disaggregated configurations record KvHintsRoute where they recorded APIRoute; their route sets and order are unchanged. The route-class invariant, which a regenerated reference cannot hide, now requires KvHintsRoute on every API route of the disaggregated frontend and APIRoute on the coordinator app. Dropping the KV-hint policy from the frontend fails it even after a regeneration, and both apps still reject the aggregated server's msgpack route class. The object.__new__ baseline records the sites on main at the merge base: test_kv_hints.py builds one server that way (NVIDIA#20269), and test_harmony_parsing.py no longer builds any (NVIDIA#21176). The harmony parsing tests are now registered as a whole file (NVIDIA#21411), so the node-id entry that the CPU-selection test used as its example is gone. The node-id check now runs select_entries on a one-entry list of its own, so it no longer depends on which node-id entries l0_cpu.yml holds. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ail on silent regressions The route snapshot now requires every configuration other than the three invalid ones to build, so a configuration that starts failing at construction cannot pass through a regenerated reference. The object.__new__ ratchet resolves 'from ... import <server class> as <name>' aliases and records each baseline site as its enclosing function or class plus the server class, so swapping one site for another in the same file no longer passes. The baseline keeps the same 52 sites in the same 26 files. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…s outside-subclass contract Every OpenAI request on the disaggregated frontend, and every Anthropic /v1/messages request, goes through OpenAIDisaggServer._wrap_entry_point. Before reorganizing that closure, pin what it does today: - tests/unittest/disaggregated/test_disagg_entry_point.py pins the per-request counters, the chat-template 400, the conversation identity from the headers, the serialization of streaming and non-streaming responses (including the guarded streamed Responses relay and model_dump(by_alias=True)), the error mapping through _handle_exception, and the single disaggregated_request trace span. - tests/unittest/disaggregated/test_disagg_server_contract.py pins the surface NeMo-RL's OpenAIDisaggServerAdaptor subclass relies on: the constructor (config, req_timeout_secs=..., coordinator_url=None), self.app with every route registered when __init__ returns, __call__(host, port, sockets=None), _wrap_entry_point, _handle_exception and _perf_metrics_collector.http_exceptions. - The OpenAIDisaggServer class docstring and tensorrt_llm/serve/AGENTS.md name that surface, so a change to it is agreed with the NeMo-RL owners first. Both files are CPU-only and collected by the unittest/disaggregated entry of tests/integration/test_lists/test-db/l0_cpu.yml. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…into prepare, run, finish and serialize steps No behavior change. The body of the closure OpenAIDisaggServer builds in _wrap_entry_point becomes a composition of five methods, so that later changes (disconnect watching, in-stream errors, request trace) land in the step they belong to instead of growing one closure: - _prepare_request: the request counters, the chat-template permission (HTTPException 400), the conversation identity, the per-request hooks - _run_entry_point: the call into the disaggregated service - _finish_request: success accounting (total_responses) - _finish_failed_request: every failure, through _handle_exception while the exception is being handled (its traceback logging needs that) - _serialize_response: StreamingResponse with the Responses stream guard, or JSONResponse(content=...model_dump(by_alias=True)), as before The trace span stays on the wrapper that FastAPI calls; the steps open no spans of their own. The statements keep their order inside the same try block: the counters run before the template check, total_responses before serialization, and a serialization failure still reaches _handle_exception. Some step parameters (raw_req of _run_entry_point, req and hooks of the finish steps) are not read yet; they are the inputs the known upcoming changes to each step need, so those changes will not have to alter the signatures again. The characterization tests and the subclass contract tests of the previous commit pass unchanged on both sides of the split. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… and disaggregated servers Create tensorrt_llm/serve/middleware/ and move ServerArrivalTimeMiddleware (from serve/responses_utils.py), PerfMetricsMiddleware (from serve/perf_metrics.py) and _apply_fastapi_middlewares (from commands/serve.py) into it, verbatim. Both servers now install their built-in middleware through serve/middleware/register.py::register_middlewares, which reproduces each app's stack exactly: the aggregated server passes adjusted_clock to both middlewares, the disaggregated server to neither. User --middleware entries are still applied after construction and stay outermost. The old import paths are removed with no compatibility shim; no code outside the repository imports them. commands/serve.py keeps an import of _apply_fastapi_middlewares for its own two launcher calls. CI bookkeeping in the same change: the multi-GPU trigger table gains tensorrt_llm/serve/middleware/; legacy-files.txt registers middleware/arrival_time.py and middleware/user_middleware.py with the configs gen-configs regenerates; ruff-legacy-baseline.json carries the moved docstring's one D212; mypy checks the new register.py. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…g request state
Replace the assignment scope["state"] = {} with scope.setdefault("state", {})
in ServerArrivalTimeMiddleware.__call__. State already present when a request
reaches this middleware - written by a user --middleware entry, or copied in by
the ASGI server from a lifespan that yields a dict - is now kept, and
server_arrival_time is added to it. Before, that state was replaced by an empty
dict. Neither trtllm-serve app yields lifespan state today and the built-in
middleware order is unchanged, so no built-in path changes behavior.
Two tests pin the new behavior (state from an outer middleware and from a
lifespan survive); both fail on the previous commit.
Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…lm/serve/_base/msgspec_route.py Generated move: _msgpack_decoder, _MsgspecRequest and _MsgspecRoute leave tensorrt_llm/serve/openai_server.py byte for byte; the module imports _MsgspecRoute back for its app router and keeps no re-export (the three names are private). The two disaggregated test importers now import the new module. OpenAIServer.health, .version and .get_model are removed here and replaced by the system router added in the following commits. tensorrt_llm/serve/_base/msgspec_route.py joins legacy-files.txt (its code comes from a legacy-formatted file), and the generated lint configs are regenerated with scripts/legacy_utils.py gen-configs. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ve the system endpoints through it New structure: tensorrt_llm/serve/_base/ (shared serving infrastructure) and tensorrt_llm/serve/api/ (endpoint packages, one router plus one serving class each; the package __init__ imports nothing so routers load lazily, by role). serve/_base/routing.py provides make_router(), which always uses the msgpack-decoding route class, and attach_routers(), which imports and attaches only the routers whose RouterSpec matches the server role. The first endpoint package, serve/api/ops/system/, serves /health, /version and /v1/models for every role. OpenAIServer builds a SystemServing with the injected _health_probe (today's probe body, SIGINT side effect included) and attaches the routers before the role dispatch; the 12 add_api_route lines for the three paths leave the four role tables. Same routes, methods, route classes, bodies and conflict-relevant order: the recorded route reference is unchanged. The layering tests gain the new packages' contracts, the route-snapshot failure message points new endpoints at the router pattern, and the serve agent docs describe where a new endpoint goes. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…w serve packages tensorrt_llm/serve/_base/ and tensorrt_llm/serve/api/ops/ join the multi-GPU trigger list: the msgpack route class decodes the disaggregated frontend's worker requests, and /health is the disaggregated readiness probe, so changes under these directories keep running the multi-GPU stages that exercise that path. The three new modules join the strict mypy file set, and the type-check pre-commit hook now also triggers on changes under tensorrt_llm/serve/ and tensorrt_llm/commands/serve_cli/, so edits to typed serving modules get local type checking; the hook still checks only the declared file set. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… tests that build routes by hand Two multi-frontend tests build an OpenAIServer without its constructor and call register_routes() themselves. Since /health, /version and /v1/models are attached through the system router by the constructor, those tests no longer saw the three routes: one asserted that /version stays local on an attached frontend (404), the other reported the three per-frontend-safe entries as removed. Both helpers now set the serving object and attach the routers before the hand registration, in the constructor's order, and the route enumeration walks included routers, which FastAPI 0.139 keeps as objects without a path in app.routes. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…n the OpenEngine metrics test The HTTP-enablement case builds a real OpenAIServer through its constructor and looked for /prometheus/metrics by reading .path off every entry of app.routes. Since the constructor attaches the system router, FastAPI 0.139 keeps an included-router object there that has no .path, and the child process failed with AttributeError. The test now matches a GET /prometheus/metrics request against the routes with the public route-matching API, which works on both ends of the pinned FastAPI range. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…d name The check that endpoint packages under tensorrt_llm/serve/api do not import each other read the package off the from-module alone, so `from tensorrt_llm.serve import api` inside an endpoint package passed and `from tensorrt_llm.serve.api.ops import system` inside ops/system failed. It now reads the package off <module>.<name> per imported name, as the layering contracts do, and a synthetic-edge test pins both forms. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…e/api/visual_gen Move the eight VisualGen request and response models (ImageGenerationRequest, ImageEditRequest, ImageObject, ImageGenerationResponse, MediaReferenceItem, VideoGenerationRequest, VideoJob, VideoJobList) out of tensorrt_llm/serve/openai_protocol.py into the new package tensorrt_llm/serve/api/visual_gen/ (protocol.py), unchanged. openai_protocol keeps the eight names importable through a lazy module __getattr__: importing it no longer loads the VisualGen models or tensorrt_llm.visual_gen, and the re-export is part of its own interface, so it never warns and has no removal release. The imports only the moved block used (UploadFile, AliasChoices, MediaRole) leave openai_protocol with it. Every importer in the repository now imports the new module; the benchmarking guide links the new file. protocol.py joins legacy-files.txt (its code comes from a legacy-formatted file) and the derived lint configs are regenerated. No behavior change. OpenAPI component names may change with the move; they are not a contract. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…isualGen team in CODEOWNERS VisualGen serving code now lives under tensorrt_llm/serve/api/visual_gen/. The new rule, placed after the /tensorrt_llm/serve rule so that it wins, makes @NVIDIA/trt-llm-torch-visual-gen-devs the owner of that directory: changes there need the VisualGen team's approval and no longer need runtime-devs. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…y re-export under a visual_gen path Add tests/unittest/api_stability/visual_gen/test_visual_gen_serve_api.py: it runs the checks of test_serve_api.py on the models of references/trtllm_serve_api.yaml that are defined in tensorrt_llm.serve.api.visual_gen.protocol, and checks that tensorrt_llm.serve.openai_protocol re-exports the eight VisualGen names lazily (importing it does not load the VisualGen module; unknown names still raise AttributeError). The file lives under a visual_gen/ path and has its own l0_cpu.yml entry so that CI selection for VisualGen-only pull requests keeps running it; the reference YAML is unchanged. Add the eight identity rows of the VisualGen names to the VisualGen shim identity table: identity only, no warning and no ban on the old path, because the re-export is openai_protocol's permanent interface. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…its no-warning contract Resolve every pinned model with the full gate's own loader, so a VisualGen model pinned in the reference YAML that is later renamed or removed fails this gate too instead of being dropped silently; the four known models stay a secondary check. Make the lazy-load probe's diagnostic best-effort and identity-based, so a failure prints which modules hold the VisualGen classes instead of crashing on module-level __getattr__ hooks while walking sys.modules. In the same fresh-interpreter probe, assert that reading each of the eight names from openai_protocol emits no FutureWarning or DeprecationWarning. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…efore splitting it Three tests run OpenAIServer.openai_chat through a server built with the real constructor and the fake generator, and pin what it answers today: a request the handler rejects keeps its 400 body byte for byte, an engine failure keeps HTTP 200 with body null (and /health_generate keeps turning that into a 500), a response that cannot be JSON-encoded keeps its 400, and a stream keeps its frames and media type. They pass on the unsplit handler and must still pass once it is split into a typed core and an HTTP wrapper. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…serve/_base/sse.py Generated move: the chat_stream_generator closure of OpenAIServer.openai_chat is lifted to module level in tensorrt_llm/serve/_base/sse.py unchanged, apart from its two free variables, which become the keyword-only parameters `server` (was `self`) and `raw_request`. openai_chat imports it back and passes server=self, raw_request=raw_request at its one call. The generator keeps its `except Exception:` clause and its two `_postprocess_result` reads. tensorrt_llm/serve/_base/sse.py joins legacy-files.txt (its code comes from a legacy-formatted file) and the generated lint configs are regenerated with scripts/legacy_utils.py gen-configs; ruff-legacy-baseline.json gains the module's one recorded F821 (the `anext` builtin ruff does not know), which openai_server.py keeps counting until its baseline is regenerated. The Anthropic stream tests patched the logger of openai_server to check what the generator logs; they now patch the logger of serve/_base/sse.py, where the generator lives. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…tokenize routers and serving objects Each new package gets an api_router.py built with make_router() and a RouterSpec in ROUTERS for exactly the roles the route had. The composition root builds RenderResources once after _init_llm and injects it, constructs the serving objects inside the role branches, puts them on app.state, and attaches the routers after the role switch, where every attach_router can check its app.state key. The two route registrations the routers now serve leave register_routes and register_mm_encoder_routes. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…enize serving classes on CPU Construction tests (every dependency is a required keyword-only argument), route tests through the real composition root and routers (msgpack bodies included), the batcher reference the health probe reads, the injected RenderResources carrying the server chat template, the serving bases and the borrowed-helper identity on OpenAIServer. The encode-only fake now encodes fixed embeddings and scripted outputs can carry disaggregated params; the app factory passes the chat-template knobs. The runtime-API tests build EmbeddingServing and patch the termination telemetry where the moved helper looks it up; the lazy-import guard also checks that importing tensorrt_llm.serve loads no endpoint package. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… and the serve guide The multi-GPU trigger table gains the three package prefixes, so code moved out of tensorrt_llm/serve/openai_server.py keeps its trigger; the three router modules join the mypy file list; the serve guide states the constructor rule of serving classes. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
… stub provide the tokenizer The test runs the real OpenAIServer constructor with _init_llm replaced by a stub. The constructor now builds the tokenize serving object for the LLM role from self.tokenizer, so the stub sets it too; a missing dependency fails at construction on purpose instead of at request time. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ition root with the tokenize serving object Both tests build a partial OpenAIServer and register its routes the way the constructor does. The constructor now puts the tokenize serving object on app.state and attaches the role's routers after the local table, so the re-enactments do the same; the tokenize router refuses to attach without its serving object. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…unyix-feat-serve-completions-serving (integration base, not part of the reviewable change) Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…nd the generate-pipeline constructor arguments Create tensorrt_llm/serve/api/openai/completions/ with the CompletionServing skeleton, and give GenerateBaseServing the keyword-only constructor arguments that the completions handler and the helpers it shares with chat and Responses read: the perf-metrics switch and collector, the adjusted steady clock, the iteration-stats wake-up event and a getter for the collector task that the lifespan creates later, the internal disaggregated auth key, the per-request speculative-decoding opt-in and its draft-token bound, the input-processor thread pool and the post-processing strategy. The new Group B files are registered in legacy-files.txt and the generated lint configs are regenerated. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…Serving and lift its closures Generated move: openai_completion leaves OpenAIServer for CompletionServing in tensorrt_llm/serve/api/openai/completions/serving.py; its five nested functions become module-level functions of tensorrt_llm/serve/api/openai/completions/_merge.py whose free variables (the serving object, the raw request, the disaggregated parameters) are keyword-only parameters, so they can be unit-tested; the six helpers it shares with the chat and Responses handlers (postproc_worker_enabled, _extract_metrics, _logit_bias_vocab_size, _validate_internal_disagg_request, _has_cache_transceiver_config, _apply_spec_decode_stats_opt_in) move into GenerateBaseServing and stay reachable on OpenAIServer as aliases of the moved objects while chat and Responses call them there. The one recorded F821 (anext) of the lifted generator_wrapper follows it into the lint baseline under its new path. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…s router and serving object The LLM role group builds CompletionServing in the composition root, next to TokenizeServing, with the generate-pipeline arguments read once by _generate_serving_kwargs(), and puts it on app.state; the new tensorrt_llm/serve/api/openai/completions/api_router.py maps POST /v1/completions to it through make_router(), so the route keeps the msgpack route class the disaggregated frontend relies on, and joins the routing table for the LLM role group. register_routes no longer registers the route itself. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…, and build the completions tests on serving objects New CPU tests: tests/unittest/serve/api/openai/completions/test_merge.py exercises each lifted function alone (usage merging, generator merging, the SSE error frame, the first-token stamp, context-only prompt ids, the error path); test_serving.py covers the constructor contract, the shared composition-root objects, the route for one and several prompts, msgpack bodies, the wired internal disaggregated auth key, the stats-collector wake-up and the role gate. The tests that built an OpenAIServer with object.__new__ to reach the completions code (the streaming error test, the list-prompt streaming test, the cache-transceiver auth test, the spec-decode opt-in test and the KV-hints handler test) now build a CompletionServing with explicit arguments, and the object.__new__ ratchet drops their rows. The direct MMEncoderServing constructions pass the new base arguments; the borrowed-helper table gains the six helpers; the fixture factory can set the internal disaggregated auth key and a cache transceiver backend. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…-GPU trigger table and type checking The multi-GPU trigger table gains the tensorrt_llm/serve/api/openai/completions/ prefix, so the completions code that left tensorrt_llm/serve/openai_server.py keeps triggering the disaggregated stages; the new router module joins the mypy file list. The moved serving and lifted-function modules stay out of it, like the other modules that hold moved legacy code. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…se-class test the new generate-pipeline arguments The multi-frontend tests re-enact the composition root by hand: they now put a CompletionServing with placeholder dependencies on app.state before attaching the role's routers, as the constructor does, because the completions router refuses to attach without it. The OpenEngine Prometheus test's constructor stub sets model_config too, since the LLM role now reads it when it builds the completions serving object. The base-class test requires every GenerateBaseServing argument, pins the attribute each one is stored as, and checks that the collector-task property follows the root's current binding. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…r what it is and cover the merged several-prompt response GenerateBaseServing's constructor argument that carries the getter of the composition root's iteration-stats collector task is now called iteration_stats_collector_task_getter at every site: the base class, the root's _generate_serving_kwargs(), the direct constructions in tests and the hand-built route tables. The stored attribute and the read-only property keep their names. The completions route test suite gains the non-streaming several-prompt case: one merged response with one choice per prompt and the summed usage, which fails when the merge call loses its serving argument or the merge drops a prompt. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…outer and pass the kv-hints handlers a request tests/unittest/llmapi/apps/test_responses_usage.py mounted OpenAIServer.anthropic_messages, which no longer exists: the Anthropic routes live in tensorrt_llm/serve/api/anthropic/api_router.py and read their serving object from app.state. The test's bare app now mounts the router's route function and carries the AnthropicMessagesServing the server's constructor built, so the Messages route still runs over the same chat core as the chat route. tests/unittest/llmapi/test_kv_hints.py called the handlers with raw_request=None. The chat wrapper now wraps the request in a RequestContext, which needs an object to delegate to, so the handlers get a minimal request (empty headers, an empty state), as the chat core tests do. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…-junyix-feat-serve-chat-serving (integration base, not part of the reviewable change) Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…server-role constructor argument ChatServing is the serving class of POST /v1/chat/completions; this commit adds its constructor (one keyword-only argument per server attribute the chat handlers read) and registers the module with the legacy formatting group of the code it will receive. GenerateBaseServing gains server_role, which the scheduling-parameter helper reads, and the composition root passes it with the other generate-pipeline arguments. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
openai_chat (the HTTP wrapper), create_chat_completion (the typed core), _create_chat_response and chat_harmony leave OpenAIServer for ChatServing; _get_scheduling_params joins GenerateBaseServing next to the other generate-pipeline helpers. The module-level tool-call helpers move with the chat code; _configure_parser_special_token_decoding goes to tensorrt_llm/serve/_base/serving.py because the Responses handler calls it too and packages under serve/api/ do not import each other. The aggregated chat backend of the Anthropic Messages serving class is ChatServing itself: its stream_model and error_message move there and the adapter class goes. The five helper aliases whose last caller was the chat code leave OpenAIServer. Every repository import of the moved helpers is rewritten; none is re-exported from openai_server.py. The moved anext of chat_harmony's generator is recorded for the new module in the legacy lint baseline. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…router and serving object The composition root builds ChatServing in the LLM role group, puts it on app.state, and hands it to the Anthropic Messages serving class as its chat backend. The chat router joins the routing table for the LLM role group; it holds two routers with one route each and attaches the one the serving object's use_harmony selects, so the route keeps the endpoint name the server gave it (openai_chat, or chat_harmony on a gpt-oss server). /health_generate stays on the root and calls the serving object's openai_chat. The new package joins the multi-GPU trigger table and the typed modules. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…on aliases the Responses handler still calls OpenAIServer.openai_responses validates the protected disaggregated fields through _validate_internal_disagg_request, which reads _has_cache_transceiver_config on the same object, so both stay reachable on the class as the moved helpers until the Responses handler moves. The three aliases whose last caller was the chat code are gone. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…rving objects New: tests/unittest/serve/api/openai/chat/test_serving.py pins the constructor (every dependency keyword-only and required, each stored where the moved handlers read it), that the serving object is the composition root's and uses its render resources, clock, stats event, input-processor pool, metrics collector, post-processing strategy and server role, the chat route on the three LLM roles (one chat, a stream ending in [DONE], msgpack bodies), the endpoint name on harmony and non-harmony servers built in one process, /health_generate through the serving object (200, 500 on a rejected chat, the beam-search arguments), the Anthropic Messages backend identity with stream_model and error_message, the wired internal auth key with its control, and the embedding and encoder roles. tests/test_common/serve/serving_stubs.py builds serving objects with placeholder dependencies for tests that call a handler off the app. Rewritten onto serving objects: the tests that built OpenAIServer with object.__new__ to call a chat handler (internal auth, disaggregated service, router, kv hints, serving extensions, Anthropic routes), the stubs of the typed-core and Anthropic behavior tests (now on server._chat_serving, with the engine-failure recorder patched where the moved core looks it up), the two multimodal fixtures on the stub server (which now gets its model config from the loader, so the constructor builds the serving objects from it), the Responses usage fixture (the chat route from its router; the Messages route keeps the mount NVIDIA#21551 gave it), the multi-frontend re-enactments (a chat serving object before attaching) and the OpenEngine Prometheus stub. Every direct construction of a generate serving object passes server_role. The ratchet and borrowed-helper tables follow. The legacy lint baseline keeps its unrelated entries where they were (the generated move had reordered two of them) and, for now, no entry for the moved code; the next commit records it. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ne and pin the server chat template through the chat route The legacy lint baseline gets the one violation recorded for the moved chat code, the anext of chat_harmony's streaming generator, under its new module: ruff releases that assume Python 3.9 report it as an undefined name, and the baseline is what keeps the legacy lint green with them. The render-resources test now also posts a chat through the route and checks that the prompt the handler renders and the ids that reach the engine come from the server chat template, which a RenderResources built from the serving object would drop. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…yix-feat-serve-ops-routers (integration base, not part of the reviewable change) Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…junyix-feat-serve-ops-routers (integration base, not part of the reviewable change) Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ster them with the legacy formatter The worker, metrics, profile, runtime-control and RL control serving classes with their constructors only; the moved handlers follow. The new Group B files, the Prometheus module among them, join legacy-files.txt before any formatter sees them, and the derived configs are regenerated. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…rving classes Generated move of the worker, metrics, profiling, runtime-control and RL control handlers from OpenAIServer into WorkerServing, MetricsServing, ProfileServing, RuntimeControlServing and RLServing, and of mount_metrics into serve/api/ops/metrics/prometheus.py as a function of the app. The three registration methods the routers replace are deleted; the server's own route table and wiring follow in the next commit. One declared edit: get_server_info reads the render resources the composition root built instead of rebuilding them per call. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…ers and serving objects The worker, metrics, control, profile, runtime-control and RL control routers join the router table, each created with make_router() and reading its serving object from app.state; the render routes and the Prometheus mount attach through the same table under five new ServerCapabilities fields (governor_queue, rl_control, runtime_control, perf_metrics, render_endpoints) that the composition root computes where it decided the same registrations before. The Prometheus mount stays last, so the JSON /metrics keeps winning and the instrumentator's middleware stays innermost. In OpenAIServer: the metrics state moves into MetricsServing (built for every role, as its collector loop ran for every role); the LLM branch of the role switch builds the worker, profile, runtime-control and RL serving objects, the resource governor (moved out of register_routes with its warning) and puts the render resources on app.state; the lifespan hands the energy monitor to the worker serving object and drives the stats machinery on the metrics one; register_routes keeps only the Responses routes; register_visual_gen_routes no longer registers /metrics, which the metrics router serves on VISUAL_GEN. mount_render_endpoints takes the RenderResources the composition root built once instead of the server. The six new router modules join the type-checked file set. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…tests on serving objects New tests under tests/unittest/serve/api/ops/ pin the constructor contract of the five serving classes, the objects they share with the composition root (the adjusted clock, the chat serving object, the render resources built once, the replay cache, the wake-up event, the metrics collector), the energy-monitor hand-over in the lifespan, the collector loop feeding the metrics serving object's buffer, the JSON /metrics winning over the Prometheus one, the control routers attaching only under their conditions, the runtime-control signature covering the declared route path, and which roles serve which routes. The tests that built OpenAIServer with object.__new__ to call a moved handler now build the serving object or attach its router: server info, profiling, runtime control, RL control, iteration stats, the clock calibration, the fingerprint cache and the startup metrics. The multi-frontend re-enactments gain the new serving objects; the clock-offset case moves onto WorkerServing. The sleep-mode test observes attach_routers instead of the deleted registration method, the governor tests build the server through the factory, and the render tests pass the RenderResources. The ratchet baseline loses the rewritten sites; the factory shuts down the metrics serving object's thread and its fake LLM answers what /server_info reads. The sleep/wakeup guide names the runtime-control package. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
…nd refresh the moved handlers' cross-references A perf-metrics configuration of the multimodal-encoder or embedding role gets no Prometheus mount today: no /prometheus/metrics, no second GET /metrics, no instrumentator middleware. Two tests pin both gates of that contract, the router-table entry (LLM only, last) and the perf_metrics capability the composition root computes, so widening either is a deliberate change. Docstrings and comments that named OpenAIServer.start_profile and stop_profile point at ProfileServing, the launcher-forwarding docstring names the router table, and the new ops test files say why the thread-leak check is off. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@coderabbitai summary
Description
Stacked on #21676 (chat serving class), #21556 (shared app assembly) and #21552 (VisualGen handler move): the first commits below belong to those PRs and disappear from this diff when they merge; this PR's own change is the last 4 commits. The integration base is two merge commits (
b9aae2f42b: #21556 into the head of #21676;463f37af4a: #21552 into that). Merge 2 carries two one-line test fixes that #21552 does not have yet and will need once it rebases onto #21556/#21447 and #21393/#21445:tests/unittest/_torch/visual_gen/test_trtllm_serve_endpoints.pyimportsServerArrivalTimeMiddlewarefromtensorrt_llm.serve.middleware.arrival_time(its old home inresponses_utilsis gone), and the_save_bytesentry oftests/unittest/serve/private_access_baseline.yamlis re-keyed totensorrt_llm/serve/api/visual_gen/serving_images.py/ImageServing.openai_image_generation. Nothing else in the merges is a change of its own.This PR moves the remaining operational endpoints off
OpenAIServerinto one serving class per endpoint group undertensorrt_llm/serve/api/ops/, each with a router created bymake_router()and attached through the router table (ROUTERSintensorrt_llm/serve/_base/routing.py):/health_generate,/v1/data_transceiver_state,/energy_metrics,/steady_clock_offset(GET and POST),/kv_cache_eventsand/server_info(ops/worker/); the JSON/metricswith the iteration-stats collector (ops/metrics/) and the Prometheus mount (ops/metrics/prometheus.py); the resource-governor routes (ops/control/, where theResourceGovernorkeeps registering its own routes);/start_profileand/stop_profile(ops/profile/); the runtime-control endpoints (ops/runtime_control/); and the deprecated RL control endpoints (ops/rl/). The render routes attach through the same table. Five newServerCapabilitiesfields (governor_queue,rl_control,runtime_control,perf_metrics,render_endpoints) carry the conditions the server used to evaluate inline; the composition root computes them inOpenAIServer._server_capabilities(), after the serving objects exist. Every serving class takes its dependencies as keyword-only constructor arguments without defaults, and the composition root hands it its own objects: the adjusted steady clock, the shared clock offset, the chat serving object, the render resources built once, the metrics collector, the wake-up event, the replay cache.Part of the trtllm-serve refactor (epic TRTLLM-17056, task TRTLLM-17267: move every endpoint implementation off
OpenAIServerinto serving classes and routers). Builds on "Add the trtllm-serve router skeleton and serve the system endpoints through it" (#21448), "Build the aggregated and disaggregated trtllm-serve apps with build_http_app and serve_uvicorn" (#21556), "Move the VisualGen handlers and modules into serve/api/visual_gen" (#21552), the embeddings, multimodal-encoder and tokenize serving classes (#21575), the completions serving class (#21640), the chat serving class (#21676) and the private-access lint (#21393, #21445).Behavior
No behavior change. Routes, route classes, conflict-relevant registration order and middleware order are pinned by the route and middleware-order snapshots, which are unchanged for every configuration. Known issues carried as they are: the
enable_iter_perf_statsgetattrdefault in/metrics;/start_profile,/stop_profile,/kv_cache_events,/steady_clock_offsetand the governor routes stay unauthenticated; the RL control endpoints stay deprecated and mutually exclusive with runtime control.Moves
Pure-move check
Pure-move check: base 463f37a, head 1ad84f4, tools dbe72ce65eef
ROUTERSServerCapabilitiesOpenAIServer.register_mm_encoder_routesOpenAIServer.mount_metricsOpenAIServer._iteration_stats_buffer_maxlenOpenAIServer._iteration_stats_collector_loopOpenAIServer._record_iteration_statOpenAIServer.get_iteration_statsOpenAIServer.start_profileOpenAIServer.stop_profileOpenAIServer._register_rl_control_routesOpenAIServer._require_rl_control_authOpenAIServer.release_memoryOpenAIServer.resume_memoryOpenAIServer.update_weightsOpenAIServer._register_runtime_control_routesOpenAIServer._call_runtime_memory_methodOpenAIServer._require_runtime_control_authOpenAIServer._runtime_control_error_responseOpenAIServer.runtime_memory_statusOpenAIServer.runtime_release_memoryOpenAIServer.runtime_resume_memoryOpenAIServer._live_tokens_per_blockOpenAIServer.data_transceiver_stateOpenAIServer.get_energy_metricsOpenAIServer.get_kv_cache_eventsOpenAIServer.get_server_infoOpenAIServer.get_steady_clock_offsetOpenAIServer.health_generateOpenAIServer.set_steady_clock_offsetOpenAIServer.__init__OpenAIServer._attach_role_routesOpenAIServer._register_launcher_forwarding_routesOpenAIServer.register_routesOpenAIServer.register_visual_gen_routesmount_render_endpoints_when_governor_queue_when_perf_metrics_when_render_endpoints_when_rl_control_when_runtime_controltensorrt_llm/serve/api/ops/control/api_router.py::*tensorrt_llm/serve/api/ops/metrics/api_router.py::*attach_routerMetricsServing.__init__MetricsServingtensorrt_llm/serve/api/ops/profile/api_router.py::*ProfileServing.__init__ProfileServingtensorrt_llm/serve/api/ops/rl/api_router.py::*RLServing.__init__RLServingtensorrt_llm/serve/api/ops/runtime_control/api_router.py::*RuntimeControlServing.__init__RuntimeControlServingtensorrt_llm/serve/api/ops/worker/api_router.py::*WorkerServing.__init__WorkerServing_ChatBackend.openai_chat_ChatBackendOpenAIServer._server_capabilitiesattach_routerReplaced
OpenAIServer.register_mm_encoder_routes->tensorrt_llm/serve/api/ops/metrics/api_router.py::get_iteration_stats: /metrics on the multimodal-encoder role comes from the metrics routerOpenAIServer._register_runtime_control_routes->tensorrt_llm/serve/api/ops/runtime_control/api_router.py::attach_router: the runtime-control routes come from their routerOpenAIServer._register_rl_control_routes->tensorrt_llm/serve/api/ops/rl/api_router.py::attach_router: the RL control routes come from their routerAdded
OpenAIServer._server_capabilities: builds the ServerCapabilities that decide which routers attach, the five new fields includedWorkerServing: serving class of the worker endpointsWorkerServing.__init__: explicit dependencies instead of server attributes, so missing wiring fails at construction_ChatBackend: local Protocol for the injected chat serving (no import of the chat package)_ChatBackend.openai_chat: the Protocol's one methodMetricsServing: serving class of /metrics, owns the iteration-stats machineryMetricsServing.__init__: explicit dependencies and the metrics state moved out of the composition root's constructorProfileServing: serving class of the profiling endpointsProfileServing.__init__: explicit dependenciesRuntimeControlServing: serving class of the runtime-control endpointsRuntimeControlServing.__init__: explicit dependencies, shared replay cacheRLServing: serving class of the RL control endpointsRLServing.__init__: explicit dependenciesattach_router: router-table entry point of the Prometheus mountattach_router: router-table entry point of the render routes_when_governor_queue: condition of the resource-governor router_when_rl_control: condition of the RL control router_when_runtime_control: condition of the runtime-control router_when_render_endpoints: condition of the render routes_when_perf_metrics: condition of the Prometheus mounttensorrt_llm/serve/api/ops/worker/api_router.py::*: new router moduletensorrt_llm/serve/api/ops/metrics/api_router.py::*: new router moduletensorrt_llm/serve/api/ops/control/api_router.py::*: new router moduletensorrt_llm/serve/api/ops/profile/api_router.py::*: new router moduletensorrt_llm/serve/api/ops/runtime_control/api_router.py::*: new router moduletensorrt_llm/serve/api/ops/rl/api_router.py::*: new router moduleRewritten
OpenAIServer.get_server_info(tensorrt_llm/serve/api/ops/worker/serving.py):RenderResources.from_server(self).fingerprint()->self._render_resources.fingerprint()Modified (declared)
OpenAIServer.__init__: metrics state moved to MetricsServing; energy-monitor hand-over and the stats machinery on the metrics serving object in the lifespan--- base
+++ head
@@ -54,10 +54,6 @@
self._media_load_executor = ThreadPoolExecutor(max_workers=media_load_workers, thread_name_prefix='trtllm_media_load')
BaseMediaIO.set_executor(self._media_load_executor)
log_allowed_local_media_path()
model_dir = Path(model)
if model_dir.exists() and model_dir.is_dir():
self.model = model_dir.name
@@ -85,7 +81,6 @@
self._iteration_stats_poll_interval: Optional[float] = 1.0 if multi_frontend_serving is not None and multi_frontend_serving.is_launcher else None
self._iteration_stats_collector_task = None
self._iteration_stats_wakeup_event = asyncio.Event()
self._adjusted_steady_clock = AdjustedSteadyClock()
self.energy_monitor = None
self.disagg_cluster_storage = None
@@ -110,14 +105,16 @@
except Exception as e:
logger.warning(f'Failed to initialize GPU energy monitoring: {e}')
self.energy_monitor = None
@@ -148,7 +145,7 @@
if self.resource_governor is not None:
self.resource_governor.close()
self.generator.shutdown()
async def validation_exception_handler(request, exc):
if request.url.path.startswith('/v1/messages'):
OpenAIServer._attach_role_routes: the method that holds the role switch: serving objects and the governor built in the role branches and put on app.state; the register_mm_encoder_routes call deleted (replaced by the router table); attach_routers with the full capabilities--- base
+++ head
@@ -1,6 +1,11 @@
def _attach_role_routes(self, app: FastAPI) -> None:
self.app = app
app.state.ops_serving_system = SystemServing(model=self.model, health_probe=self._health_probe)
if role_group_of(self.server_role) is RoleGroup.LLM:
self._register_launcher_forwarding_routes()
from tensorrt_llm.serve.api.openai.chat import api_router as chat_router
@@ -19,7 +24,6 @@
from tensorrt_llm.serve.api.openai.mm_encoder import api_router as mm_encoder_router
from tensorrt_llm.serve.api.openai.mm_encoder.serving import MMEncoderServing
setattr(app.state, mm_encoder_router.APP_STATE_KEY, MMEncoderServing(render_resources=self._render_resources, multimodal_server_config=self.multimodal_server_config, allow_request_chat_template=self.allow_request_chat_template, **self._generate_serving_kwargs()))
assert getattr(self.generator.args, 'encode_only', False), 'generator must be an encode_only=True LLM for the embedding server'
from tensorrt_llm.serve.api.openai.embeddings import api_router as embeddings_router
@@ -34,5 +38,30 @@
from tensorrt_llm.serve.api.openai.completions import api_router as completions_router
from tensorrt_llm.serve.api.openai.completions.serving import CompletionServing
setattr(app.state, completions_router.APP_STATE_KEY, CompletionServing(**self._generate_serving_kwargs()))
OpenAIServer.register_routes: the operational routes, the governor, the render routes and the Prometheus mount come from routers; the Responses routes stay until their router exists--- base
+++ head
@@ -1,27 +1,4 @@
def register_routes(self):
self.app.add_api_route('/v1/responses/{response_id}', self.openai_responses_get_response, methods=['GET'])
self.app.add_api_route('/v1/responses/{response_id}', self.openai_responses_delete_response, methods=['DELETE'])
OpenAIServer.register_visual_gen_routes: its GET /metrics registration is deleted (the metrics router serves /metrics on VISUAL_GEN), and its docstring no longer says /metrics is registered there--- base
+++ head
@@ -1,10 +1,8 @@
def register_visual_gen_routes(self):
"""Wire VisualGen serving for the VISUAL_GEN role.
self.media_storage_path = self._visual_gen_api.init_serving_state(self.app.state, generator=self.generator, model=self.model, adjusted_clock=self._adjusted_steady_clock)
logger.info(f'VisualGen media storage path: {self.media_storage_path}')
OpenAIServer._register_launcher_forwarding_routes: its docstring names the router table instead of the deleted registration method; the body is unchanged--- base
+++ head
@@ -1,5 +1,5 @@
def _register_launcher_forwarding_routes(self) -> None:
if self._launcher_forwarder is None:
return
for method, path in FORWARDED_ROUTES:
mount_render_endpoints: takes the RenderResources the composition root built instead of the server--- base
+++ head
@@ -1,4 +1,4 @@
-def mount_render_endpoints(app: FastAPI, server: Any, *, enabled: Optional[bool]=None) -> bool:
+def mount_render_endpoints(app: FastAPI, resources: RenderResources, *, enabled: Optional[bool]=None) -> bool:
"""Mount the render routes on a serving worker's app.
@@ -9,5 +9,5 @@
"""
if not (render_endpoints_enabled() if enabled is None else enabled):
return False
return True
ServerCapabilities: five new fields: governor_queue, rl_control, runtime_control, perf_metrics, render_endpoints--- base
+++ head
@@ -4,3 +4,8 @@
role: Optional[ServerRole]
role_group: RoleGroup
harmony_model: bool = False
ROUTERS: the RouterSpecs of the routers this PR adds--- base
+++ head
@@ -1 +1 @@
-ROUTERS: tuple[RouterSpec, ...] = (RouterSpec('tensorrt_llm.serve.api.ops.system.api_router', roles=ALL_ROLE_GROUPS), RouterSpec('tensorrt_llm.serve.api.openai.embeddings.api_router', roles=frozenset({RoleGroup.EMBEDDING})), RouterSpec('tensorrt_llm.serve.api.openai.mm_encoder.api_router', roles=frozenset({RoleGroup.MM_ENCODER})), RouterSpec('tensorrt_llm.serve.api.openai.tokenize.api_router', roles=frozenset({RoleGroup.LLM})), RouterSpec('tensorrt_llm.serve.api.openai.completions.api_router', roles=frozenset({RoleGroup.LLM})), RouterSpec('tensorrt_llm.serve.api.openai.chat.api_router', roles=frozenset({RoleGroup.LLM})), RouterSpec('tensorrt_llm.serve.api.anthropic.api_router', roles=frozenset({RoleGroup.LLM}), when=_anthropic_routes_enabled), RouterSpec('tensorrt_llm.serve.api.visual_gen.api_router', roles=frozenset({RoleGroup.VISUAL_GEN})))
+ROUTERS: tuple[RouterSpec, ...] = (RouterSpec('tensorrt_llm.serve.api.ops.system.api_router', roles=ALL_ROLE_GROUPS), RouterSpec('tensorrt_llm.serve.api.ops.worker.api_router', roles=frozenset({RoleGroup.LLM})), RouterSpec('tensorrt_llm.serve.api.ops.metrics.api_router', roles=ALL_ROLE_GROUPS - {RoleGroup.EMBEDDING}), RouterSpec('tensorrt_llm.serve.api.ops.control.api_router', roles=frozenset({RoleGroup.LLM}), when=_when_governor_queue), RouterSpec('tensorrt_llm.serve.api.openai.embeddings.api_router', roles=frozenset({RoleGroup.EMBEDDING})), RouterSpec('tensorrt_llm.serve.api.openai.mm_encoder.api_router', roles=frozenset({RoleGroup.MM_ENCODER})), RouterSpec('tensorrt_llm.serve.api.openai.tokenize.api_router', roles=frozenset({RoleGroup.LLM})), RouterSpec('tensorrt_llm.serve.api.openai.completions.api_router', roles=frozenset({RoleGroup.LLM})), RouterSpec('tensorrt_llm.serve.api.openai.chat.api_router', roles=frozenset({RoleGroup.LLM})), RouterSpec('tensorrt_llm.serve.api.anthropic.api_router', roles=frozenset({RoleGroup.LLM}), when=_anthropic_routes_enabled), RouterSpec('tensorrt_llm.serve.api.ops.profile.api_router', roles=frozenset({RoleGroup.LLM})), RouterSpec('tensorrt_llm.serve.api.ops.runtime_control.api_router', roles=frozenset({RoleGroup.LLM}), when=_when_runtime_control), RouterSpec('tensorrt_llm.serve.api.ops.rl.api_router', roles=frozenset({RoleGroup.LLM}), when=_when_rl_control), RouterSpec('tensorrt_llm.serve.api.visual_gen.api_router', roles=frozenset({RoleGroup.VISUAL_GEN})), RouterSpec('tensorrt_llm.serve.render.api_router', roles=frozenset({RoleGroup.LLM}), when=_when_render_endpoints), RouterSpec('tensorrt_llm.serve.api.ops.metrics.prometheus', roles=frozenset({RoleGroup.LLM}), when=_when_perf_metrics))
24 definitions moved verbatim; 0 differences; 1 rewritten (declared); 3 replaced; 26 added; 8 modified (declared); shim exports 0 names