fix(thunderagent): cherry-pick support topology-only wrapped backends (#12175) - #12595
Conversation
Signed-off-by: cpakkamisaac-sae <cpakkamisaac@nvidia.com> (cherry picked from commit ab04a7f) Signed-off-by: Dan Gil <dagil@nvidia.com>
github-actions
Bot
added
fix
documentation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bad896aca9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| help="Comma-separated list of endpoint types to enable. Options: " | ||
| "'chat', 'completions', or 'none'. Use 'completions' for models " | ||
| "without chat templates. Use 'none' for topology-only workers " | ||
| "fronted by another Dynamo service.", |
There was a problem hiding this comment.
none in the unified Rust worker
When this shared option is used by a unified backend, such as python -m dynamo.tokenspeed ... --endpoint-types none, the value bypasses this Python parser and is forwarded through WorkerConfig.from_runtime_config to _backend.Worker; lib/backend-common/src/worker.rs::parse_endpoint_types has no none arm and rejects registration with unknown endpoint type 'none'. Add equivalent parsing and validation in the Rust worker before advertising this option to every DynamoRuntimeArgGroup consumer.
AGENTS.md reference: AGENTS.md:L17-L23
Useful? React with 👍 / 👎.
| if "none" in types: | ||
| if len(types) > 1: | ||
| raise ValueError( | ||
| "Endpoint type 'none' cannot be combined with other endpoint types" | ||
| ) | ||
| return ModelType.Empty |
There was a problem hiding this comment.
For an SGLang worker launched with both --endpoint-types none and the supported --use-sglang-tokenizer mode, this returns ModelType.Empty, but _register_model_with_runtime_config subsequently rewrites every non-embedding type to ModelType.Chat. The worker therefore still advertises a public chat surface, allowing frontend traffic—including the session-final request this change is meant to protect—to bypass ThunderAgent; preserve Empty in that override or reject the incompatible option combination.
Useful? React with 👍 / 👎.
| if "none" in types: | ||
| if len(types) > 1: | ||
| raise ValueError( | ||
| "Endpoint type 'none' cannot be combined with other endpoint types" | ||
| ) | ||
| return ModelType.Empty |
There was a problem hiding this comment.
When a vLLM worker uses --endpoint-types none with LoRA enabled, this hides only its base-model card. Loading an adapter later reaches VllmWorkerHandler.load_lora, where decode and aggregated workers unconditionally publish the adapter as ModelType.Chat | ModelType.Completions instead of honoring config.endpoint_types; requests for that adapter can therefore route directly to the wrapped backend and bypass ThunderAgent. Propagate the empty surface to dynamically registered LoRA cards as well.
Useful? React with 👍 / 👎.
![dynamo-review-agent[bot]](https://avatars.githubusercontent.com/u/201626793?s=60&v=4){kind=small}
|
3 participants
Summary
Cherry-pick of #12175 (
ab04a7fc05, merged to main 2026-07-31) ontorelease/1.4.0.Fixes NVBug 6550880 / DYN-3728, a P0 on the 1.4.0 board: the session-final close request bypasses
thunderagent_routerand is forwarded straight to the engine.QA reported the fix as present on main but absent from the release branch. Confirmed:
ab04a7fc05is not an ancestor ofrelease/1.4.0, and no cherry-pick had been opened in the three days since it merged.Scope
Applies cleanly, no conflicts. Identical to the main commit:
components/src/dynamo/common/utils/endpoint_types.py+15/-4components/src/dynamo/common/configuration/groups/runtime_args.py+4/-1components/src/dynamo/common/utils/tests/test_endpoint_types.py+36 (new)components/src/dynamo/thunderagent_router/README.md+7Original authorship preserved;
-xrecords the source commit.Validation
Carries the upstream unit tests added in #12175. No behavioural delta from the main commit.