fix(router): honor Kubernetes worker namespace suffix - #12160
Conversation
|
👋 Hi cpakkamisaac-sae! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
WalkthroughChangesRouter namespace resolution
Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@components/src/dynamo/router/tests/test_args.py`:
- Line 12: Update the module-level pytestmark declaration in test_args.py to
include the required pytest.mark.router marker alongside the existing pre_merge,
unit, and gpu_0 markers.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 13c3c42d-b50b-4a72-9e0f-59a1b64aed5b
📒 Files selected for processing (2)
components/src/dynamo/router/args.pycomponents/src/dynamo/router/tests/test_args.py
|
Hi @ishandhanani, could you please review this PR when you have a chance? It fixes the namespace mismatch described in #12016 by honoring the Kubernetes worker-namespace suffix for router endpoints used by the ThunderAgent demo. I added unit coverage and validated the fix on AKS against Dynamo Platform v1.3.0. |
ishandhanani
left a comment
There was a problem hiding this comment.
Some nits. Pls ping me on slack when this is ready
|
@ishandhanani updated based on your recommendations. Changes made:
I also revalidated the behavior end-to-end on AKS with the ThunderAgent deployment from #12016: the router resolved the suffixed worker namespace, discovered the vLLM worker, and both |
|
/ok-to-test 452b571 |
This comment has been minimized.
This comment has been minimized.
Signed-off-by: cpakkamisaac-sae <cpakkamisaac@nvidia.com>
Signed-off-by: cpakkamisaac-sae <cpakkamisaac@nvidia.com>
Signed-off-by: cpakkamisaac-sae <cpakkamisaac@nvidia.com>
Head branch was pushed to by a user without write access
452b571 to
9a53ad2
Compare
|
/ok-to-test 9a53ad2 |
Overview:
Fix standalone router worker-endpoint resolution for Kubernetes deployments that publish worker-class components in a suffixed Dynamo namespace during rolling-update-aware DGD rendering.
Details:
get_worker_namespace()soDYN_NAMESPACE_WORKER_SUFFIXis included when the operator injects it into worker-class router pods.config.namespaceas the baseDYN_NAMESPACE, so the standalone router continues to serve its own endpoints in the DGD namespace while discovering backend workers in the suffixed worker namespace.DYN_NAMESPACE, andDYN_NAMESPACE_WORKER_SUFFIX.routerpytest marker to the new router test module.Validation:
PYTHONPATH=components/src UV_CACHE_DIR=/tmp/codex-uv-cache uv run --no-project --with pytest --with pytest-asyncio python -m pytest -c /dev/null components/src/dynamo/router/tests/test_args.py components/src/dynamo/router/tests/test_standalone_router.pyaks-dynamo-thunderagentin resource groupcpakkamisaac-sae-msft-east(eastus) using aStandard_NC4as_T4_v3GPU node pool and modelQwen/Qwen3-0.6B.dynamo-platform-1.2.1chart: the router watcheddynamo-system-thunderagent-demo.backend.generatewhile the vLLM worker published underdynamo-system-thunderagent-demo-588c6644.backend.generate; the second same-session ThunderAgent request returned HTTP 500 with no routable endpoints.dynamo-platform-1.2.1: router discovery watcheddynamo-system-thunderagent-demo-03302af8.backend.generate, the DGD became Ready, and first plus second same-session/v1/chat/completionscalls returned HTTP 200.v1.3.0/ chartdynamo-platform-1.3.0, applying the large Grove/KAI CRDs with server-side apply where needed. The DGD became Ready, router logs showed discovery againstdynamo-system-thunderagent-demo-03302af8.backend.generate,/v1/modelsreturned HTTP 200, and both first plus second same-session/v1/chat/completionscalls returned HTTP 200.v1.3.0still does not listrouteras a validDynamoComponentDeployment.spec.typeenum value (frontend,worker,prefill,decode,planner,eppare valid), so the demo manifest still needsThunderAgentRouterrepresented ascomponentType: worker. That is separate from the router worker-namespace bug fixed here.Where should the reviewer start?
Start with
components/src/dynamo/router/args.py, specificallyDynamoRouterConfig.validate(), then reviewcomponents/src/dynamo/router/tests/test_args.pyfor the namespace-resolution cases.Related Issues
🔗 This PR is linked to an issue: