Skip to content

fix(operator): remove lws worker gpu pre-validation - #11115

Merged
tmonty12 merged 1 commit into
ai-dynamo:mainfrom
panpan0000:fix/lws-worker-validation-removal
Jul 6, 2026
Merged

fix(operator): remove lws worker gpu pre-validation#11115
tmonty12 merged 1 commit into
ai-dynamo:mainfrom
panpan0000:fix/lws-worker-validation-removal

Conversation

@panpan0000

@panpan0000 panpan0000 commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Overview:

Summary

  • remove the LWS worker check that requires nvidia.com/gpu
  • let Kubernetes and device admission decide whether the rendered pod is valid
  • keep the existing main container validation and leave NVIDIA-specific GMS/DRA paths unchanged

Details:

  • delete the hardcoded GPU limit check from generateWorkerPodTemplateSpec()
  • add a regression test that renders an LWS worker pod with valid main container command/args and no GPU resource

Validation

  • go test ./internal/controller -run TestGenerateWorkerPodTemplateSpecDoesNotRequireGPUResource -count=1
  • go test ./internal/controller -run Test_generateDeployment_Strategy -count=1

Where should the reviewer start?

  • deploy/operator/internal/controller/dynamocomponentdeployment_controller.go
  • deploy/operator/internal/controller/dynamocomponentdeployment_controller_test.go

Related Issues

🔗 This PR is linked to an issue:

Summary by CodeRabbit

  • Bug Fixes
    • Worker pod templates can now be generated successfully even when the main container does not specify GPU resource limits.
    • Improved compatibility for deployments that use CPU-only resource settings.

Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
@panpan0000
panpan0000 requested a review from a team as a code owner July 1, 2026 09:28
@copy-pr-bot

copy-pr-bot Bot commented Jul 1, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@panpan0000
panpan0000 temporarily deployed to external_collaborator July 1, 2026 09:28 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

👋 Hi panpan0000! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added fix external-contribution Pull request is from an external contributor deployment::k8s Relates to dynamo deployment in kubernetes labels Jul 1, 2026
@coderabbitai

coderabbitai Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The controller's generateWorkerPodTemplateSpec function no longer validates that the main container has an nvidia.com/gpu resource limit set, removing the prior error-returning check. A corresponding unit test was added to verify pod template generation succeeds without GPU resources.

Changes

GPU Requirement Removal

Layer / File(s) Summary
Remove GPU limit validation
deploy/operator/internal/controller/dynamocomponentdeployment_controller.go
The check that returned an error when the main container's nvidia.com/gpu limit was missing or zero is removed from generateWorkerPodTemplateSpec, which now proceeds directly to return the template after main-container validation.
Test coverage for GPU-less pod templates
deploy/operator/internal/controller/dynamocomponentdeployment_controller_test.go
A new test, TestGenerateWorkerPodTemplateSpecDoesNotRequireGPUResource, builds a DynamoComponentDeployment with only CPU resources and verifies generateWorkerPodTemplateSpec succeeds, returning a non-nil template with correct role label and main container name.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Related PRs: None identified.

Suggested labels: None

Suggested reviewers: None

Poem:
A rabbit hopped through GPU gates,
No limit checked, no error waits.
Worker pods now run on less,
CPU alone, no more distress.
Tests confirm the change is neat—
Hop along, review complete! 🐇

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly states the core change: removing the LWS worker GPU pre-validation.
Description check ✅ Passed The description covers the required sections and includes a related issue reference and validation notes.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies"


Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
deploy/operator/internal/controller/dynamocomponentdeployment_controller_test.go (1)

3459-3468: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Consider asserting the resulting container has no GPU resource set.

The test confirms generateWorkerPodTemplateSpec succeeds without a GPU limit on input, but doesn't assert that the generated container's Resources.Limits/Requests actually omit nvidia.com/gpu. Since the test's purpose is specifically to lock in the "GPU not required" behavior, checking this closes the loop against future regressions (e.g., some other path force-defaulting a GPU limit).

♻️ Suggested additional assertion
 	require.Equal(t, commonconsts.MainContainerName, got.Spec.Containers[0].Name)
+	_, hasGPULimit := got.Spec.Containers[0].Resources.Limits[corev1.ResourceName(commonconsts.KubeResourceGPUNvidia)]
+	require.False(t, hasGPULimit, "worker pod template should not require nvidia.com/gpu")
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@deploy/operator/internal/controller/dynamocomponentdeployment_controller_test.go`
around lines 3459 - 3468, The test for generateWorkerPodTemplateSpec currently
verifies success and labels, but it does not lock in the “no GPU required”
behavior. Update the test to assert the returned container from
got.Spec.Containers[0] has no nvidia.com/gpu entry in either Resources.Limits or
Resources.Requests, so future changes cannot silently add a GPU resource.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In
`@deploy/operator/internal/controller/dynamocomponentdeployment_controller_test.go`:
- Around line 3459-3468: The test for generateWorkerPodTemplateSpec currently
verifies success and labels, but it does not lock in the “no GPU required”
behavior. Update the test to assert the returned container from
got.Spec.Containers[0] has no nvidia.com/gpu entry in either Resources.Limits or
Resources.Requests, so future changes cannot silently add a GPU resource.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d031715b-eb01-4d46-89de-ddd77d36384a

📥 Commits

Reviewing files that changed from the base of the PR and between 076ce9d and 6ad355e.

📒 Files selected for processing (2)
  • deploy/operator/internal/controller/dynamocomponentdeployment_controller.go
  • deploy/operator/internal/controller/dynamocomponentdeployment_controller_test.go
💤 Files with no reviewable changes (1)
  • deploy/operator/internal/controller/dynamocomponentdeployment_controller.go

@rmccorm4

rmccorm4 commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution @panpan0000.

@sttts @tmonty12 @julienmancuso to help review

@rmccorm4

rmccorm4 commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

/ok to test 6ad355e

@tmonty12
tmonty12 merged commit 3faef7a into ai-dynamo:main Jul 6, 2026
83 checks passed
@panpan0000

Copy link
Copy Markdown
Contributor Author

thx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deployment::k8s Relates to dynamo deployment in kubernetes external-contribution Pull request is from an external contributor fix size/M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants