Skip to content

fix(nemo-deployments): release docker gpu for missing one-shot containers - #948

Merged
ironcommit merged 1 commit into
mainfrom
docker-missing-container-gpu-release/rsadler
Jul 29, 2026
Merged

fix(nemo-deployments): release docker gpu for missing one-shot containers#948
ironcommit merged 1 commit into
mainfrom
docker-missing-container-gpu-release/rsadler

Conversation

@ironcommit

@ironcommit ironcommit commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary by CodeRabbit

  • Bug Fixes
    • Fixed GPU resource handling when one-shot deployments lose their container.
    • One-shot deployments now correctly report a failed status and release allocated GPU resources.
    • Preserved existing behavior for deployments configured to remain lost when containers are missing.

…ners

Signed-off-by: Ryan S <267728323+ironcommit@users.noreply.github.com>
@ironcommit
ironcommit requested a review from tylersbray July 28, 2026 17:17
@ironcommit
ironcommit requested review from a team as code owners July 28, 2026 17:17
@github-actions github-actions Bot added the fix label Jul 28, 2026
@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4881619b-915c-4475-bb47-c7ae0759f02d

📥 Commits

Reviewing files that changed from the base of the PR and between e13178a and cd0816e.

📒 Files selected for processing (2)
  • plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/backend.py
  • plugins/nemo-deployments/tests/unit/backends/docker/test_backend_mocked.py

📝 Walkthrough

Walkthrough

DockerDeploymentBackend.read_status now releases GPU allocations when one-shot deployments lose their containers, reuses the deployment key across exit paths, and adds tests for release and non-release cases.

Changes

Docker deployment status cleanup

Layer / File(s) Summary
Status handling and regression coverage
plugins/nemo-deployments/src/nemo_deployments_plugin/backends/docker/backend.py, plugins/nemo-deployments/tests/unit/backends/docker/test_backend_mocked.py
read_status computes dep_key before lookup, releases GPUs for missing Never and OnFailure containers, reuses the key for exit handling, and verifies terminal status and release behavior in mocked tests.

Possibly related PRs

Suggested labels: test

Suggested reviewers: tylersbray, benmccown

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: releasing Docker GPU resources when one-shot containers are missing.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch docker-missing-container-gpu-release/rsadler

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor
Suite Lines Covered Line Rate Branch Rate
Unit Tests 27565/35320 78.0% 62.5%
Integration Tests 16091/34032 47.3% 19.8%

@tylersbray tylersbray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice.

@ironcommit
ironcommit added this pull request to the merge queue Jul 29, 2026
Merged via the queue into main with commit fd377b4 Jul 29, 2026
60 checks passed
@ironcommit
ironcommit deleted the docker-missing-container-gpu-release/rsadler branch July 29, 2026 02:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants