fix(plugin): release node lock on getAllocateResponse failure - #2262
fix(plugin): release node lock on getAllocateResponse failure#2262manmathbh wants to merge 1 commit into
Conversation
When getAllocateResponse fails in the non-MIG Allocate path, PodAllocationFailed was not called, leaking the node lock. Every other error path in this function correctly releases the lock before returning. Signed-off-by: Manmath Hatte <manmathcode@gmail.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: manmathbh The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughChangesNVIDIA allocation error handling
Estimated code review effort: 1 (Trivial) | ~5 minutes Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Fixes a node-lock leak in the NVIDIA device plugin Allocate() non-MIG path by ensuring the failure path for getAllocateResponse() performs the same cleanup as other error exits (releasing the node lock via PodAllocationFailed(...)).
Changes:
- Add missing
PodAllocationFailed(nodename, current, NodeLockNvidia)call whengetAllocateResponse(...)fails in the non-MIG allocation branch. - Align the non-MIG
getAllocateResponseerror handling with the existing MIG branch and other non-MIG error exits to prevent stalehami.io/mutex.lockon the node.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
pls answer Shouren's question on #2214 first, if kubelet retries Allocate after an error then releasing the lock here may be wrong and the existing 7 call sites have the same problem, this needs a decision before adding an 8th. also cotishq had offered to fix that issue, pls sync w/ them. |
What type of PR is this?
/kind bug
What this PR does / why we need it:
getAllocateResponse failure path in the non-MIG Allocate loop was missing
a PodAllocationFailed call. Every other error exit in the same function
calls it, this was the only one that skipped it.
Node lock stays set on the node after the failure. Next GPU request on
that node either hangs or fails.
Which issue(s) this PR fixes:
Fixes #2214
Special notes for your reviewer:
Only affects the non-MIG path. getAllocateResponse isn't called in the
MIG branch.
Does this PR introduce a user-facing change?:
NONESummary by CodeRabbit