Repository navigation
fix: restore pod device usage in scheduler Filter on error paths (#1491) #2044
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 6 commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
3c679ae
fix: restore pod device usage in scheduler Filter on error paths (#1491)
3bfe898
fix: make restorePod idempotent (reset removed flag after restore)
27bc2ea
fix: revert tentative AddUsage before restorePod on PatchPodAnnotatio…
e75cb89
fix #2044 review: address gemini+mesutoezdil rollback bugs
d1bb2a2
fix: remove extra closing brace after conflict resolution
d006325
refactor: replace removed flag with pi != nil check in restorePod
76d95b2
test: verify multiple failed Filter attempts do not accumulate stale …
1fc1972
fix(scheduler): add podLister guard to restorePod for deleted-pod rac…
10aedc9
Merge remote-tracking branch 'upstream/master' into fix/filter-pod-leak
42210ce
fix: fix rangeint lint error in scheduler_test.go
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pingxin403 I’m concerned about restoring the previous allocation record when Filter fails. If the same Pod re-enters Filter, the existing record is likely a stale reservation from an earlier failed scheduling attempt. Restoring it may keep GPU and quota resources reserved indefinitely, causing another Pod leak. Could you clarify in which scenario the old allocation is still considered valid and should be restored?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the concern. Added Test_Filter_MultipleFailedAttemptsNoAccumulation in 76d95b2 to verify this. The test runs 3 consecutive failed Filter calls for the same pod and asserts that:
podManager always has exactly 2 pods (no accumulation of stale records)
The pod's nodeID and device allocation remain consistent across all attempts
The old allocation record represents the pod's current running allocation on the node. TakeAndDeletePod is called at the start of every Filter to temporarily remove it, and restorePod puts it back on failure. Since each Filter call does TakeAndDeletePod first, there's no accumulation — the record is removed before re-evaluation and restored only if the re-scheduling fails.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pingxin403 The test case isn't the real concern. The problem is the Pod's expected state during Filter: it shouldn't be holding any resources—HAMi just reserves them at this stage. So
TakeAndDeletePodis correct, but there's no need to restore on failure. While normally safe, if the Pod is deleted during Filter and that deletion is processed before the restore, the re-added record will leak again.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the clarification. Two answers:
Why restore is needed — TakeAndDeletePod is not about "reserving" new resources; it's about temporarily removing the pod's existing running allocation from tracking so we can correctly evaluate resource availability for the new placement. If Filter fails, that pod is still running on the node with its GPUs actively in use. Without restorePod, the scheduler loses visibility into that pod's GPU consumption, which can lead to double-booking the same GPU for a different pod. The restore is simply putting back the record of what's already running.
Deleted-pod race — Agreed, and now guarded via podLister.Pods(ns).Get(name) before restorePod. In production the lister is informer-backed: if the pod was deleted mid-Filter, Get returns NotFound and we skip restore + clean up the tentative cache entry. All three existing leak tests (Test_Filter_PodLeakOnFailure, Test_Filter_PodLeakOnPatchFailure, Test_Filter_MultipleFailedAttemptsNoAccumulation) still pass with no behavioral change since the lister is nil in tests (= no-op guard).
Force-pushed to 10aedc9.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Second and last reminder @pingxin403: Answers must be written by human being. You can view the relevant rule here.
https://github.com/Project-HAMi/HAMi/blob/master/CONTRIBUTING.md#contribution-gates
"4. Review replies. The reply you post must be written by you and must address the specific point raised. Verbatim or canned AI replies, or replies that do not engage the comment, lead to the PR being closed."
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pingxin403 Can you tell me in which scenario will a Pod enter the Filter Phase in scheduler while still keep running on a Node?