fix(scheduler): guard against zero-value division in ComputeScore (#1… - #1820
Conversation
…oject-HAMi#1780) ComputeScore and ComputeDefaultScore divide by device capacity fields (Count, Totalcore, Totalmem) without zero-checks. When any field is 0, float32 division produces NaN/Inf, which breaks sort.Sort's ordering requirement and causes an index-out-of-range panic — crashing the entire scheduler extender and halting all GPU scheduling cluster-wide. Return score 0 for devices/nodes with zero capacity so the scheduler can gracefully mark them as unschedulable instead of panicking. Signed-off-by: lin121291 <4jp33f9e@gmail.com>
|
Welcome @lin121291! It looks like this is your first PR to Project-HAMi/HAMi 🎉 |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces safety checks in the GPU and node policy scoring functions to prevent division-by-zero errors when device capacity metrics are zero. It also includes comprehensive unit tests covering these edge cases. A review comment suggests an additional nil-pointer check for the device object to further enhance the robustness of the scoring logic.
|
please resolve the AI comments |
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 1 file with indirect coverage changes 🚀 New features to boost your workflow:
|
Signed-off-by: lin121291 <4jp33f9e@gmail.com>
1741e5b to
e50c98a
Compare
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: lin121291, Shouren The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
…oject-HAMi#1… (Project-HAMi#1820) * fix(scheduler): guard against zero-value division in ComputeScore (Project-HAMi#1780) ComputeScore and ComputeDefaultScore divide by device capacity fields (Count, Totalcore, Totalmem) without zero-checks. When any field is 0, float32 division produces NaN/Inf, which breaks sort.Sort's ordering requirement and causes an index-out-of-range panic — crashing the entire scheduler extender and halting all GPU scheduling cluster-wide. Return score 0 for devices/nodes with zero capacity so the scheduler can gracefully mark them as unschedulable instead of panicking. Signed-off-by: lin121291 <4jp33f9e@gmail.com> * fix(scheduler): add nil check for Device pointer in ComputeScore Signed-off-by: lin121291 <4jp33f9e@gmail.com> --------- Signed-off-by: lin121291 <4jp33f9e@gmail.com>
3 participants
What type of PR is this?
bug
What this PR does / why we need it:
ComputeScore(gpu_policy.go:78-80) andComputeDefaultScore(node_policy.go:90-92) divide by device capacity fields (
Count,Totalcore,Totalmem) without zero-checks. When any field is 0,float32 division produces NaN/Inf, which breaks
sort.Sort'stotal-ordering requirement — causing an index-out-of-range panic
that kills the entire scheduler extender process.
Since the offending pod stays Pending, kube-scheduler retries it on
every restart, creating a CrashLoopBackOff loop that halts all GPU
scheduling cluster-wide.
This PR adds zero-value guards to both functions so they return
score 0 instead of panicking, letting the scheduler gracefully
mark devices/nodes as unschedulable.
Note:
fitInDevicesalready guards against "request more GPUs thanthe node has" at score.go:65, but that check runs after
ComputeScore. The trigger is a device reporting zero-valuecapacity, not an empty device list.
Which issue(s) this PR fixes:
Fixes #1780
Special notes for your reviewer:
ComputeDefaultScorein node_policy.go has the same unguardeddivision pattern — fixed in this PR as well
Does this PR introduce a user-facing change?:
Scheduler no longer panics when a device reports zero-value capacity
fields. Pods that cannot be satisfied are correctly marked as
unschedulable instead of crashing the scheduler.