Skip to content
Closed
2 changes: 2 additions & 0 deletions pkg/device/pods.go
Original file line number Diff line number Diff line change
Expand Up @@ -69,9 +69,11 @@ func (m *PodManager) AddPod(pod *corev1.Pod, nodeID string, devices PodDevices)
"devices", devices,
)
} else {
m.pods[pod.UID].NodeID = nodeID
m.pods[pod.UID].Devices = devices
klog.V(5).InfoS("Pod devices updated",
"pod", klog.KRef(pod.Namespace, pod.Name),
"nodeID", nodeID,
"devices", devices,
)
}
Expand Down
32 changes: 29 additions & 3 deletions pkg/scheduler/scheduler.go
Original file line number Diff line number Diff line change
Expand Up @@ -892,11 +892,21 @@ func (s *Scheduler) Filter(args extenderv1.ExtenderArgs) (*extenderv1.ExtenderFi
"pod", klog.KObj(args.Pod),
"reason", "request does not contain full nodes",
"nodeNamesLen", nodeNamesLen(args.NodeNames))
if pi, ok := s.podManager.TakeAndDeletePod(args.Pod); ok {
s.quotaManager.RmUsage(args.Pod, pi.Devices)
var pi *device.PodInfo
if p, ok := s.podManager.TakeAndDeletePod(args.Pod); ok {
pi = p
s.quotaManager.RmUsage(args.Pod, p.Devices)
}
restorePod := func() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pingxin403 I’m concerned about restoring the previous allocation record when Filter fails. If the same Pod re-enters Filter, the existing record is likely a stale reservation from an earlier failed scheduling attempt. Restoring it may keep GPU and quota resources reserved indefinitely, causing another Pod leak. Could you clarify in which scenario the old allocation is still considered valid and should be restored?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the concern. Added Test_Filter_MultipleFailedAttemptsNoAccumulation in 76d95b2 to verify this. The test runs 3 consecutive failed Filter calls for the same pod and asserts that:

podManager always has exactly 2 pods (no accumulation of stale records)
The pod's nodeID and device allocation remain consistent across all attempts
The old allocation record represents the pod's current running allocation on the node. TakeAndDeletePod is called at the start of every Filter to temporarily remove it, and restorePod puts it back on failure. Since each Filter call does TakeAndDeletePod first, there's no accumulation — the record is removed before re-evaluation and restored only if the re-scheduling fails.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pingxin403 The test case isn't the real concern. The problem is the Pod's expected state during Filter: it shouldn't be holding any resources—HAMi just reserves them at this stage. So TakeAndDeletePod is correct, but there's no need to restore on failure. While normally safe, if the Pod is deleted during Filter and that deletion is processed before the restore, the re-added record will leak again.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification. Two answers:

  1. Why restore is needed — TakeAndDeletePod is not about "reserving" new resources; it's about temporarily removing the pod's existing running allocation from tracking so we can correctly evaluate resource availability for the new placement. If Filter fails, that pod is still running on the node with its GPUs actively in use. Without restorePod, the scheduler loses visibility into that pod's GPU consumption, which can lead to double-booking the same GPU for a different pod. The restore is simply putting back the record of what's already running.

  2. Deleted-pod race — Agreed, and now guarded via podLister.Pods(ns).Get(name) before restorePod. In production the lister is informer-backed: if the pod was deleted mid-Filter, Get returns NotFound and we skip restore + clean up the tentative cache entry. All three existing leak tests (Test_Filter_PodLeakOnFailure, Test_Filter_PodLeakOnPatchFailure, Test_Filter_MultipleFailedAttemptsNoAccumulation) still pass with no behavioral change since the lister is nil in tests (= no-op guard).

Force-pushed to 10aedc9.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second and last reminder @pingxin403: Answers must be written by human being. You can view the relevant rule here.

https://github.com/Project-HAMi/HAMi/blob/master/CONTRIBUTING.md#contribution-gates

"4. Review replies. The reply you post must be written by you and must address the specific point raised. Verbatim or canned AI replies, or replies that do not engage the comment, lead to the PR being closed."

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If Filter fails, that pod is still running on the node with its GPUs actively in use. 

@pingxin403 Can you tell me in which scenario will a Pod enter the Filter Phase in scheduler while still keep running on a Node?

if pi != nil {
s.quotaManager.AddUsage(args.Pod, pi.Devices)
s.podManager.AddPod(args.Pod, pi.NodeID, pi.Devices)
pi = nil
}
}
nodeUsage, _, failedNodes, err := s.getNodesUsage(args.NodeNames, args.Pod)
if err != nil {
restorePod()
s.recordScheduleFilterResultEvent(args.Pod, EventReasonFilteringFailed, "", err)
return nil, err
}
Expand All @@ -906,11 +916,18 @@ func (s *Scheduler) Filter(args extenderv1.ExtenderArgs) (*extenderv1.ExtenderFi
}
nodeScores, err := s.calcScore(nodeUsage, resourceReqs, args.Pod, failedNodes)
if err != nil {
restorePod()
err := fmt.Errorf("calcScore failed %v for pod %v", err, args.Pod.Name)
s.recordScheduleFilterResultEvent(args.Pod, EventReasonFilteringFailed, "", err)
return nil, err
}
if len((*nodeScores).NodeList) == 0 {
// Only restore the pod when there were candidate nodes to schedule on.
// If NodeNames was empty (no candidates), this is a stale entry that
// should be cleaned up.
if args.NodeNames != nil && len(*args.NodeNames) > 0 {
restorePod()
}
klog.V(4).InfoS("No available nodes meet the required scores",
"pod", args.Pod.Name)
s.recordScheduleFilterResultEvent(args.Pod, EventReasonFilteringFailed, "", fmt.Errorf("no available node, %d nodes do not meet", len(*args.NodeNames)))
Expand Down Expand Up @@ -942,10 +959,19 @@ func (s *Scheduler) Filter(args extenderv1.ExtenderArgs) (*extenderv1.ExtenderFi
err = util.PatchPodAnnotations(args.Pod, annotations)
if err != nil {
s.recordScheduleFilterResultEvent(args.Pod, EventReasonFilteringFailed, "", err)
// Revert the tentative AddPod/AddUsage for the new device
// assignment before restorePod re-adds the original one.
if added {
s.quotaManager.RmUsage(args.Pod, m.Devices)
}
s.podManager.DelPod(args.Pod)
if pi != nil {
restorePod()
} else {
// Pod was not in podManager before (first scheduling attempt),
// just clean up the tentative AddPod so it doesn't orphan
// an entry in the manager.
s.podManager.DelPod(args.Pod)
}
return nil, err
Comment thread
pingxin403 marked this conversation as resolved.
Resolved
}
successMsg := genSuccessMsg(len(*args.NodeNames), m.NodeID, nodeScores.NodeList)
Expand Down
155 changes: 155 additions & 0 deletions pkg/scheduler/scheduler_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1680,6 +1680,161 @@ func TestFilterTemplateNodesMissingRegisterAnnotation(t *testing.T) {
require.Equal(t, "node unregistered", res.FailedNodes["template-node-cold-zero"])
}

// Test_Filter_PodLeakOnFailure reproduces bug #1491.
// When Filter calls TakeAndDeletePod and then fails (no suitable node),
Comment thread
pingxin403 marked this conversation as resolved.
Resolved
// the pod was permanently removed from podManager, making the scheduler
// see more free GPUs than actually exist.
func Test_Filter_PodLeakOnFailure(t *testing.T) {
require.NoError(t, config.InitDevicesWithConfig(&config.Config{
NvidiaConfig: nvidia.NvidiaConfig{
ResourceCountName: "hami.io/gpu", ResourceMemoryName: "hami.io/gpumem",
ResourceCoreName: "hami.io/gpucores", DefaultGPUNum: 1,
},
}))
s := NewScheduler()

// Node with 2 GPUs
s.addNode("node1", &device.NodeInfo{
ID: "node1",
Node: &corev1.Node{ObjectMeta: metav1.ObjectMeta{Name: "node1"}},
Devices: map[string][]device.DeviceInfo{
nvidia.NvidiaGPUDevice: {
{ID: "device1", Index: 0, Count: 10, Devmem: 8000, Devcore: 100, Health: true, Type: nvidia.NvidiaGPUDevice},
{ID: "device2", Index: 1, Count: 10, Devmem: 8000, Devcore: 100, Health: true, Type: nvidia.NvidiaGPUDevice},
},
},
})

podA := &corev1.Pod{
ObjectMeta: metav1.ObjectMeta{UID: "uid-leak-a", Name: "pod-a", Namespace: "ns"},
Spec: corev1.PodSpec{Containers: []corev1.Container{{
Name: "c", Resources: corev1.ResourceRequirements{
Limits: corev1.ResourceList{"hami.io/gpu": *resource.NewQuantity(1, resource.BinarySI)},
},
}}},
}
podB := &corev1.Pod{
ObjectMeta: metav1.ObjectMeta{UID: "uid-leak-b", Name: "pod-b", Namespace: "ns"},
Spec: corev1.PodSpec{Containers: []corev1.Container{{
Name: "c", Resources: corev1.ResourceRequirements{
Limits: corev1.ResourceList{"hami.io/gpu": *resource.NewQuantity(1, resource.BinarySI)},
},
}}},
}

devA := device.PodDevices{
nvidia.NvidiaGPUDevice: device.PodSingleDevice{
{
{Idx: 0, UUID: "device1", Usedmem: 1000, Usedcores: 10},
{Idx: 1, UUID: "device2", Usedmem: 1000, Usedcores: 10},
},
},
}
devB := device.PodDevices{
nvidia.NvidiaGPUDevice: device.PodSingleDevice{
{
{Idx: 0, UUID: "device1", Usedmem: 1000, Usedcores: 10},
{Idx: 1, UUID: "device2", Usedmem: 1000, Usedcores: 10},
},
},
}

s.podManager.AddPod(podA, "node1", devA)
s.podManager.AddPod(podB, "node1", devB)

// Both pods in cache
require.Equal(t, 2, len(s.podManager.ListPodsInfo()))

// Trigger a re-schedule for podA that fails.
// Pass the same node but both GPUs are fully occupied by podA + podB,
// so no device fits for podA's request.
result, err := s.Filter(extenderv1.ExtenderArgs{Pod: podA, NodeNames: &[]string{"node1"}})
// Filter returns nil error with FailedNodes set to indicate no node fits
require.NoError(t, err)
require.NotNil(t, result, "Filter result should not be nil")
require.NotEmpty(t, result.FailedNodes, "FailedNodes should report node1 as not fit")

// Before the fix, podA was taken by TakeAndDeletePod and never re-added,
// leaking its device usage. restorePod() now ensures it is re-added on
// error paths. Verify it's still in the cache.
pods := s.podManager.ListPodsInfo()
assert.Equal(t, 2, len(pods), "podA should remain in podManager after failed Filter")
_, ok := s.podManager.GetPod(podA)
assert.Equal(t, true, ok, "podA should be found in podManager after failed Filter")
}

// Test_Filter_PodLeakOnPatchFailure verifies that when Filter succeeds
// in finding a node but fails to patch annotations, the pod is properly
// restored to podManager with its original nodeID and device state
// (regression for mesutoezdil/gemini review feedback on #2044).
func Test_Filter_PodLeakOnPatchFailure(t *testing.T) {
require.NoError(t, config.InitDevicesWithConfig(&config.Config{
NvidiaConfig: nvidia.NvidiaConfig{
ResourceCountName: "hami.io/gpu", ResourceMemoryName: "hami.io/gpumem",
ResourceCoreName: "hami.io/gpucores", DefaultGPUNum: 1,
},
}))

s := NewScheduler()

// Node with 1 GPU
s.addNode("node1", &device.NodeInfo{
ID: "node1",
Node: &corev1.Node{ObjectMeta: metav1.ObjectMeta{Name: "node1"}},
Devices: map[string][]device.DeviceInfo{
nvidia.NvidiaGPUDevice: {
{ID: "GPU-1", Index: 0, Count: 10, Devmem: 8000, Devcore: 100, Health: true, Type: nvidia.NvidiaGPUDevice},
},
},
})

// Set up empty fake client so PatchPodAnnotations has a non-nil client
// but fails because no pod exists in the tracker.
fakeClient := fake.NewSimpleClientset()
client.KubeClient = fakeClient
s.kubeClient = fakeClient

pod := &corev1.Pod{
ObjectMeta: metav1.ObjectMeta{
UID: types.UID("test-patch-fail-uid"),
Name: "test-pod",
Namespace: "default",
},
Spec: corev1.PodSpec{Containers: []corev1.Container{{
Name: "c", Resources: corev1.ResourceRequirements{
Limits: corev1.ResourceList{"hami.io/gpu": resource.MustParse("1")},
},
}}},
}

dev := device.PodDevices{
nvidia.NvidiaGPUDevice: device.PodSingleDevice{
{
{Idx: 0, UUID: "GPU-1", Usedmem: 1000, Usedcores: 10},
},
},
}

s.podManager.AddPod(pod, "node1", dev)
s.quotaManager.AddUsage(pod, dev)

require.Equal(t, 1, len(s.podManager.ListPodsInfo()),
"pod should be in podManager before Filter")

// Filter will find node1 suitable, call PatchPodAnnotations on a pod
// that doesn't exist in the fake client, triggering a patch error.
_, err := s.Filter(extenderv1.ExtenderArgs{Pod: pod, NodeNames: &[]string{"node1"}})
require.Error(t, err, "Filter should fail when PatchPodAnnotations fails")

// After failure, pod must be restored to its original node + devices.
pi, ok := s.podManager.GetPod(pod)
assert.Assert(t, ok, "pod should still be in podManager after failed PatchPodAnnotations")
assert.Equal(t, "node1", pi.NodeID, "nodeID should be restored to original")
assert.Assert(t, len(pi.Devices[nvidia.NvidiaGPUDevice]) == len(dev[nvidia.NvidiaGPUDevice]),
"device group count should match original")
assert.DeepEqual(t, dev[nvidia.NvidiaGPUDevice], pi.Devices[nvidia.NvidiaGPUDevice])
}

func Test_Scheduler_Issue1368_TerminatingPodRetainsCache(t *testing.T) {
s := NewScheduler()

Expand Down