Race Condition Between Filter() and onAddPod() Causes Double Counting

[aniket866]

Closes #2478
Race Condition Between Filter() and onAddPod() Causes Double Counting

Field	Details
Location	pkg/scheduler/scheduler.go
Description	In Filter(), the pod is deleted from the pod manager (line 956), then device usage is recalculated (line 959), then the pod is re-added (line 998). Meanwhile, the pod informer's onAddPod() (line 138) can fire concurrently and also call s.podManager.AddPod(). Since the PodManager uses UID-based keys, if onAddPod fires between line 956 and line 998, the pod could be added twice with different device allocations, leading to double-counted resource usage in the quota manager.
Reason	No mutex protects the Filter → delete → recalculate → re-add cycle against informer callbacks. The PodManager has its own internal lock, but the sequence of operations across Filter and informer is not atomic.

sequenceDiagram
    participant Filter as Filter()
    participant Informer as onAddPod()
    participant PM as PodManager
    participant QM as QuotaManager
    Filter->>PM: TakeAndDeletePod(pod)
    Note over Filter: Pod removed from cache
    Informer->>PM: AddPod(pod, nodeID, devices_old)
    PM-->>Informer: added=true
    Informer->>QM: AddUsage(pod, devices_old)
    Filter->>PM: AddPod(pod, nodeID, devices_new)
    PM-->>Filter: added=false already exists
    Note over QM: devices_old usage counted but never removed!

Loading

Summary by CodeRabbit

• Bug Fixes
  • Improved scheduler reliability by serializing concurrent operations for each pod.
  • Ensured pod locks are released after successful processing and retried when release initially fails.
  • Cleaned up unused lock state to prevent memory growth.
• Tests
  • Added coverage for concurrent pod handling, lock cleanup, successful binding, and lock-release retries.

Summary by CodeRabbit

• Bug Fixes

  • Improved scheduling reliability by serializing operations for each pod.
  • Prevented stale pod-lock entries from accumulating over time.
  • Safely ignores unlock attempts for unknown pods.

• Tests

  • Added coverage for locking, unlocking, concurrent access, sequential behavior, and lock cleanup.

[hami-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: aniket866
Once this PR has been reviewed and has the lgtm label, please assign fouof for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f939857d-0db5-4fc0-8636-3a10db81ec0b

📥 Commits

Reviewing files that changed from the base of the PR and between 7d49e48 and 074b752.

📒 Files selected for processing (2)

• pkg/scheduler/pod_lock.go
• pkg/scheduler/pod_lock_test.go

🚧 Files skipped from review as they are similar to previous changes (2)

• pkg/scheduler/pod_lock_test.go
• pkg/scheduler/pod_lock.go

---

📝 Walkthrough

Walkthrough

The scheduler adds a reference-counted lock manager keyed by Kubernetes pod UID. It serializes concurrent operations for each UID, removes unused lock entries, ignores unknown unlocks, and includes concurrency and cleanup tests.

Changes

Per-pod scheduler locking

Layer / File(s)	Summary
Reference-counted pod lock manager pkg/scheduler/pod_lock.go	Adds UID-keyed lock creation, reference counting, serialized locking, safe unlock handling, and cleanup after the final unlock.
Pod lock concurrency and cleanup validation pkg/scheduler/pod_lock_test.go	Tests blocking and sequential behavior across goroutines, lock reuse, and removal of unused lock entries.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested labels: enhancement

Suggested reviewers: mesutoezdil, wawa0210

Poem

A rabbit guards each pod by name,
One UID, one orderly game.
Locks wait their turn, then disappear,
When no paws remain near.
The tests hop twice and prove it clear.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Linked Issues check	⚠️ Warning	The changes add a per-pod lock manager, but do not show integration with Filter() or onAddPod() to prevent the race in issue #2478.	Integrate PodLockManager into Filter() and onAddPod(), including the required lock cleanup and retry handling.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the race condition and double-counting problem addressed by the linked issue.
Out of Scope Changes check	✅ Passed	The lock manager and its tests are directly related to preventing the per-pod race described in issue #2478.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/scheduler/pod_lock.go`:
- Line 1: Add the repository-standard Apache 2.0 license header before the
package declaration in pkg/scheduler/pod_lock.go (lines 1-1) and
pkg/scheduler/pod_lock_test.go (lines 1-1), leaving the remaining file contents
unchanged.

In `@pkg/scheduler/scheduler_test.go`:
- Around line 2330-2334: Update Scheduler.Bind’s successful Pods(...).Bind path
to call s.releaseAllDevices(node, current) before returning the successful
result. Ensure this release occurs after a successful bind and preserves the
existing error-path behavior so node locks are released before callers proceed
or retry.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 8232fac3-6232-46e4-901f-e9c060d990ed

📥 Commits

Reviewing files that changed from the base of the PR and between 3616313 and 7d49e48.

📒 Files selected for processing (4)

• pkg/scheduler/pod_lock.go
• pkg/scheduler/pod_lock_test.go
• pkg/scheduler/scheduler.go
• pkg/scheduler/scheduler_test.go

[codecov]

Codecov Report

❌ Patch coverage is 86.95652% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pkg/scheduler/pod_lock.go	86.95%	2 Missing and 1 partial ⚠️

Flag	Coverage Δ
unittests	64.30% <86.95%> (+0.09%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
pkg/scheduler/pod_lock.go	86.95% <86.95%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[maishivamhoo123]

@aniket866 this Pr is just a duplicate work of #1773 please go through this once.
Thank you!

[mesutoezdil]

podlockmanager is never wired into filter() or onaddpod(), scheduler.go is untouched. this does not actually fix the race in #2478, it just adds unused code