Fix: adopt to gangScheduling feature gates on k8s v1.36+ - #2206
Conversation
Signed-off-by: archlitchi <mengxuan.li@dynamia.ai>
Signed-off-by: archlitchi <mengxuan.li@dynamia.ai>
Signed-off-by: archlitchi <mengxuan.li@dynamia.ai>
Signed-off-by: archlitchi <mengxuan.li@dynamia.ai>
Signed-off-by: archlitchi <mengxuan.li@dynamia.ai>
Signed-off-by: limengxuan <mengxuan.li@dynamia.ai>
Signed-off-by: archlitchi <mengxuan.li@dynamia.ai>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThe change classifies duplicate node-lock errors with ChangesNode-lock contention classification
PodGroup membership detection
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested labels: Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 1 file with indirect coverage changes 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/util/nodelock/nodelock_test.go`:
- Around line 457-483: Update TestSetNodeLockRaceIsRetryable and its comments to
reflect that the test performs sequential SetNodeLock calls and only verifies
the already-locked path, or rewrite it to run concurrent LockNode callers so it
actually exercises the race and validates retryable contention classification.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 03ec1363-9ce4-428d-b33a-45c7eebefc27
📒 Files selected for processing (4)
pkg/util/nodelock/nodelock.gopkg/util/nodelock/nodelock_test.gopkg/util/util.gopkg/util/util_test.go
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: archlitchi, FouoF The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
PodGroup 是**命名空间作用域**对象 —— coscheduling 的 label、Volcano / Koordinator / kube-batch 的注解、k8s 1.36 原生的 spec.schedulingGroup、以及 PodGroup ownerReference,全部在 Pod 自己的命名空间内解析。但此前所有"判断两个 Pod 是否同一个 gang"的地方都直接比裸名字,而 scheduler 的 Pod informer 是 集群级的,于是两个租户各自建一个叫 "training" 的 PodGroup 就会互相当成兄弟 Pod。 多租户集群里 gang 名往往来自工作负载名(Kubeflow PyTorchJob 等按 job 名建 PodGroup),同名概率并不低。 受影响的四处判定: 1. IndexerKeyPodGangName 索引(filter_predicate.go) —— 按裸名建索引, ListByIndexValue 会返回其它命名空间的同名 gang Pod。被 filter 的 FindGangSiblingDomain 与 preempt 的跨 Pod 拓扑对齐消费。 2. GangAnchorComponent(device/types.go) —— 遍历 n.nodePods(含节点上所有 命名空间)按裸名匹配兄弟,会把别的租户的 Pod 当锚点,把本 gang 锚到错误的 NVLink 组件上; strict 模式下可能进一步导致节点被误拒。 3. isProtectedFromPreemption(preempt) —— "避免抢占兄弟 Pod"按裸名匹配,会 误保护无关租户的 Pod,导致抢占凑不齐受害者、抢占方一直 Pending。 4. 抢占事件里列出的 pod group 名 —— 受害者可能跨命名空间,裸名有歧义。 修复: 新增 util.PodGangKey(pod) 返回 "<namespace>/<name>",上述四处判定与 req.GangName 的生产端统一改用它。util.PodHasGangName 保留原语义(返回用户书写 的原始名),仅用于展示和"是否属于某个 gang"的布尔判断 —— sortVictimsByPreference 只取布尔值,不受影响。 测试: cross_pod_test 的 fixture 补上命名空间并新增回归用例(其它命名空间的同名 gang 不算兄弟); 新增 Test_PodGangKey 覆盖限定、跨命名空间不相等、非 gang、nil。 背景: 排查 Project-HAMi/HAMi#2206(适配 k8s 1.36 原生 gangScheduling)时顺带发现。 该 PR 的主体改动(识别 spec.schedulingGroup)本项目已支持且覆盖面更全,另一半 (nodelock 争用重试)本项目无对应机制。 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
/kind bug
PR:#2066 only support coScheduling plugin, but doesn't support GangScheduling feature gate implemented in v1.36.
How to use:
on
kube-apiserver,kube-scheduler,kube-controller-managerkube-apiserver, add the parameter below:Example:
Result:

I'll later add a user-guide to website
Summary by CodeRabbit