Prioritize Workload which fits in its CQ's Nominal Capacity

[gabesaba]

What type of PR is this?

/kind bug

What this PR does / why we need it:

We fix the calculation of borrowing in FlavorAssignment for Preemption fit mode. Now, it matches how we calculate borrowing for Fit mode. This ensures that a workload which fits within its CQ's nominal capacity is sorted before an earlier created workload which pushes its CQ over nominal capacity. This sorting is important as this borrowing workload may invalidate the non-borrowing workload's preemption calculations

Which issue(s) this PR fixes:

Fixes #3405

Special notes for your reviewer:

We tweak an integ test which relied on details of FlavorAssigner's borrowing calculations for ordering. By updating the priority of pendingWl, we ensure that it reservers resources and prevents blockedWl from scheduling.

Does this PR introduce a user-facing change?

Determine borrowing more accurately, allowing preempting workloads which fit in nominal quota to schedule faster

[netlify]

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	25d59d5
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/673771af57da550008030873

[mimowo]

I synced with @gabesaba and did review on a call. The main changes since the initial version were:

• added && a.cq.HasParent() to borrow := a.cq.BorrowingWith(fr, val) && a.cq.HasParent()
• improve readability of the new test case scenario by matching the workload names to the issue description

Hope this helps, feel free to ask more question in comments if you feel so.

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gabesaba, mimowo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mimowo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: ac0f7b4cf0e067a2d008edbd4b25b6cdf4b2232d

[mimowo]

We are going to cherry-pick this PR along with #3536

[mimowo]

/cherry-pick release-0.9

[k8s-infra-cherrypick-robot]

@mimowo: new pull request created: #3550

In response to this:

/cherry-pick release-0.9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.