Fix drain for evicting terminal DS pods and pods with local storage

[ravisantoshgudimetla]

What this PR does / why we need it:

As of now draining a node is not possible even if there are only terminal pods(whose state is successful/failed) with local volumes, managed by DS. Since a terminal pod can be safely deleted any time, we should allow this operation.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Release note:

Drain should delete terminal pods.

[ravisantoshgudimetla]

xref: https://bugzilla.redhat.com/show_bug.cgi?id=1628693

[soltysh]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ravisantoshgudimetla, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/kubectl/OWNERS [soltysh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[soltysh]

This is a significant blocker from a drain POV, setting milestone to v1.12

[ravisantoshgudimetla]

/priority critical-urgent

[ravisantoshgudimetla]

/cherrypick release-1.11

[dims]

/uncc

[tpepper]

@soltysh and others here: Given the lateness in the 1.12 cycle, and that this appears to be a regression perhaps back as far as 1.10 if I'm researching the OpenShift<->Kubernetes versions mappings correctly, how critically urgent is it that this merge in the next four or five days? Versus being merged after code freeze and then backported to multiple releases where needed?

[ravisantoshgudimetla]

@tpepper If you are talking in terms of tests, I think this won't cause regression failure but yes, it is regression from previous releases(it is mostly a bug that went unnoticed). If this is causing upgrade tests to fail, I will look more into this. So, let's proceed with the merge, we can always revert, if we are not able to fix it in time.

[tpepper]

Honestly given the description, I'd prefer we hold off on merging this at least until tomorrow. I'm trying to manage a small and specific set of critical bug fix changes today and clearly view their test impact ahead of building RC later today.

[guineveresaenger]

/retest
/hold

Agreeing with Tim.

[ravisantoshgudimetla]

Honestly given the description, I'd prefer we hold off on merging this at least until tomorrow. I'm trying to manage a small and specific set of critical bug fix changes today and clearly view their test impact ahead of building RC later today.

I understand your concern, let's hold it back till tomorrow.

EDIT: Just saw @guineveresaenger's hold. Thanks @guineveresaenger, I am fine with holding this PR until tomorrow.

[derekwaynecarr]

@ravisantoshgudimetla upgrade tests should have been failing without this fix as we know upgrades fail due to this condition. should we augment upgrade tests to cover this scenario?

[ravisantoshgudimetla]

@derekwaynecarr I am planning to cherrypick this fix till 1.10. Do you mean upgrade from 1.9 to 1.10 should fail without this fix?

[tpepper]

We got the test results we were looking for today. I'm intrigued to see if this does help on some upgrade failures.

/hold cancel

[ravisantoshgudimetla]

/cherrypick release-1.11