WIP: deploy with DaemonSet

[pohly]

What type of PR is this?
/kind cleanup

What this PR does / why we need it:

DaemonSet has the advantage that "kubectl drain --ignore-daemonsets"
keeps the driver pods running, which is required if normal pods on the
node are to be evicted. When using a StatefulSet, it can happen that
the driver pod gets evicted first and then the other pods cannot be
evicted because their volumes can no longer be unpublished and
unstaged.

The downside is that we still have to ensure that the driver only runs
on a single node. With DaemonSet, that is only possible by picking a
node in advance and using that node in a node selector. This node
selection tries to ensure that the pod can really run by checking
various conditions (node ready, network ready, no taints), but this is
less complete than the checks done by kube-scheduler (ignores load).

Does this PR introduce a user-facing change?:

The non-distributed deployments use a DaemonSet to mimick best practices for CSI drivers. The driver still only runs on a single node which has to be chosen in advance based on various criteria (node ready, network ready, no taints) by the deploy script.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pohly

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [pohly]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[pohly]

It's going to be more complex also in other places: the pod name is no longer deterministic, therefore sanity testing will have to be configured differently. The code which updates the test driver config must be updated.

I haven't tackled that yet...

[k8s-ci-robot]

@pohly: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-kubernetes-csi-csi-driver-host-path-1-19-on-kubernetes-1-19	481fc1a	link	/test pull-kubernetes-csi-csi-driver-host-path-1-19-on-kubernetes-1-19
pull-kubernetes-csi-csi-driver-host-path-1-18-on-kubernetes-1-18	481fc1a	link	/test pull-kubernetes-csi-csi-driver-host-path-1-18-on-kubernetes-1-18
pull-kubernetes-csi-csi-driver-host-path-1-21-on-kubernetes-1-21	481fc1a	link	/test pull-kubernetes-csi-csi-driver-host-path-1-21-on-kubernetes-1-21

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[pohly]

/remove-lifecycle stale

@jsafrane do you think using a "proper" DaemonSet is worth the extra complexity or can I close this PR?

Further work would be needed if we want to do this.

[jsafrane]

I briefly looked if we can create a Pod that would like a DaemonSet pod without all the shell overhead here. It's probably not possible. kubelet checks Pod owner and even GETs the DaemonSet:
https://github.com/kubernetes/kubectl/blob/4defba0cec1f594eb410c69bff05b51cddfba8ff/pkg/drain/filters.go#L174-L204

On the other hand, there are other conditions when a pod is skipped during drain:
https://github.com/kubernetes/kubectl/blob/4defba0cec1f594eb410c69bff05b51cddfba8ff/pkg/drain/filters.go#L154-L159

They all seems to be hard to achieve, perhaps our driver Pod could disguise as mirror pod, with a special annotation?
https://github.com/kubernetes/kubectl/blob/4defba0cec1f594eb410c69bff05b51cddfba8ff/pkg/drain/filters.go#L207

[jsafrane]

perhaps our driver Pod could disguise as mirror pod, with a special annotation?

No, that annotation has very specific handling, such a pod is deleted unless it's really a mirror pod.

[jsafrane]

Just brainstorming: would it be better to run a "probe" pod, see where it's scheduled and run the driver DaemonSet on the same node?
Alternatively, what is actually goal of the deploy script? All tests run their own driver in go code, without this script. Would it make sense to run the DaemonSet everywhere and use distributed provisioning here?

[pohly]

Just brainstorming: would it be better to run a "probe" pod, see where it's scheduled and run the driver DaemonSet on the same node?

That would be simpler. It's a bit slower (needs to wait for one pod to start), but that shouldn't matter much.

Alternatively, what is actually goal of the deploy script? All tests run their own driver in go code, without this script.

The hostpath tests in k/k do that. But all Kubernetes-CSI Prow jobs deploy the hostpath driver once per cluster using deploy.sh, then run the k/k e2e.test with an external driver config.

Would it make sense to run the DaemonSet everywhere and use distributed provisioning here?

That would massively increase the number of pods during parallel k/k tests. Resource consumption and overloading the cluster are already a problem.

[k8s-ci-robot]

@pohly: PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closed this PR.

Details

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.