Add distributed snapshotting support to kubernetes-distributed deployment - #653
Add distributed snapshotting support to kubernetes-distributed deployment#653mnencia wants to merge 1 commit into
Conversation
k8s-ci-robot
added
release-note
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: mnencia The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Welcome @mnencia! |
|
Hi @mnencia. Thanks for your PR. I'm waiting for a kubernetes-csi member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
needs-ok-to-test
k8s-ci-robot
removed
the
do-not-merge/work-in-progress
Replace the single-node StatefulSet csi-hostpath deployment with the distributed DaemonSet variant, which runs the CSI driver on every node with per-node provisioning, snapshotting, and resizing via --node-deployment sidecars. This enables multi-node test scenarios (pod failover, node drain, etc.) where PVCs can be created on different nodes instead of all landing on the same one. The resizer sidecar uses a custom image (ghcr.io/mnencia/csi-resizer:node-deployment) built from kubernetes-csi/external-resizer#573, pending upstream merge. The distributed deployment manifests are sourced from mnencia/csi-driver-host-path#dev/651-with-resizer, pending kubernetes-csi/csi-driver-host-path#653. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
|
/ok-to-test |
k8s-ci-robot
added
ok-to-test
| run kubectl apply --kustomize "${TEMP_DIR}" | ||
| done | ||
|
|
||
| # Install snapshot CRDs and snapshot-controller if not already present. |
There was a problem hiding this comment.
I think kubernetes-distributed should be in sync with deploy.sh. Either both of them should install snapshot CRDs and the controller or none of them. I personally prefer keep them as they are, without the CRD and controller installation.
There was a problem hiding this comment.
Done. Stripped the snapshot CRDs and snapshot-controller install from deploy.sh and the matching cleanup from destroy.sh. The README now documents that snapshot-controller --enable-distributed-snapshotting=true (and the VolumeSnapshot* CRDs) is a prerequisite the operator installs separately, the same way deploy/util/deploy-hostpath.sh works.
While I was at it I also reverted the snapshotDataSource: true flip in deploy/kubernetes-distributed/test-driver.yaml. That was the cause of the pull-kubernetes-csi-csi-driver-host-distributed-on-kubernetes-1-28 failure: with the snapshotter sidecar present but no fix for cross-node snapshot restore, the snapshot E2E suite deadlocks at PreBind waiting for bindTimeout. Cross-node restore is handled by a follow-up controller (csi-topology-coordinator) that I will open as a separate PR; the test-driver flag flips back on there once the controller is in place.
k8s-ci-robot
added
size/XXL
k8s-ci-robot
added
size/L
…ment Add the csi-snapshotter sidecar with --node-deployment=true to the plugin DaemonSet, so each node runs its own snapshotter against its local .snap data. Add a VolumeSnapshotClass for the hostpath.csi.k8s.io driver. Bind one csi-hostpathplugin-sa ServiceAccount to both the external-provisioner and external-snapshotter ClusterRoles and Roles, matching the kubernetes-latest deployment layout. The distributed plugin pod runs both sidecars on every node, so a single ServiceAccount must hold both role sets. deploy.sh does not install the external snapshot-controller or the VolumeSnapshot CRDs. Operators must install snapshot-controller with --enable-distributed-snapshotting=true and the matching CRDs before deploying the driver, the same as for deploy/util/deploy-hostpath.sh. The README documents the prerequisite. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Replace the single-node StatefulSet csi-hostpath deployment with the distributed DaemonSet variant, which runs the CSI driver on every node with per-node provisioning, snapshotting, and resizing via --node-deployment sidecars. This enables multi-node test scenarios (pod failover, node drain, etc.) where PVCs can be created on different nodes instead of all landing on the same one. The resizer sidecar uses a custom image (ghcr.io/mnencia/csi-resizer:node-deployment) built from kubernetes-csi/external-resizer#573, pending upstream merge. The distributed deployment manifests are sourced from mnencia/csi-driver-host-path#dev/651-with-resizer, pending kubernetes-csi/csi-driver-host-path#653. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
|
The Kubernetes project currently lacks enough contributors to adequately respond to all PRs. This bot triages PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
kubernetes-prow
Bot
added
the
lifecycle/stale
|
/remove-lifecycle stale |
kubernetes-prow
Bot
removed
the
lifecycle/stale
4 participants
What type of PR is this?
/kind feature
What this PR does / why we need it:
The distributed DaemonSet deployment (
deploy/kubernetes-distributed/) only includes the csi-provisioner sidecar. This adds the csi-snapshotter with--node-deployment=true, so snapshots work on each node's local volumes.To support multiple sidecar RBAC roles, this introduces a unified ServiceAccount (
csi-hostpathplugin-sa) with explicit ClusterRoleBindings for both the provisioner and snapshotter roles, following the same pattern used by thekubernetes-latestdeployment.The deploy script now handles the full snapshot infrastructure: CRD installation, snapshot-controller deployment with
--enable-distributed-snapshotting=true, and node-reader RBAC (required for distributed snapshotting but commented out in the upstream snapshot-controller RBAC). If the snapshot-controller was already deployed (e.g., by prow.sh) without the flag, the script patches it. The destroy script cleans up all of these resources.Snapshot E2E tests are enabled via
snapshotDataSourceandSnapshotClassin test-driver.yaml.Which issue(s) this PR fixes:
Part of #651
Special notes for your reviewer:
Builds on prior work in #392 by @denisok, which went stale before merging.
The distributed snapshotting feature in external-snapshotter (kubernetes-csi/external-snapshotter#585) requires coordination between two components:
--enable-distributed-snapshotting=trueto label VolumeSnapshotContent objects with node affinity--node-deployment=trueto filter by those labelsThe upstream snapshot-controller RBAC has Node read permissions commented out. The deploy script applies them via a separate
snapshot-controller-node-readerClusterRole.Tested on a 3-node Kind cluster: provisioning on different workers, snapshot creation and deletion all work correctly.
Does this PR introduce a user-facing change?: