Skip to content

Add documentation to deploy and test SnapshotMetadata support #575

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Jan 24, 2025
+335 −1
Merged

Add documentation to deploy and test SnapshotMetadata support #575

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
ebb9d84
Update deploy script to deploy snapshot-metadata service
PrasadG193 Dec 6, 2024
dd826ba
Update deploy/util/deploy-hostpath.sh
PrasadG193 Dec 6, 2024
6886d26
Update manifest links
PrasadG193 Dec 10, 2024
0c52080
Add documentation for testing SnapshotMetadata service
PrasadG193 Dec 17, 2024
295e9cd
Refer snapshot metadata client pod manifest from repo
PrasadG193 Dec 19, 2024
80c7ebe
Correct pod manifest file name
PrasadG193 Dec 20, 2024
eafaa37
Do not create snapshot metadata rbac by default
PrasadG193 Dec 20, 2024
dde7e9b
Set hostpathplugin image tag to canary
PrasadG193 Dec 20, 2024
99a5d45
Correct snapshot name in kubectl get cmd
PrasadG193 Jan 8, 2025
bb2cf22
Consistent sample snapshotcontent name
PrasadG193 Jan 9, 2025
8110ea0
Use released image tag for hostpath deployment
PrasadG193 Jan 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 24 additions & 1 deletion deploy/kubernetes-1.27/hostpath/csi-hostpath-plugin.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,24 @@ subjects:
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/instance: hostpath.csi.k8s.io
app.kubernetes.io/part-of: csi-driver-host-path
app.kubernetes.io/name: csi-hostpathplugin
app.kubernetes.io/component: snapshot-metadata-cluster-role
name: csi-hostpathplugin-snapshot-metadata-cluster-role
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: external-snapshot-metadata-runner
subjects:
- kind: ServiceAccount
name: csi-hostpathplugin-sa
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
Expand Down Expand Up @@ -219,12 +237,14 @@ spec:
serviceAccountName: csi-hostpathplugin-sa
containers:
- name: hostpath
image: registry.k8s.io/sig-storage/hostpathplugin:v1.15.0
# TODO: Set release tag after new release is made with SnapshotMetadata support
image: gcr.io/k8s-staging-sig-storage/hostpathplugin:canary
Copy link
Contributor Author

@PrasadG193 PrasadG193 Jan 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Rakshith-R @xing-yang We cannot use released tag here until new release is made having support for SnapshotMetada service. I don't think we can use canary image tag here.

Copy link

@Rakshith-R Rakshith-R Jan 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Rakshith-R @xing-yang We cannot use released tag here until new release is made having support for SnapshotMetada service. I don't think we can use canary image tag here.

I think we can override it to canary while testing it at ext-snapshot-metadata repo + SnapshotMetadata service is guarded by a flag too in deploy script.

Let's leave it at a released tag here ?

Copy link
Contributor Author

@PrasadG193 PrasadG193 Jan 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is one option. Its just until the release is made, if someone want to try the feature (using steps documented at docs/example-snapshot-metadata.md), they won't be able to setup things.
@xing-yang any thoughts?

Copy link
Contributor Author

@PrasadG193 PrasadG193 Jan 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the DPWG meeting, we decided to move ahead with canary image tag for now. This will be overridden at release time.

Copy link
Contributor

@xing-yang xing-yang Jan 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On a second thought, we can't merge a canary image as that will affect CI for all the sidecars. We need to hold off this PR until we cut a release for K8s 1.32. We are still waiting for a few other sidecars to be released before doing that.

Copy link

@Rakshith-R Rakshith-R Jan 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Rakshith-R @xing-yang We cannot use released tag here until new release is made having support for SnapshotMetada service. I don't think we can use canary image tag here.

I think we can override it to canary while testing it at ext-snapshot-metadata repo + SnapshotMetadata service is guarded by a flag too in deploy script.

Let's leave it at a released tag here ?

I don't think we need to hold the entire pr.

Let's go with this approach of leaving driver image at a released tag and overriding the image to canary and turn on snap-metadata flag to test the feature.

@PrasadG193 @xing-yang ^

args:
- "--drivername=hostpath.csi.k8s.io"
- "--v=5"
- "--endpoint=$(CSI_ENDPOINT)"
- "--nodeid=$(KUBE_NODE_NAME)"
# end hostpath args
env:
- name: CSI_ENDPOINT
value: unix:///csi/csi.sock
Expand Down Expand Up @@ -367,6 +387,8 @@ spec:
- mountPath: /csi
name: socket-dir

# end csi containers

volumes:
- hostPath:
path: /var/lib/kubelet/plugins/csi-hostpath
Expand Down Expand Up @@ -394,3 +416,4 @@ spec:
path: /dev
type: Directory
name: dev-dir
# end csi volumes
23 changes: 23 additions & 0 deletions deploy/kubernetes-1.27/hostpath/csi-snapshot-metadata-sidecar.patch
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
- name: csi-snapshot-metadata
# TODO: Replace main tag with actual released tag once external-snapshot-metadata is released
image: gcr.io/k8s-staging-sig-storage/csi-snapshot-metadata:canary
imagePullPolicy: "IfNotPresent"
command:
args:
- "--csi-address=/csi/csi.sock"
- "--tls-cert=/tmp/certificates/tls.crt"
- "--tls-key=/tmp/certificates/tls.key"
readinessProbe:
exec:
command:
- "/bin/grpc_health_probe"
- "-addr=:50051"
- "-tls"
- "-tls-no-verify"
initialDelaySeconds: 5
volumeMounts:
- mountPath: /csi
name: socket-dir
- name: csi-snapshot-metadata-server-certs
mountPath: /tmp/certificates
readOnly: true
49 changes: 48 additions & 1 deletion deploy/util/deploy-hostpath.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ default_kubelet_data_dir=/var/lib/kubelet
# - CSI_PROVISIONER_TAG
# - CSI_SNAPSHOTTER_REGISTRY
# - CSI_SNAPSHOTTER_TAG
# - CSI_SNAPSHOT_METADATA_REGISTRY
# - CSI_SNAPSHOT_METADATA_TAG
# - HOSTPATHPLUGIN_REGISTRY
# - HOSTPATHPLUGIN_TAG
#
Expand Down Expand Up @@ -142,25 +144,37 @@ function volume_mode_conversion () {
[ "${VOLUME_MODE_CONVERSION_TESTS}" == "true" ]
}

function snapshot_metadata () {
[ "${SNAPSHOT_METADATA_TESTS}" == "true" ]
}

# In addition, the RBAC rules can be overridden separately.
# For snapshotter 2.0+, the directory has changed.
SNAPSHOTTER_RBAC_RELATIVE_PATH="rbac.yaml"
if version_gt $(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-snapshotter.yaml" csi-snapshotter "${UPDATE_RBAC_RULES}") "v1.255.255"; then
SNAPSHOTTER_RBAC_RELATIVE_PATH="csi-snapshotter/rbac-csi-snapshotter.yaml"
fi
SNAPSHOT_METADATA_RBAC_RELATIVE_PATH="snapshot-metadata-cluster-role.yaml"
SNAPSHOT_METADATA_SIDECAR_PATCH_RELATIVE_PATH="${BASE_DIR}/hostpath/csi-snapshot-metadata-sidecar.patch"

CSI_PROVISIONER_RBAC_YAML="https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-provisioner.yaml" csi-provisioner false)/deploy/kubernetes/rbac.yaml"
: ${CSI_PROVISIONER_RBAC:=https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-provisioner.yaml" csi-provisioner "${UPDATE_RBAC_RULES}")/deploy/kubernetes/rbac.yaml}
CSI_ATTACHER_RBAC_YAML="https://raw.githubusercontent.com/kubernetes-csi/external-attacher/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-attacher.yaml" csi-attacher false)/deploy/kubernetes/rbac.yaml"
: ${CSI_ATTACHER_RBAC:=https://raw.githubusercontent.com/kubernetes-csi/external-attacher/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-attacher.yaml" csi-attacher "${UPDATE_RBAC_RULES}")/deploy/kubernetes/rbac.yaml}
CSI_SNAPSHOTTER_RBAC_YAML="https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-snapshotter.yaml" csi-snapshotter false)/deploy/kubernetes/${SNAPSHOTTER_RBAC_RELATIVE_PATH}"
: ${CSI_SNAPSHOTTER_RBAC:=https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-snapshotter.yaml" csi-snapshotter "${UPDATE_RBAC_RULES}")/deploy/kubernetes/${SNAPSHOTTER_RBAC_RELATIVE_PATH}}
CSI_SNAPSHOT_METADATA_RBAC_YAML="https://raw.githubusercontent.com/kubernetes-csi/external-snapshot-metadata/$(rbac_version "${BASE_DIR}/hostpath/csi-snapshot-metadata-sidecar.patch" csi-snapshot-metadata false)/deploy/${SNAPSHOT_METADATA_RBAC_RELATIVE_PATH}"
: ${CSI_SNAPSHOT_METADATA_RBAC:=https://raw.githubusercontent.com/kubernetes-csi/external-snapshot-metadata/$(rbac_version "${BASE_DIR}/hostpath/csi-snapshot-metadata-sidecar.patch" csi-snapshot-metadata "${UPDATE_RBAC_RULES}")/deploy/${SNAPSHOT_METADATA_RBAC_RELATIVE_PATH}}
CSI_RESIZER_RBAC_YAML="https://raw.githubusercontent.com/kubernetes-csi/external-resizer/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-resizer.yaml" csi-resizer false)/deploy/kubernetes/rbac.yaml"
: ${CSI_RESIZER_RBAC:=https://raw.githubusercontent.com/kubernetes-csi/external-resizer/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-resizer.yaml" csi-resizer "${UPDATE_RBAC_RULES}")/deploy/kubernetes/rbac.yaml}

CSI_EXTERNALHEALTH_MONITOR_RBAC_YAML="https://raw.githubusercontent.com/kubernetes-csi/external-health-monitor/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-plugin.yaml" csi-external-health-monitor-controller false)/deploy/kubernetes/external-health-monitor-controller/rbac.yaml"
: ${CSI_EXTERNALHEALTH_MONITOR_RBAC:=https://raw.githubusercontent.com/kubernetes-csi/external-health-monitor/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-plugin.yaml" csi-external-health-monitor-controller "${UPDATE_RBAC_RULES}")/deploy/kubernetes/external-health-monitor-controller/rbac.yaml}

CSI_SNAPSHOT_METADATA_TLS_CERT_YAML="https://raw.githubusercontent.com/kubernetes-csi/external-snapshot-metadata/$(rbac_version "${BASE_DIR}/hostpath/csi-snapshot-metadata-sidecar.patch" csi-snapshot-metadata false)/deploy/example/csi-driver/testdata/csi-snapshot-metadata-tls-secret.yaml"
SNAPSHOT_METADATA_SERVICE_CR_YAML="https://raw.githubusercontent.com/kubernetes-csi/external-snapshot-metadata/$(rbac_version "${BASE_DIR}/hostpath/csi-snapshot-metadata-sidecar.patch" csi-snapshot-metadata false)/deploy/example/csi-driver/testdata/snapshotmetadataservice.yaml"
CSI_SNAPSHOT_METADATA_SERVICE_YAML="https://raw.githubusercontent.com/kubernetes-csi/external-snapshot-metadata/$(rbac_version "${BASE_DIR}/hostpath/csi-snapshot-metadata-sidecar.patch" csi-snapshot-metadata false)/deploy/example/csi-driver/testdata/csi-snapshot-metadata-service.yaml"

INSTALL_CRD=${INSTALL_CRD:-"false"}

# Some images are not affected by *_REGISTRY/*_TAG and IMAGE_* variables.
Expand All @@ -176,7 +190,11 @@ run () {

# rbac rules
echo "applying RBAC rules"
for component in CSI_PROVISIONER CSI_ATTACHER CSI_SNAPSHOTTER CSI_RESIZER CSI_EXTERNALHEALTH_MONITOR; do
components=(CSI_PROVISIONER CSI_ATTACHER CSI_SNAPSHOTTER CSI_RESIZER CSI_EXTERNALHEALTH_MONITOR)
if snapshot_metadata; then
components+=(CSI_SNAPSHOT_METADATA)
fi
for component in "${components[@]}"; do
eval current="\${${component}_RBAC}"
eval original="\${${component}_RBAC_YAML}"
if [ "$current" != "$original" ]; then
Expand Down Expand Up @@ -210,13 +228,42 @@ EOF
run kubectl apply --kustomize "${TEMP_DIR}"
done

# deploy snapshot-metadata service components
if snapshot_metadata; then
echo "applying snapshot metadata components"
for component in ${CSI_SNAPSHOT_METADATA_TLS_CERT_YAML} ${SNAPSHOT_METADATA_SERVICE_CR_YAML} ${CSI_SNAPSHOT_METADATA_SERVICE_YAML}; do
run curl "${component}" --output "${TEMP_DIR}"/snap-metadata-comp.yaml --silent --location
cat <<- EOF > "${TEMP_DIR}"/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

commonLabels:
app.kubernetes.io/instance: hostpath.csi.k8s.io
app.kubernetes.io/part-of: csi-driver-host-path

resources:
- ./snap-metadata-comp.yaml
EOF

run kubectl apply --kustomize "${TEMP_DIR}"
done
fi

# deploy hostpath plugin and registrar sidecar
echo "deploying hostpath components"
for i in $(ls ${BASE_DIR}/hostpath/*.yaml | sort); do
echo " $i"
if volume_mode_conversion; then
sed -i -e 's/# end csi-provisioner args/- \"--prevent-volume-mode-conversion=true\"\n # end csi-provisioner args/' $i
fi

# Add external-snapshot-metadata sidecar to the driver, mount TLS certs,
# and enable snapshot-metadata service
if snapshot_metadata; then
sed -i -e "/# end csi containers/r ${SNAPSHOT_METADATA_SIDECAR_PATCH_RELATIVE_PATH}" $i
sed -i -e 's/# end csi volumes/- name: csi-snapshot-metadata-server-certs\n secret:\n secretName: csi-snapshot-metadata-certs\n # end csi volumes/' $i
sed -i -e 's/# end hostpath args/- \"--enable-snapshot-metadata\"\n # end hostpath args/' $i
fi
modified="$(cat "$i" | sed -e "s;${default_kubelet_data_dir}/;${KUBELET_DATA_DIR}/;" | while IFS= read -r line; do
nocomments="$(echo "$line" | sed -e 's/ *#.*$//')"
if echo "$nocomments" | grep -q '^[[:space:]]*image:[[:space:]]*'; then
Expand Down
Loading