Can not mount blob to AKS cluster via Python image using blobfuse2. #1079
Comments
|
As you have set "allow-other: true" ensure this feature is enable in /etc/fuse.conf file. |
Thank you. In my case, I deployed on a Debian 11 distribution: to Configure the Microsoft package repository.wget https://packages.microsoft.com/config/debian/11/packages-microsoft-prod.deb Then I installed blobfuse2. But in the ./docker/dockerfile the installation is as below: Install fuse libraryRUN The different is I didn't install ca-certificates vim rsyslog and fuse but fuse3 |
|
If you are running blobfuse on a container you can follow this. |
|
I think the AKS version is somewhat related. One of my cluster is using For instance, I got an error like the following. Even though I did Also on a related issue: #1078 (I think) Even after adding After restarting the pod the second time that has the blob-fuse mount, the mount disappeared...
# After a fresh kubectl apply
I0308 20:43:54.932883 6583 utils.go:75] GRPC call: /csi.v1.Node/NodeStageVolume
I0308 20:43:54.932896 6583 utils.go:76] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["-o allow_other","--file-cache-timeout-in-seconds=120","--use-attr-cache=true","-o attr_timeout=120","-o entry_timeout=120","-o negative_timeout=120","--log-level=LOG_DEBUG","--cache-size-mb=1000","--virtual-directory=true"]}},"access_mode":{"mode":3}},"volume_context":{"containerName":"backups-prod","resourceGroup":"rg-backups-prod"},"volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0308 20:43:54.933243 6583 blob.go:414] volumeID(rg-backups-prod#stpabackupsprod#backups-prod) authEnv: []
I0308 20:43:54.933290 6583 nodeserver.go:349] target /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount
protocol
volumeId rg-backups-prod#stpabackupsprod#backups-prod
context map[containerName:backups-prod resourceGroup:rg-backups-prod]
mountflags [-o allow_other --file-cache-timeout-in-seconds=120 --use-attr-cache=true -o attr_timeout=120 -o entry_timeout=120 -o negative_timeout=120 --log-level=LOG_DEBUG --cache-size-mb=1000 --virtual-directory=true]
mountOptions [-o allow_other --file-cache-timeout-in-seconds=120 --use-attr-cache=true -o attr_timeout=120 -o entry_timeout=120 -o negative_timeout=120 --log-level=LOG_DEBUG --cache-size-mb=1000 --virtual-directory=true --cancel-list-on-mount-seconds=10 --empty-dir-check=false --tmp-path=/mnt/rg-backups-prod#stpabackupsprod#backups-prod --container-name=backups-prod --pre-mount-validate=true --use-https=true]
args /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount -o allow_other --file-cache-timeout-in-seconds=120 --use-attr-cache=true -o attr_timeout=120 -o entry_timeout=120 -o negative_timeout=120 --log-level=LOG_DEBUG --cache-size-mb=1000 --virtual-directory=true --cancel-list-on-mount-seconds=10 --empty-dir-check=false --tmp-path=/mnt/rg-backups-prod#stpabackupsprod#backups-prod --container-name=backups-prod --pre-mount-validate=true --use-https=true
serverAddress stpabackupsprod.blob.core.windows.net
I0308 20:43:54.933314 6583 nodeserver.go:144] mouting using blobfuse proxy
I0308 20:43:54.933699 6583 nodeserver.go:158] calling BlobfuseProxy: MountAzureBlob function
I0308 20:43:55.136247 6583 nodeserver.go:397] volume(rg-backups-prod#stpabackupsprod#backups-prod) mount on "/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount" succeeded
I0308 20:43:55.136276 6583 utils.go:82] GRPC response: {}
I0308 20:43:55.138681 6583 utils.go:75] GRPC call: /csi.v1.Node/NodePublishVolume
I0308 20:43:55.138693 6583 utils.go:76] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount","target_path":"/var/lib/kubelet/pods/30befd9c-d850-464f-9cb9-9ddb789ab67f/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["-o allow_other","--file-cache-timeout-in-seconds=120","--use-attr-cache=true","-o attr_timeout=120","-o entry_timeout=120","-o negative_timeout=120","--log-level=LOG_DEBUG","--cache-size-mb=1000","--virtual-directory=true"]}},"access_mode":{"mode":3}},"volume_context":{"containerName":"backups-prod","csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"orientdbservice-58c79ffb79-ltccl","csi.storage.k8s.io/pod.namespace":"pa-data-services","csi.storage.k8s.io/pod.uid":"30befd9c-d850-464f-9cb9-9ddb789ab67f","csi.storage.k8s.io/serviceAccount.name":"default","resourceGroup":"rg-backups-prod"},"volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0308 20:43:55.139102 6583 nodeserver.go:122] NodePublishVolume: volume rg-backups-prod#stpabackupsprod#backups-prod mounting /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount at /var/lib/kubelet/pods/30befd9c-d850-464f-9cb9-9ddb789ab67f/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount with mountOptions: [bind]
I0308 20:43:55.139124 6583 mount_linux.go:219] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount /var/lib/kubelet/pods/30befd9c-d850-464f-9cb9-9ddb789ab67f/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount)
I0308 20:43:55.140316 6583 mount_linux.go:219] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount /var/lib/kubelet/pods/30befd9c-d850-464f-9cb9-9ddb789ab67f/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount)
I0308 20:43:55.141314 6583 nodeserver.go:138] NodePublishVolume: volume rg-backups-prod#stpabackupsprod#backups-prod mount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount at /var/lib/kubelet/pods/30befd9c-d850-464f-9cb9-9ddb789ab67f/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount successfully
I0308 20:43:55.141325 6583 utils.go:82] GRPC response: {}
# After the first deletion of the pod, notice there is a `GRPC error: rpc error: code` due to `device or resource busy`
I0308 20:44:25.288113 6583 utils.go:75] GRPC call: /csi.v1.Node/NodePublishVolume
I0308 20:44:25.288123 6583 utils.go:76] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount","target_path":"/var/lib/kubelet/pods/07a6dec5-faba-451e-a9c3-f95dd627ad8a/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["-o allow_other","--file-cache-timeout-in-seconds=120","--use-attr-cache=true","-o attr_timeout=120","-o entry_timeout=120","-o negative_timeout=120","--log-level=LOG_DEBUG","--cache-size-mb=1000","--virtual-directory=true"]}},"access_mode":{"mode":3}},"volume_context":{"containerName":"backups-prod","csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"orientdbservice-58c79ffb79-shw75","csi.storage.k8s.io/pod.namespace":"pa-data-services","csi.storage.k8s.io/pod.uid":"07a6dec5-faba-451e-a9c3-f95dd627ad8a","csi.storage.k8s.io/serviceAccount.name":"default","resourceGroup":"rg-backups-prod"},"volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0308 20:44:25.288521 6583 nodeserver.go:122] NodePublishVolume: volume rg-backups-prod#stpabackupsprod#backups-prod mounting /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount at /var/lib/kubelet/pods/07a6dec5-faba-451e-a9c3-f95dd627ad8a/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount with mountOptions: [bind]
I0308 20:44:25.288538 6583 mount_linux.go:219] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount /var/lib/kubelet/pods/07a6dec5-faba-451e-a9c3-f95dd627ad8a/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount)
I0308 20:44:25.289794 6583 mount_linux.go:219] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount /var/lib/kubelet/pods/07a6dec5-faba-451e-a9c3-f95dd627ad8a/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount)
I0308 20:44:25.290840 6583 nodeserver.go:138] NodePublishVolume: volume rg-backups-prod#stpabackupsprod#backups-prod mount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount at /var/lib/kubelet/pods/07a6dec5-faba-451e-a9c3-f95dd627ad8a/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount successfully
I0308 20:44:25.290856 6583 utils.go:82] GRPC response: {}
I0308 20:44:26.495366 6583 utils.go:75] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0308 20:44:26.495384 6583 utils.go:76] GRPC request: {"target_path":"/var/lib/kubelet/pods/30befd9c-d850-464f-9cb9-9ddb789ab67f/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0308 20:44:26.495443 6583 nodeserver.go:199] NodeUnpublishVolume: unmounting volume rg-backups-prod#stpabackupsprod#backups-prod on /var/lib/kubelet/pods/30befd9c-d850-464f-9cb9-9ddb789ab67f/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0308 20:44:26.495461 6583 mount_helper_common.go:93] unmounting "/var/lib/kubelet/pods/30befd9c-d850-464f-9cb9-9ddb789ab67f/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount" (corruptedMount: false, mounterCanSkipMountPointChecks: true)
I0308 20:44:26.495474 6583 mount_linux.go:361] Unmounting /var/lib/kubelet/pods/30befd9c-d850-464f-9cb9-9ddb789ab67f/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0308 20:44:26.496899 6583 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/pods/30befd9c-d850-464f-9cb9-9ddb789ab67f/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount"
E0308 20:44:26.497033 6583 utils.go:80] GRPC error: rpc error: code = Internal desc = failed to unmount target "/var/lib/kubelet/pods/30befd9c-d850-464f-9cb9-9ddb789ab67f/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount": remove /var/lib/kubelet/pods/30befd9c-d850-464f-9cb9-9ddb789ab67f/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount: device or resource busy
I0308 20:44:26.998468 6583 utils.go:75] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0308 20:44:26.998492 6583 utils.go:76] GRPC request: {"target_path":"/var/lib/kubelet/pods/30befd9c-d850-464f-9cb9-9ddb789ab67f/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0308 20:44:26.998561 6583 nodeserver.go:199] NodeUnpublishVolume: unmounting volume rg-backups-prod#stpabackupsprod#backups-prod on /var/lib/kubelet/pods/30befd9c-d850-464f-9cb9-9ddb789ab67f/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0308 20:44:26.998583 6583 mount_helper_common.go:93] unmounting "/var/lib/kubelet/pods/30befd9c-d850-464f-9cb9-9ddb789ab67f/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount" (corruptedMount: false, mounterCanSkipMountPointChecks: true)
I0308 20:44:26.998598 6583 mount_linux.go:361] Unmounting /var/lib/kubelet/pods/30befd9c-d850-464f-9cb9-9ddb789ab67f/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0308 20:44:27.000033 6583 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/pods/30befd9c-d850-464f-9cb9-9ddb789ab67f/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount"
I0308 20:44:27.000108 6583 nodeserver.go:204] NodeUnpublishVolume: unmount volume rg-backups-prod#stpabackupsprod#backups-prod on /var/lib/kubelet/pods/30befd9c-d850-464f-9cb9-9ddb789ab67f/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount successfully
I0308 20:44:27.000119 6583 utils.go:82] GRPC response: {}
# After second deletion of the pod
I0308 20:45:33.367209 6583 utils.go:75] GRPC call: /csi.v1.Node/NodePublishVolume
I0308 20:45:33.367220 6583 utils.go:76] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount","target_path":"/var/lib/kubelet/pods/0051302b-af83-404f-b041-556a95df86c5/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["-o allow_other","--file-cache-timeout-in-seconds=120","--use-attr-cache=true","-o attr_timeout=120","-o entry_timeout=120","-o negative_timeout=120","--log-level=LOG_DEBUG","--cache-size-mb=1000","--virtual-directory=true"]}},"access_mode":{"mode":3}},"volume_context":{"containerName":"backups-prod","csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"orientdbservice-58c79ffb79-tvgr5","csi.storage.k8s.io/pod.namespace":"pa-data-services","csi.storage.k8s.io/pod.uid":"0051302b-af83-404f-b041-556a95df86c5","csi.storage.k8s.io/serviceAccount.name":"default","resourceGroup":"rg-backups-prod"},"volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0308 20:45:33.367598 6583 nodeserver.go:122] NodePublishVolume: volume rg-backups-prod#stpabackupsprod#backups-prod mounting /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount at /var/lib/kubelet/pods/0051302b-af83-404f-b041-556a95df86c5/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount with mountOptions: [bind]
I0308 20:45:33.367615 6583 mount_linux.go:219] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount /var/lib/kubelet/pods/0051302b-af83-404f-b041-556a95df86c5/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount)
I0308 20:45:33.368790 6583 mount_linux.go:219] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount /var/lib/kubelet/pods/0051302b-af83-404f-b041-556a95df86c5/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount)
I0308 20:45:33.369743 6583 nodeserver.go:138] NodePublishVolume: volume rg-backups-prod#stpabackupsprod#backups-prod mount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount at /var/lib/kubelet/pods/0051302b-af83-404f-b041-556a95df86c5/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount successfully
I0308 20:45:33.369764 6583 utils.go:82] GRPC response: {}
I0308 20:45:34.573365 6583 utils.go:75] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0308 20:45:34.573379 6583 utils.go:76] GRPC request: {"target_path":"/var/lib/kubelet/pods/07a6dec5-faba-451e-a9c3-f95dd627ad8a/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0308 20:45:34.573418 6583 nodeserver.go:199] NodeUnpublishVolume: unmounting volume rg-backups-prod#stpabackupsprod#backups-prod on /var/lib/kubelet/pods/07a6dec5-faba-451e-a9c3-f95dd627ad8a/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0308 20:45:34.573440 6583 mount_helper_common.go:93] unmounting "/var/lib/kubelet/pods/07a6dec5-faba-451e-a9c3-f95dd627ad8a/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount" (corruptedMount: false, mounterCanSkipMountPointChecks: true)
I0308 20:45:34.573464 6583 mount_linux.go:361] Unmounting /var/lib/kubelet/pods/07a6dec5-faba-451e-a9c3-f95dd627ad8a/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0308 20:45:34.575595 6583 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/pods/07a6dec5-faba-451e-a9c3-f95dd627ad8a/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount"
I0308 20:45:34.575655 6583 nodeserver.go:204] NodeUnpublishVolume: unmount volume rg-backups-prod#stpabackupsprod#backups-prod on /var/lib/kubelet/pods/07a6dec5-faba-451e-a9c3-f95dd627ad8a/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount successfully
I0308 20:45:34.575666 6583 utils.go:82] GRPC response: {}btw, this is my PV mount options apiVersion: v1
kind: PersistentVolume
metadata:
name: yourPV
spec:
capacity:
storage: 500Gi
accessModes:
- ReadOnlyMany
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: yourPVC
namespace: pa-data-services
# Retain allows the feature that even deleting the PVC & PV the external Azure Blob Container or Azure File Shares WILL NOT get deleted.
# That is quite important, because that means even somehow the ENTIRE Kubernetes cluster is gone. We still have our data.
persistentVolumeReclaimPolicy: Retain
storageClassName: azureblob-fuse-premium
csi:
# This driver, however, does not come pre-installed when creating an AKS. Also, it seems to be in preview.
# So there is an elaborated steps to enable and install it.
# See the README.md section of: ## About blob-csi-driver
driver: blob.csi.azure.com
readOnly: false
# make sure this volumeid is unique in the cluster
# `#` is not allowed in self defined volumeHandle
volumeHandle: rg-backups-prod#stpabackupsprod#backups-prod
volumeAttributes:
resourceGroup: rG
containerName: containerName
nodeStageSecretRef:
name: stpabackupsprod
namespace: pa-data-services
mountOptions:
- '-o allow_other'
- '--file-cache-timeout-in-seconds=120'
- '--use-attr-cache=true'
- '-o attr_timeout=120'
- '-o entry_timeout=120'
- '-o negative_timeout=120'
- '--log-level=LOG_DEBUG'
- '--cache-size-mb=1000'
- '--virtual-directory=true'Wait a minute... I just re-read the whole Use Azure Blob storage Container Storage Interface (CSI) driver documentation by Azure. Specifically, the following paragraph & callout box, does that mean
This is what the kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: azureblob-nfs-premium
labels:
addonmanager.kubernetes.io/mode: EnsureExists
kubernetes.io/cluster-service: 'true'
provisioner: blob.csi.azure.com
parameters:
protocol: nfs
skuName: Premium_LRS
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: Immediate |
|
Blobfuse has no restriciton related to HNS, you can use it for both FNS and HNS. |
CC: @andyzhangx Hm... but when I tried to mount an Azure Storage Account that does not have HNS & does not have NFS v3 enabled using But if I were to use the same Just a reminder, all these are under a specific version AKS, that is
|
|
@pa-mc first error is expected since NFSv3 is not enabled in your account in the beginning. |
|
btw, for HNS support, you need to set
|
@andyzhangx thanks for the clarification! In my case, I am using static provisioning instead of dynamic provisioning. A related question: How do I determine which version of blobfuse AKS is using? Or how do I force Because when I tried to use ❯ kubectl exec -it csi-blob-node-lfkx9 -c blob -n kube-system -- /bin/sh
# blobfuse -v
blobfuse 1.4.5
# blobfuse2 -v
blobfuse2 version 2.0.2 |
|
@pa-mc AKS 1.25 is on Ubuntu 22.04 and it only supports blobfuse v2, and before 1.25, AKS is on Ubuntu 18.04, both blobfuse v1 & v2 are supported. I would suggest using |
|
@andyzhangx @vibhansa-msft Sorry for the long reply below, but I think this will illustrate my specific edge case that causes blobfuse2 to break, please hear me out :) The ScenarioWe have a database (OrientDB) backup daily on an Azure blob store that does not support NFS v3 protocol, so all I can work with is Blobfuse v1 or v2. azure blob store property
I have two cluster one is Now here is the specific scenario, due to the limitation with OrientDB (long story... I'll spare you the details here...), for each data ETL run, we have to kill the
Starting with
|
|
@pa-mc thanks for the sharing. So with blobfuse2 on AKS 1.24 or 1.25, there is |
|
If some operation is going on or there is a shell connected to that path, unmount is expected to fail. Before you kill the pod you shall do a graceful unmount otherwise there are chances of data loss as well. If blobfuse was in between a transfer and you just killed the pod or container then transfer will be terminated, and next mount will lose the locally cached data as well. |
@vibhansa-msft one point is that with v1, it does not have such |
|
There is no difference in unmount process as such, also unmount is triggered from kernel and we just get a call from the system to shutdown the binary. Ideally I would expect both to fail in case some operations are going on. Quite possible that v1 was not reporting the problem while v2 does. |
Yes, for the following two combinations, the first pod deletion you will see the
Also I don't quite understand why would restart of a pod, why would CSI do a for instance# second delete
I0309 15:27:44.061632 7519 utils.go:75] GRPC call: /csi.v1.Node/NodePublishVolume
I0309 15:27:44.061653 7519 utils.go:76] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount","target_path":"/var/lib/kubelet/pods/42c84958-666e-467c-b86e-a7fcd70e3a26/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["-o allow_other","--file-cache-timeout-in-seconds=120","--use-attr-cache=true","-o attr_timeout=120","-o entry_timeout=120","-o negative_timeout=120","--log-level=LOG_DEBUG","--cache-size-mb=1000","--virtual-directory=true"]}},"access_mode":{"mode":3}},"volume_context":{"containerName":"backups-prod","csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"orientdbservice-57b7898756-bmggl","csi.storage.k8s.io/pod.namespace":"pa-data-services","csi.storage.k8s.io/pod.uid":"42c84958-666e-467c-b86e-a7fcd70e3a26","csi.storage.k8s.io/serviceAccount.name":"default","protocol":"fuse2","resourceGroup":"rg-backups-prod","storageAccount":"stpabackupsprod"},"volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0309 15:27:44.062199 7519 nodeserver.go:122] NodePublishVolume: volume rg-backups-prod#stpabackupsprod#backups-prod mounting /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount at /var/lib/kubelet/pods/42c84958-666e-467c-b86e-a7fcd70e3a26/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount with mountOptions: [bind]
I0309 15:27:44.062282 7519 mount_linux.go:219] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount /var/lib/kubelet/pods/42c84958-666e-467c-b86e-a7fcd70e3a26/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount)
I0309 15:27:44.063724 7519 mount_linux.go:219] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount /var/lib/kubelet/pods/42c84958-666e-467c-b86e-a7fcd70e3a26/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount)
I0309 15:27:44.065165 7519 nodeserver.go:138] NodePublishVolume: volume rg-backups-prod#stpabackupsprod#backups-prod mount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount at /var/lib/kubelet/pods/42c84958-666e-467c-b86e-a7fcd70e3a26/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount successfully
I0309 15:27:44.065183 7519 utils.go:82] GRPC response: {}
I0309 15:27:45.268586 7519 utils.go:75] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0309 15:27:45.268602 7519 utils.go:76] GRPC request: {"target_path":"/var/lib/kubelet/pods/148f2d31-08ed-4af0-85b9-880ecf31d661/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0309 15:27:45.268656 7519 nodeserver.go:199] NodeUnpublishVolume: unmounting volume rg-backups-prod#stpabackupsprod#backups-prod on /var/lib/kubelet/pods/148f2d31-08ed-4af0-85b9-880ecf31d661/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0309 15:27:45.268782 7519 mount_helper_common.go:93] unmounting "/var/lib/kubelet/pods/148f2d31-08ed-4af0-85b9-880ecf31d661/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount" (corruptedMount: false, mounterCanSkipMountPointChecks: true)
I0309 15:27:45.268792 7519 mount_linux.go:361] Unmounting /var/lib/kubelet/pods/148f2d31-08ed-4af0-85b9-880ecf31d661/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0309 15:27:45.271301 7519 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/pods/148f2d31-08ed-4af0-85b9-880ecf31d661/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount"
I0309 15:27:45.271358 7519 nodeserver.go:204] NodeUnpublishVolume: unmount volume rg-backups-prod#stpabackupsprod#backups-prod on /var/lib/kubelet/pods/148f2d31-08ed-4af0-85b9-880ecf31d661/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount successfully
I0309 15:27:45.271367 7519 utils.go:82] GRPC response: {} |
I'm pretty sure there is nothing connected to that path while I did the experiments except the pod that has the path mounted to it. In this case, I did deliberately restarted the Pod, but in practice I have no control over when or if the Pod would restart or not. The graceful unmount should be handled by the CSI driver or k8s Pod API right? |
|
I'm not sure if this is asking too much, but it would be great if we get on a zoom / team call so that I can screenshare with you guys, much better than reading my super long experiments :P |
@pa-mc there are two pods in the
that's expected. |
Right, make sense. Because the killing of a Pod will remove the old Pod and create a new Pod under the same Deployment. I guess I just thought the /NodeUnpublishVolume would come first. Since the deletion of a Pod happend first. However, that's just a side note, the key mystry for me is why this behaviour wouldn't happen for |
|
Hi @pa-mc, what's the business impact for this behavior change? My assumption is that on AKS 1.25, it would create a new replica first, and then delete old replica, that's the reason why NodePublishVolume comes first and then NodeUnpublishVolume comes next. It could be related to |
Hi @andyzhangx, yeah right now it's not a huge deal just yet. Because our production AKS cluster is still in However, this is quite kludgy obviously, and I'd love to get to the bottom of this issue regardless things are "working" or not ;) I've created an Azure Support ticket P.S.: Someone from Azure support reached out, and we will do a Team screenshare to show them exactly how I produced that error. I asked them to record it if they can (i don't mind). If they did record it, I will share it with you guys. |
|
@pa-mc per our testing pipeline results, as long as you are using blobfuse v2 (no matter it's AKS 1.24 or 1.25), it would have such
So my assumption is that there could be implementation difference between v1 and v2, and it looks like there is no side effect since second time unmount always succeeds after around 0.6s |
|
Hi @andyzhangx sorry for the late reply, wife & son gotta pretty sick and the last man standing :P Thank you so much for running those tests, and I will find some time this week and dig into it. I think my experimentation also showed the unmount will be successfull after the second restart of a Pod. However, it's the mounting had some problem. Because after second restart of a Pod, even thoug it said Regardless, I will keep you updated with my findings and perhaps do a screenrecording it and share it here. |
@pa-mc if second restart would make pod broken, can you ssh to the pod, and run kubectl exec -it nginx-blob -- df -hFilesystem Size Used Avail Use% Mounted on ... blobfuse 14G 41M 13G 1% /mnt/blob ... |
Hi @andyzhangx I tried, yup, as you anticipated, after the second restart, there is no Here are the full comments & outputs
❯ kubectl exec -it -n pa-data-services deployments/orientdbservice -- /bin/bash
Defaulted container "orientdbservice" out of: orientdbservice, orientdbservicehandler
root@orientdbservice-58c79ffb79-bz7j9:/orientdb# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 248G 27G 222G 11% /
tmpfs 64M 0 64M 0% /dev
/dev/root 248G 27G 222G 11% /etc/hosts
blobfuse2 1000M 0 1000M 0% /mnt/stpabackupsprod
shm 64M 0 64M 0% /dev/shm
tmpfs 28G 12K 28G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 16G 0 16G 0% /proc/acpi
tmpfs 16G 0 16G 0% /proc/scsi
tmpfs 16G 0 16G 0% /sys/firmware
❯ kubectl exec -it -n pa-data-services deployments/orientdbservice -- /bin/bash
Defaulted container "orientdbservice" out of: orientdbservice, orientdbservicehandler
root@orientdbservice-58c79ffb79-j9rk5:/orientdb# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 248G 27G 222G 11% /
tmpfs 64M 0 64M 0% /dev
/dev/root 248G 27G 222G 11% /etc/hosts
blobfuse2 1000M 0 1000M 0% /mnt/stpabackupsprod
shm 64M 0 64M 0% /dev/shm
tmpfs 28G 12K 28G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 16G 0 16G 0% /proc/acpi
tmpfs 16G 0 16G 0% /proc/scsi
tmpfs 16G 0 16G 0% /sys/firmware
I0314 14:31:14.985754 5729 utils.go:82] GRPC response: {}
I0314 14:31:16.089754 5729 utils.go:75] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0314 14:31:16.089810 5729 utils.go:76] GRPC request: {"target_path":"/var/lib/kubelet/pods/dca3dffa-7785-47c8-a73f-02b144cea174/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0314 14:31:16.089871 5729 nodeserver.go:199] NodeUnpublishVolume: unmounting volume rg-backups-prod#stpabackupsprod#backups-prod on /var/lib/kubelet/pods/dca3dffa-7785-47c8-a73f-02b144cea174/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0314 14:31:16.089886 5729 mount_helper_common.go:93] unmounting "/var/lib/kubelet/pods/dca3dffa-7785-47c8-a73f-02b144cea174/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount" (corruptedMount: false, mounterCanSkipMountPointChecks: true)
I0314 14:31:16.089897 5729 mount_linux.go:361] Unmounting /var/lib/kubelet/pods/dca3dffa-7785-47c8-a73f-02b144cea174/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0314 14:31:16.091324 5729 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/pods/dca3dffa-7785-47c8-a73f-02b144cea174/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount"
E0314 14:31:16.091364 5729 utils.go:80] GRPC error: rpc error: code = Internal desc = failed to unmount target "/var/lib/kubelet/pods/dca3dffa-7785-47c8-a73f-02b144cea174/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount": remove /var/lib/kubelet/pods/dca3dffa-7785-47c8-a73f-02b144cea174/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount: device or resource busy
I0314 14:31:16.592842 5729 utils.go:75] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0314 14:31:16.592857 5729 utils.go:76] GRPC request: {"target_path":"/var/lib/kubelet/pods/dca3dffa-7785-47c8-a73f-02b144cea174/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0314 14:31:16.592912 5729 nodeserver.go:199] NodeUnpublishVolume: unmounting volume rg-backups-prod#stpabackupsprod#backups-prod on /var/lib/kubelet/pods/dca3dffa-7785-47c8-a73f-02b144cea174/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0314 14:31:16.592933 5729 mount_helper_common.go:93] unmounting "/var/lib/kubelet/pods/dca3dffa-7785-47c8-a73f-02b144cea174/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount" (corruptedMount: false, mounterCanSkipMountPointChecks: true)
I0314 14:31:16.592947 5729 mount_linux.go:361] Unmounting /var/lib/kubelet/pods/dca3dffa-7785-47c8-a73f-02b144cea174/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0314 14:31:16.594108 5729 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/pods/dca3dffa-7785-47c8-a73f-02b144cea174/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount"
I0314 14:31:16.594186 5729 nodeserver.go:204] NodeUnpublishVolume: unmount volume rg-backups-prod#stpabackupsprod#backups-prod on /var/lib/kubelet/pods/dca3dffa-7785-47c8-a73f-02b144cea174/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount successfully
I0314 14:31:16.594200 5729 utils.go:82] GRPC response: {}
❯ kubectl exec -it -n pa-data-services deployments/orientdbservice -- /bin/bash
Defaulted container "orientdbservice" out of: orientdbservice, orientdbservicehandler
root@orientdbservice-58c79ffb79-phddp:/orientdb# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 248G 27G 222G 11% /
tmpfs 64M 0 64M 0% /dev
/dev/root 248G 27G 222G 11% /etc/hosts
shm 64M 0 64M 0% /dev/shm
tmpfs 28G 12K 28G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 16G 0 16G 0% /proc/acpi
tmpfs 16G 0 16G 0% /proc/scsi
tmpfs 16G 0 16G 0% /sys/firmware
all the manifest files I used
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-stpabackupsprod-blob-container
spec:
capacity:
storage: 500Gi
accessModes:
- ReadOnlyMany
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: pvc-stpabackupsprod-blob-container
namespace: pa-data-services
persistentVolumeReclaimPolicy: Retain
storageClassName: azureblob-fuse-premium
csi:
driver: blob.csi.azure.com
readOnly: false
volumeHandle: rg-backups-prod#stpabackupsprod#backups-prod
volumeAttributes:
resourceGroup: rg-backups-prod
storageAccount: stpabackupsprod
containerName: backups-prod
protocol: fuse2
nodeStageSecretRef:
name: stpabackupsprod
namespace: pa-data-services
mountOptions:
- '-o allow_other'
- '--file-cache-timeout-in-seconds=120'
- '--use-attr-cache=true'
- '--cancel-list-on-mount-seconds=5'
- '-o attr_timeout=120'
- '-o entry_timeout=120'
- '-o negative_timeout=120'
- '--log-level=LOG_DEBUG'
- '--cache-size-mb=1000'
- '--virtual-directory=true'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-stpabackupsprod-blob-container
namespace: pa-data-services
labels:
service: orientdb
type: pv-claim
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 500Gi
volumeName: pv-stpabackupsprod-blob-container
storageClassName: azureblob-nfs-premium
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: orientdbservice
namespace: pa-data-services
labels:
service: orientdb
spec:
replicas: 1
selector:
matchLabels:
service: orientdb
template:
metadata:
name: orientdbservice
labels:
service: orientdb
type: container-deployment
spec:
containers:
- name: orientdbservice
image: datacontainers.azurecr.io/pa-orientdb-enterprise:main
resources:
requests:
cpu: 2000m
memory: 12Gi
imagePullPolicy: Always
env:
- name: ORIENTDB_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: pa-data-secrets
key: ORIENTDB_ROOT_PASSWORD
- name: ORIENTDB_OPTS_MEMORY
value: "-Xms2g -Xmx12g -Dstorage.diskCache.bufferSize=36000 -Dstorage.useWAL=true -server -XX:+PerfDisableSharedMem"
ports:
- containerPort: 2424
name: port-binary
- containerPort: 2480
name: port-http
volumeMounts:
- mountPath: /orientdb/databases
name: orientdb-data
subPath: databases
- mountPath: /orientdb/backup
name: orientdb-data
subPath: backup
- mountPath: /mnt/stpabackupsprod
name: orientdb-stpabackupsprod
- name: orientdbservicehandler
image: datacontainers.azurecr.io/odb-handler:main
resources: {}
imagePullPolicy: Always
env:
- name: APP_HOST
value: "0.0.0.0"
- name: AZURE_STORAGE_CONNECTION_STRING
valueFrom:
secretKeyRef:
name: stpabackupsprod
key: AZURE_STORAGE_CONNECTION_STRING
- name: AZURE_STORAGE_AUTH_MODE
valueFrom:
configMapKeyRef:
name: odb-handler-configs
key: AZURE_STORAGE_AUTH_MODE
- name: AZURE_STORAGE_CONTAINER_NAME
valueFrom:
configMapKeyRef:
name: odb-handler-configs
key: AZURE_STORAGE_CONTAINER_NAME
- name: COMMAND_ENVIRONMENT
valueFrom:
configMapKeyRef:
name: odb-handler-configs
key: COMMAND_ENVIRONMENT
- name: KUBERNETES_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: KUBERNETES_POD
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: KUBERNETES_IN_CLUSTER
valueFrom:
configMapKeyRef:
name: odb-handler-configs
key: KUBERNETES_IN_CLUSTER
- name: BACKUP_CLIENTMAP
valueFrom:
configMapKeyRef:
name: odb-handler-configs
key: BACKUP_CLIENTMAP
- name: BACKUP_PARSE_FORMAT
valueFrom:
configMapKeyRef:
name: odb-handler-configs
key: BACKUP_PARSE_FORMAT
- name: BACKUP_KWARG_DEFAULTS
valueFrom:
configMapKeyRef:
name: odb-handler-configs
key: BACKUP_KWARG_DEFAULTS
- name: BACKUP_JINJA_FORMAT
valueFrom:
configMapKeyRef:
name: odb-handler-configs
key: BACKUP_JINJA_FORMAT
ports:
- containerPort: 8000
name: odbhandlerport
imagePullSecrets:
- name: pa-datacontainers-acr
volumes:
- name: orientdb-data
emptyDir: {}
- name: orientdb-stpabackupsprod
persistentVolumeClaim:
claimName: pvc-stpabackupsprod-blob-container
tolerations:
- key: "dbcluster"
operator: "Equal"
value: "true"
effect: "NoSchedule"
nodeSelector:
agentpool: dbcluster
---
kind: Service
apiVersion: v1
metadata:
name: orientdbservice
namespace: pa-data-services
labels:
service: orientdb
type: nodeport-service
spec:
type: NodePort
selector:
service: orientdb
type: container-deployment
ports:
- protocol: TCP
port: 2480
name: http
---
kind: Service
apiVersion: v1
metadata:
name: orientdbservice-private
namespace: pa-data-services
labels:
service: orientdb
type: private-orientdbservice-load-balancer-service
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
type: LoadBalancer
loadBalancerIP: 10.224.0.10
ports:
- protocol: TCP
port: 2424
name: odb-binary
- protocol: TCP
port: 8000
name: odb-handler-http
selector:
service: orientdb
type: container-deploymentP.S.: I think you 100% know what exactly my sceniaro is, but I will try to find some time today to do a screen recording and share it anyway. Also, I really appreciate for all the help! |
|
I have reproed the above issue, the second pod deletion would make blobfuse mount broken. I find where the problem is, blob csi driver would mount blob storage to a globalmount dir first, and then bind mount during
|
|
@andyzhangx @vibhansa-msft Two 🐐 s 😄 So @andyzhangx It's alright, I'm not in a hurry right now, I can wait till the the AKS release to happen. I've changed my |
|
I just filed a PR to solve the problem from blobfuse2 side, please take a look. #1088 @vibhansa-msft @pa-mc @andyzhangx |
@pa-mc pls check on https://github.com/Azure/AKS/releases, and the fix would be in 0319 release, and https://releases.aks.azure.com/ site would show you the release rollout progress on every region. This csi driver version upgrade would happen in the backend, you don't need to do anything. And yes, NFS mount is safe now on AKS 1.25 |
Hi @vibhansa-msft Thank you very much. |
vibhansa-msft
pushed a commit
that referenced
this issue
Mar 22, 2023
#1079 Pod deletion will cause global mount point to disappear. The root cause is csi driver will bind mount the global mount point into Pod volume immediately after blobfuse2 mount return successfully. However, fuse is still mounting at background, so actually bind mount and fuse mount happened at the same time, this will lead to a problem that unmount the bind mount (when deleting a Pod) will cause the original mount point unmounted as well. #1081 Mount is actually failed, but the errno returned by blobfuse2 is 0 and no error log (both terminal and log file).
|
Fixed with #1088 |
|
Thanks for all the help @vibhansa-msft @andyzhangx @cvvz !! Here is the BlobfuseV2 CSI driver logs after the patch. No longer GRPC errors, and mount persists after numerous restart of the Pod :) Again, thank you so much all of your help!❯ kubectl logs csi-blob-node-44xp2 -c blob -n kube-system -f
I0323 15:02:03.652753 5917 main.go:117] set up prometheus server on [::]:29635
I0323 15:02:03.653280 5917 blob.go:235]
DRIVER INFORMATION:
-------------------
Build Date: "2023-03-18T08:11:26Z"
Compiler: gc
Driver Name: blob.csi.azure.com
Driver Version: v1.19.2
Git Commit: b313e4f5b30b4cc62ac5fd37041c0b85299a2cf1
Go Version: go1.19.1
Platform: linux/amd64
Streaming logs below:
I0323 15:02:03.653297 5917 blob.go:238] driver userAgent: blob.csi.azure.com/v1.19.2 AKS
I0323 15:02:03.653408 5917 azure.go:73] set QPS(25.000000) and QPS Burst(50) for driver kubeClient
I0323 15:02:03.653628 5917 azure.go:87] reading cloud config from secret kube-system/azure-cloud-provider
I0323 15:02:03.706825 5917 azure.go:94] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0323 15:02:03.706842 5917 azure.go:99] could not read cloud config from secret kube-system/azure-cloud-provider
I0323 15:02:03.706847 5917 azure.go:102] AZURE_CREDENTIAL_FILE env var set as /etc/kubernetes/azure.json
I0323 15:02:03.706869 5917 azure.go:113] read cloud config from file: /etc/kubernetes/azure.json successfully
I0323 15:02:03.707416 5917 azure_auth.go:245] Using AzurePublicCloud environment
I0323 15:02:03.707432 5917 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I0323 15:02:03.707436 5917 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
I0323 15:02:03.707463 5917 azure_auth.go:113] azure: User Assigned MSI ID is client ID
I0323 15:02:03.707498 5917 azure.go:775] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000
I0323 15:02:03.707528 5917 azure_interfaceclient.go:74] Azure InterfacesClient (read ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707536 5917 azure_interfaceclient.go:77] Azure InterfacesClient (write ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707546 5917 azure_vmsizeclient.go:68] Azure VirtualMachineSizesClient (read ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707549 5917 azure_vmsizeclient.go:71] Azure VirtualMachineSizesClient (write ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707558 5917 azure_snapshotclient.go:70] Azure SnapshotClient (read ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707561 5917 azure_snapshotclient.go:73] Azure SnapshotClient (write ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707569 5917 azure_storageaccountclient.go:70] Azure StorageAccountClient (read ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707575 5917 azure_storageaccountclient.go:73] Azure StorageAccountClient (write ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707582 5917 azure_diskclient.go:68] Azure DisksClient using API version: 2022-03-02
I0323 15:02:03.707588 5917 azure_diskclient.go:73] Azure DisksClient (read ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707591 5917 azure_diskclient.go:76] Azure DisksClient (write ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707597 5917 azure_vmclient.go:70] Azure VirtualMachine client (read ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707602 5917 azure_vmclient.go:73] Azure VirtualMachine client (write ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707607 5917 azure_vmssclient.go:70] Azure VirtualMachineScaleSetClient (read ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707610 5917 azure_vmssclient.go:73] Azure VirtualMachineScaleSetClient (write ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707617 5917 azure_vmssvmclient.go:75] Azure vmssVM client (read ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707622 5917 azure_vmssvmclient.go:78] Azure vmssVM client (write ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707631 5917 azure_routeclient.go:69] Azure RoutesClient (read ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707634 5917 azure_routeclient.go:72] Azure RoutesClient (write ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707641 5917 azure_subnetclient.go:70] Azure SubnetsClient (read ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707646 5917 azure_subnetclient.go:73] Azure SubnetsClient (write ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707651 5917 azure_routetableclient.go:69] Azure RouteTablesClient (read ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707654 5917 azure_routetableclient.go:72] Azure RouteTablesClient (write ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707661 5917 azure_loadbalancerclient.go:70] Azure LoadBalancersClient (read ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707666 5917 azure_loadbalancerclient.go:73] Azure LoadBalancersClient (write ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707674 5917 azure_securitygroupclient.go:70] Azure SecurityGroupsClient (read ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707687 5917 azure_securitygroupclient.go:73] Azure SecurityGroupsClient (write ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707693 5917 azure_publicipclient.go:74] Azure PublicIPAddressesClient (read ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707699 5917 azure_publicipclient.go:77] Azure PublicIPAddressesClient (write ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707710 5917 azure_blobclient.go:67] Azure BlobClient using API version: 2021-09-01
I0323 15:02:03.707723 5917 azure_vmasclient.go:70] Azure AvailabilitySetsClient (read ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707726 5917 azure_vmasclient.go:73] Azure AvailabilitySetsClient (write ops) using rate limit config: QPS=10, bucket=100
I0323 15:02:03.707773 5917 azure.go:1006] attach/detach disk operation rate limit QPS: 6.000000, Bucket: 10
I0323 15:02:03.707784 5917 azure.go:149] starting node server on node(aks-dbcluster-58864392-vmss00000w)
I0323 15:02:03.707794 5917 blob.go:243] cloud: AzurePublicCloud, location: canadacentral, rg: pa-data-pa-data-aks-devtest-rg-node-rg, VnetName: aks-vnet-20107475, VnetResourceGroup: , SubnetName: aks-subnet
I0323 15:02:03.709518 5917 mount_linux.go:283] Detected umount with safe 'not mounted' behavior
I0323 15:02:03.709561 5917 driver.go:80] Enabling controller service capability: CREATE_DELETE_VOLUME
I0323 15:02:03.709568 5917 driver.go:80] Enabling controller service capability: EXPAND_VOLUME
I0323 15:02:03.709571 5917 driver.go:80] Enabling controller service capability: SINGLE_NODE_MULTI_WRITER
I0323 15:02:03.709575 5917 driver.go:99] Enabling volume access mode: SINGLE_NODE_WRITER
I0323 15:02:03.709579 5917 driver.go:99] Enabling volume access mode: SINGLE_NODE_READER_ONLY
I0323 15:02:03.709584 5917 driver.go:99] Enabling volume access mode: SINGLE_NODE_SINGLE_WRITER
I0323 15:02:03.709588 5917 driver.go:99] Enabling volume access mode: SINGLE_NODE_MULTI_WRITER
I0323 15:02:03.709593 5917 driver.go:99] Enabling volume access mode: MULTI_NODE_READER_ONLY
I0323 15:02:03.709596 5917 driver.go:99] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER
I0323 15:02:03.709603 5917 driver.go:99] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I0323 15:02:03.709607 5917 driver.go:90] Enabling node service capability: STAGE_UNSTAGE_VOLUME
I0323 15:02:03.709612 5917 driver.go:90] Enabling node service capability: SINGLE_NODE_MULTI_WRITER
I0323 15:02:03.709722 5917 server.go:114] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}
I0323 15:02:03.841253 5917 utils.go:75] GRPC call: /csi.v1.Identity/GetPluginInfo
I0323 15:02:03.841268 5917 utils.go:76] GRPC request: {}
I0323 15:02:03.843209 5917 utils.go:82] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.19.2"}
I0323 15:02:04.627990 5917 utils.go:75] GRPC call: /csi.v1.Identity/GetPluginInfo
I0323 15:02:04.628002 5917 utils.go:76] GRPC request: {}
I0323 15:02:04.628044 5917 utils.go:82] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.19.2"}
I0323 15:02:04.784151 5917 utils.go:75] GRPC call: /csi.v1.Node/NodeGetInfo
I0323 15:02:04.784168 5917 utils.go:76] GRPC request: {}
I0323 15:02:04.784233 5917 utils.go:82] GRPC response: {"node_id":"aks-dbcluster-58864392-vmss00000w"}
I0323 15:02:55.096971 5917 utils.go:75] GRPC call: /csi.v1.Node/NodeStageVolume
I0323 15:02:55.096987 5917 utils.go:76] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/40d24c216b1da97a64a7ec1a32b5019a13c40b0172ad3fa7a0af09e4bb96c5a8/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":3}},"volume_context":{"containerName":"sync-backups-prod","protocol":"nfs","resourceGroup":"devtest-datalake","storageAccount":"nfssyncstpabackupsprod"},"volume_id":"devtest-datalake#nfssyncstpabackupsprod#sync-backups-prod"}
I0323 15:02:55.097295 5917 blob.go:416] volumeID(devtest-datalake#nfssyncstpabackupsprod#sync-backups-prod) authEnv: []
I0323 15:02:55.097307 5917 nodeserver.go:311] target /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/40d24c216b1da97a64a7ec1a32b5019a13c40b0172ad3fa7a0af09e4bb96c5a8/globalmount
protocol nfs
volumeId devtest-datalake#nfssyncstpabackupsprod#sync-backups-prod
context map[containerName:sync-backups-prod protocol:nfs resourceGroup:devtest-datalake storageAccount:nfssyncstpabackupsprod]
mountflags []
serverAddress nfssyncstpabackupsprod.blob.core.windows.net
I0323 15:02:55.097373 5917 mount_linux.go:244] Detected OS without systemd
I0323 15:02:55.097380 5917 mount_linux.go:219] Mounting cmd (mount) with arguments (-t nfs -o sec=sys,vers=3,nolock nfssyncstpabackupsprod.blob.core.windows.net:/nfssyncstpabackupsprod/sync-backups-prod /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/40d24c216b1da97a64a7ec1a32b5019a13c40b0172ad3fa7a0af09e4bb96c5a8/globalmount)
I0323 15:02:58.320611 5917 blob.go:856] skip chmod on targetPath(/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/40d24c216b1da97a64a7ec1a32b5019a13c40b0172ad3fa7a0af09e4bb96c5a8/globalmount) since mode is already 020000000777)
I0323 15:02:58.320647 5917 nodeserver.go:330] volume(devtest-datalake#nfssyncstpabackupsprod#sync-backups-prod) mount nfssyncstpabackupsprod.blob.core.windows.net:/nfssyncstpabackupsprod/sync-backups-prod on /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/40d24c216b1da97a64a7ec1a32b5019a13c40b0172ad3fa7a0af09e4bb96c5a8/globalmount succeeded
I0323 15:02:58.320660 5917 utils.go:82] GRPC response: {}
I0323 15:02:58.323347 5917 utils.go:75] GRPC call: /csi.v1.Node/NodePublishVolume
I0323 15:02:58.323360 5917 utils.go:76] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/40d24c216b1da97a64a7ec1a32b5019a13c40b0172ad3fa7a0af09e4bb96c5a8/globalmount","target_path":"/var/lib/kubelet/pods/7bbb3fd3-7baa-4aa3-9887-30a3e3560092/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":3}},"volume_context":{"containerName":"sync-backups-prod","csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"orientdbservice-58c79ffb79-mfx94","csi.storage.k8s.io/pod.namespace":"pa-data-services","csi.storage.k8s.io/pod.uid":"7bbb3fd3-7baa-4aa3-9887-30a3e3560092","csi.storage.k8s.io/serviceAccount.name":"default","protocol":"nfs","resourceGroup":"devtest-datalake","storageAccount":"nfssyncstpabackupsprod"},"volume_id":"devtest-datalake#nfssyncstpabackupsprod#sync-backups-prod"}
I0323 15:02:58.323694 5917 nodeserver.go:122] NodePublishVolume: volume devtest-datalake#nfssyncstpabackupsprod#sync-backups-prod mounting /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/40d24c216b1da97a64a7ec1a32b5019a13c40b0172ad3fa7a0af09e4bb96c5a8/globalmount at /var/lib/kubelet/pods/7bbb3fd3-7baa-4aa3-9887-30a3e3560092/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount with mountOptions: [bind]
I0323 15:02:58.323713 5917 mount_linux.go:219] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/40d24c216b1da97a64a7ec1a32b5019a13c40b0172ad3fa7a0af09e4bb96c5a8/globalmount /var/lib/kubelet/pods/7bbb3fd3-7baa-4aa3-9887-30a3e3560092/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount)
I0323 15:02:58.324736 5917 mount_linux.go:219] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/40d24c216b1da97a64a7ec1a32b5019a13c40b0172ad3fa7a0af09e4bb96c5a8/globalmount /var/lib/kubelet/pods/7bbb3fd3-7baa-4aa3-9887-30a3e3560092/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount)
I0323 15:02:58.325599 5917 nodeserver.go:138] NodePublishVolume: volume devtest-datalake#nfssyncstpabackupsprod#sync-backups-prod mount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/40d24c216b1da97a64a7ec1a32b5019a13c40b0172ad3fa7a0af09e4bb96c5a8/globalmount at /var/lib/kubelet/pods/7bbb3fd3-7baa-4aa3-9887-30a3e3560092/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount successfully
I0323 15:02:58.325607 5917 utils.go:82] GRPC response: {}
I0323 17:18:56.187847 5917 utils.go:75] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0323 17:18:56.187865 5917 utils.go:76] GRPC request: {"target_path":"/var/lib/kubelet/pods/7bbb3fd3-7baa-4aa3-9887-30a3e3560092/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_id":"devtest-datalake#nfssyncstpabackupsprod#sync-backups-prod"}
I0323 17:18:56.187947 5917 nodeserver.go:203] NodeUnpublishVolume: unmounting volume devtest-datalake#nfssyncstpabackupsprod#sync-backups-prod on /var/lib/kubelet/pods/7bbb3fd3-7baa-4aa3-9887-30a3e3560092/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0323 17:18:56.200704 5917 mount_helper_common.go:93] unmounting "/var/lib/kubelet/pods/7bbb3fd3-7baa-4aa3-9887-30a3e3560092/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount" (corruptedMount: false, mounterCanSkipMountPointChecks: true)
I0323 17:18:56.200725 5917 mount_linux.go:361] Unmounting /var/lib/kubelet/pods/7bbb3fd3-7baa-4aa3-9887-30a3e3560092/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0323 17:18:56.218001 5917 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/pods/7bbb3fd3-7baa-4aa3-9887-30a3e3560092/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount"
I0323 17:18:56.218204 5917 nodeserver.go:208] NodeUnpublishVolume: unmount volume devtest-datalake#nfssyncstpabackupsprod#sync-backups-prod on /var/lib/kubelet/pods/7bbb3fd3-7baa-4aa3-9887-30a3e3560092/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount successfully
I0323 17:18:56.218217 5917 utils.go:82] GRPC response: {}
I0323 17:18:56.290082 5917 utils.go:75] GRPC call: /csi.v1.Node/NodeUnstageVolume
I0323 17:18:56.290095 5917 utils.go:76] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/40d24c216b1da97a64a7ec1a32b5019a13c40b0172ad3fa7a0af09e4bb96c5a8/globalmount","volume_id":"devtest-datalake#nfssyncstpabackupsprod#sync-backups-prod"}
I0323 17:18:56.290159 5917 nodeserver.go:422] NodeUnstageVolume: volume devtest-datalake#nfssyncstpabackupsprod#sync-backups-prod unmounting on /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/40d24c216b1da97a64a7ec1a32b5019a13c40b0172ad3fa7a0af09e4bb96c5a8/globalmount
I0323 17:18:56.290182 5917 mount_helper_common.go:93] unmounting "/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/40d24c216b1da97a64a7ec1a32b5019a13c40b0172ad3fa7a0af09e4bb96c5a8/globalmount" (corruptedMount: false, mounterCanSkipMountPointChecks: true)
I0323 17:18:56.290196 5917 mount_linux.go:361] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/40d24c216b1da97a64a7ec1a32b5019a13c40b0172ad3fa7a0af09e4bb96c5a8/globalmount
I0323 17:18:59.347361 5917 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/40d24c216b1da97a64a7ec1a32b5019a13c40b0172ad3fa7a0af09e4bb96c5a8/globalmount"
I0323 17:18:59.347716 5917 nodeserver.go:427] NodeUnstageVolume: volume devtest-datalake#nfssyncstpabackupsprod#sync-backups-prod unmount on /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/40d24c216b1da97a64a7ec1a32b5019a13c40b0172ad3fa7a0af09e4bb96c5a8/globalmount successfully
I0323 17:18:59.347737 5917 utils.go:82] GRPC response: {}
I0323 17:21:03.958644 5917 utils.go:75] GRPC call: /csi.v1.Node/NodeStageVolume
I0323 17:21:03.958656 5917 utils.go:76] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["-o allow_other","--file-cache-timeout-in-seconds=120","--use-attr-cache=true","--cancel-list-on-mount-seconds=5","-o attr_timeout=120","-o entry_timeout=120","-o negative_timeout=120","--log-level=LOG_DEBUG","--cache-size-mb=1000","--virtual-directory=true"]}},"access_mode":{"mode":3}},"volume_context":{"containerName":"backups-prod","protocol":"fuse2","resourceGroup":"rg-backups-prod","storageAccount":"stpabackupsprod"},"volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0323 17:21:03.959018 5917 blob.go:416] volumeID(rg-backups-prod#stpabackupsprod#backups-prod) authEnv: []
I0323 17:21:03.959057 5917 nodeserver.go:353] target /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount
protocol fuse2
volumeId rg-backups-prod#stpabackupsprod#backups-prod
context map[containerName:backups-prod protocol:fuse2 resourceGroup:rg-backups-prod storageAccount:stpabackupsprod]
mountflags [-o allow_other --file-cache-timeout-in-seconds=120 --use-attr-cache=true --cancel-list-on-mount-seconds=5 -o attr_timeout=120 -o entry_timeout=120 -o negative_timeout=120 --log-level=LOG_DEBUG --cache-size-mb=1000 --virtual-directory=true]
mountOptions [-o allow_other --file-cache-timeout-in-seconds=120 --use-attr-cache=true --cancel-list-on-mount-seconds=5 -o attr_timeout=120 -o entry_timeout=120 -o negative_timeout=120 --log-level=LOG_DEBUG --cache-size-mb=1000 --virtual-directory=true --pre-mount-validate=true --use-https=true --empty-dir-check=false --tmp-path=/mnt/rg-backups-prod#stpabackupsprod#backups-prod --container-name=backups-prod]
args /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount -o allow_other --file-cache-timeout-in-seconds=120 --use-attr-cache=true --cancel-list-on-mount-seconds=5 -o attr_timeout=120 -o entry_timeout=120 -o negative_timeout=120 --log-level=LOG_DEBUG --cache-size-mb=1000 --virtual-directory=true --pre-mount-validate=true --use-https=true --empty-dir-check=false --tmp-path=/mnt/rg-backups-prod#stpabackupsprod#backups-prod --container-name=backups-prod
serverAddress stpabackupsprod.blob.core.windows.net
I0323 17:21:03.959084 5917 nodeserver.go:144] mouting using blobfuse proxy
I0323 17:21:03.959595 5917 nodeserver.go:158] calling BlobfuseProxy: MountAzureBlob function
I0323 17:21:06.856924 5917 nodeserver.go:401] volume(rg-backups-prod#stpabackupsprod#backups-prod) mount on "/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount" succeeded
I0323 17:21:06.856956 5917 utils.go:82] GRPC response: {}
I0323 17:21:06.859270 5917 utils.go:75] GRPC call: /csi.v1.Node/NodePublishVolume
I0323 17:21:06.859282 5917 utils.go:76] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount","target_path":"/var/lib/kubelet/pods/e25f0f5f-1027-4dc3-bc25-bb906fdee5e5/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["-o allow_other","--file-cache-timeout-in-seconds=120","--use-attr-cache=true","--cancel-list-on-mount-seconds=5","-o attr_timeout=120","-o entry_timeout=120","-o negative_timeout=120","--log-level=LOG_DEBUG","--cache-size-mb=1000","--virtual-directory=true"]}},"access_mode":{"mode":3}},"volume_context":{"containerName":"backups-prod","csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"orientdbservice-58c79ffb79-thh2f","csi.storage.k8s.io/pod.namespace":"pa-data-services","csi.storage.k8s.io/pod.uid":"e25f0f5f-1027-4dc3-bc25-bb906fdee5e5","csi.storage.k8s.io/serviceAccount.name":"default","protocol":"fuse2","resourceGroup":"rg-backups-prod","storageAccount":"stpabackupsprod"},"volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0323 17:21:06.859636 5917 nodeserver.go:122] NodePublishVolume: volume rg-backups-prod#stpabackupsprod#backups-prod mounting /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount at /var/lib/kubelet/pods/e25f0f5f-1027-4dc3-bc25-bb906fdee5e5/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount with mountOptions: [bind]
I0323 17:21:06.859651 5917 mount_linux.go:219] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount /var/lib/kubelet/pods/e25f0f5f-1027-4dc3-bc25-bb906fdee5e5/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount)
I0323 17:21:06.860741 5917 mount_linux.go:219] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount /var/lib/kubelet/pods/e25f0f5f-1027-4dc3-bc25-bb906fdee5e5/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount)
I0323 17:21:06.861609 5917 nodeserver.go:138] NodePublishVolume: volume rg-backups-prod#stpabackupsprod#backups-prod mount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount at /var/lib/kubelet/pods/e25f0f5f-1027-4dc3-bc25-bb906fdee5e5/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount successfully
I0323 17:21:06.861619 5917 utils.go:82] GRPC response: {}
I0323 17:22:34.743930 5917 utils.go:75] GRPC call: /csi.v1.Node/NodePublishVolume
I0323 17:22:34.743942 5917 utils.go:76] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount","target_path":"/var/lib/kubelet/pods/ced5f698-8b9a-4f2c-ab2d-7896f9bcd4ed/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["-o allow_other","--file-cache-timeout-in-seconds=120","--use-attr-cache=true","--cancel-list-on-mount-seconds=5","-o attr_timeout=120","-o entry_timeout=120","-o negative_timeout=120","--log-level=LOG_DEBUG","--cache-size-mb=1000","--virtual-directory=true"]}},"access_mode":{"mode":3}},"volume_context":{"containerName":"backups-prod","csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"orientdbservice-58c79ffb79-9r5cp","csi.storage.k8s.io/pod.namespace":"pa-data-services","csi.storage.k8s.io/pod.uid":"ced5f698-8b9a-4f2c-ab2d-7896f9bcd4ed","csi.storage.k8s.io/serviceAccount.name":"default","protocol":"fuse2","resourceGroup":"rg-backups-prod","storageAccount":"stpabackupsprod"},"volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0323 17:22:34.744362 5917 nodeserver.go:122] NodePublishVolume: volume rg-backups-prod#stpabackupsprod#backups-prod mounting /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount at /var/lib/kubelet/pods/ced5f698-8b9a-4f2c-ab2d-7896f9bcd4ed/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount with mountOptions: [bind]
I0323 17:22:34.744400 5917 mount_linux.go:219] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount /var/lib/kubelet/pods/ced5f698-8b9a-4f2c-ab2d-7896f9bcd4ed/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount)
I0323 17:22:34.745739 5917 mount_linux.go:219] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount /var/lib/kubelet/pods/ced5f698-8b9a-4f2c-ab2d-7896f9bcd4ed/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount)
I0323 17:22:34.746746 5917 nodeserver.go:138] NodePublishVolume: volume rg-backups-prod#stpabackupsprod#backups-prod mount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount at /var/lib/kubelet/pods/ced5f698-8b9a-4f2c-ab2d-7896f9bcd4ed/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount successfully
I0323 17:22:34.746763 5917 utils.go:82] GRPC response: {}
I0323 17:22:36.253015 5917 utils.go:75] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0323 17:22:36.253073 5917 utils.go:76] GRPC request: {"target_path":"/var/lib/kubelet/pods/e25f0f5f-1027-4dc3-bc25-bb906fdee5e5/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0323 17:22:36.253134 5917 nodeserver.go:203] NodeUnpublishVolume: unmounting volume rg-backups-prod#stpabackupsprod#backups-prod on /var/lib/kubelet/pods/e25f0f5f-1027-4dc3-bc25-bb906fdee5e5/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0323 17:22:36.253151 5917 mount_helper_common.go:93] unmounting "/var/lib/kubelet/pods/e25f0f5f-1027-4dc3-bc25-bb906fdee5e5/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount" (corruptedMount: false, mounterCanSkipMountPointChecks: true)
I0323 17:22:36.253161 5917 mount_linux.go:361] Unmounting /var/lib/kubelet/pods/e25f0f5f-1027-4dc3-bc25-bb906fdee5e5/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0323 17:22:36.255379 5917 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/pods/e25f0f5f-1027-4dc3-bc25-bb906fdee5e5/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount"
I0323 17:22:36.255442 5917 nodeserver.go:208] NodeUnpublishVolume: unmount volume rg-backups-prod#stpabackupsprod#backups-prod on /var/lib/kubelet/pods/e25f0f5f-1027-4dc3-bc25-bb906fdee5e5/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount successfully
I0323 17:22:36.255453 5917 utils.go:82] GRPC response: {}
I0323 17:23:23.927984 5917 utils.go:75] GRPC call: /csi.v1.Node/NodePublishVolume
I0323 17:23:23.928001 5917 utils.go:76] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount","target_path":"/var/lib/kubelet/pods/43e2afd8-7973-4fc4-9136-17855d409f9b/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["-o allow_other","--file-cache-timeout-in-seconds=120","--use-attr-cache=true","--cancel-list-on-mount-seconds=5","-o attr_timeout=120","-o entry_timeout=120","-o negative_timeout=120","--log-level=LOG_DEBUG","--cache-size-mb=1000","--virtual-directory=true"]}},"access_mode":{"mode":3}},"volume_context":{"containerName":"backups-prod","csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"orientdbservice-58c79ffb79-n9zqv","csi.storage.k8s.io/pod.namespace":"pa-data-services","csi.storage.k8s.io/pod.uid":"43e2afd8-7973-4fc4-9136-17855d409f9b","csi.storage.k8s.io/serviceAccount.name":"default","protocol":"fuse2","resourceGroup":"rg-backups-prod","storageAccount":"stpabackupsprod"},"volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0323 17:23:23.928461 5917 nodeserver.go:122] NodePublishVolume: volume rg-backups-prod#stpabackupsprod#backups-prod mounting /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount at /var/lib/kubelet/pods/43e2afd8-7973-4fc4-9136-17855d409f9b/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount with mountOptions: [bind]
I0323 17:23:23.928486 5917 mount_linux.go:219] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount /var/lib/kubelet/pods/43e2afd8-7973-4fc4-9136-17855d409f9b/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount)
I0323 17:23:23.929710 5917 mount_linux.go:219] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount /var/lib/kubelet/pods/43e2afd8-7973-4fc4-9136-17855d409f9b/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount)
I0323 17:23:23.930783 5917 nodeserver.go:138] NodePublishVolume: volume rg-backups-prod#stpabackupsprod#backups-prod mount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount at /var/lib/kubelet/pods/43e2afd8-7973-4fc4-9136-17855d409f9b/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount successfully
I0323 17:23:23.930803 5917 utils.go:82] GRPC response: {}
I0323 17:23:25.134756 5917 utils.go:75] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0323 17:23:25.135162 5917 utils.go:76] GRPC request: {"target_path":"/var/lib/kubelet/pods/ced5f698-8b9a-4f2c-ab2d-7896f9bcd4ed/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0323 17:23:25.135288 5917 nodeserver.go:203] NodeUnpublishVolume: unmounting volume rg-backups-prod#stpabackupsprod#backups-prod on /var/lib/kubelet/pods/ced5f698-8b9a-4f2c-ab2d-7896f9bcd4ed/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0323 17:23:25.135316 5917 mount_helper_common.go:93] unmounting "/var/lib/kubelet/pods/ced5f698-8b9a-4f2c-ab2d-7896f9bcd4ed/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount" (corruptedMount: false, mounterCanSkipMountPointChecks: true)
I0323 17:23:25.135335 5917 mount_linux.go:361] Unmounting /var/lib/kubelet/pods/ced5f698-8b9a-4f2c-ab2d-7896f9bcd4ed/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0323 17:23:25.137375 5917 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/pods/ced5f698-8b9a-4f2c-ab2d-7896f9bcd4ed/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount"
I0323 17:23:25.137468 5917 nodeserver.go:208] NodeUnpublishVolume: unmount volume rg-backups-prod#stpabackupsprod#backups-prod on /var/lib/kubelet/pods/ced5f698-8b9a-4f2c-ab2d-7896f9bcd4ed/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount successfully
I0323 17:23:25.137480 5917 utils.go:82] GRPC response: {}
I0323 17:24:18.535154 5917 utils.go:75] GRPC call: /csi.v1.Node/NodePublishVolume
I0323 17:24:18.535166 5917 utils.go:76] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount","target_path":"/var/lib/kubelet/pods/a406cc56-4df3-432c-ba71-7b95f0786597/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["-o allow_other","--file-cache-timeout-in-seconds=120","--use-attr-cache=true","--cancel-list-on-mount-seconds=5","-o attr_timeout=120","-o entry_timeout=120","-o negative_timeout=120","--log-level=LOG_DEBUG","--cache-size-mb=1000","--virtual-directory=true"]}},"access_mode":{"mode":3}},"volume_context":{"containerName":"backups-prod","csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"orientdbservice-58c79ffb79-qnhzt","csi.storage.k8s.io/pod.namespace":"pa-data-services","csi.storage.k8s.io/pod.uid":"a406cc56-4df3-432c-ba71-7b95f0786597","csi.storage.k8s.io/serviceAccount.name":"default","protocol":"fuse2","resourceGroup":"rg-backups-prod","storageAccount":"stpabackupsprod"},"volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0323 17:24:18.535564 5917 nodeserver.go:122] NodePublishVolume: volume rg-backups-prod#stpabackupsprod#backups-prod mounting /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount at /var/lib/kubelet/pods/a406cc56-4df3-432c-ba71-7b95f0786597/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount with mountOptions: [bind]
I0323 17:24:18.535579 5917 mount_linux.go:219] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount /var/lib/kubelet/pods/a406cc56-4df3-432c-ba71-7b95f0786597/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount)
I0323 17:24:18.536922 5917 mount_linux.go:219] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount /var/lib/kubelet/pods/a406cc56-4df3-432c-ba71-7b95f0786597/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount)
I0323 17:24:18.537909 5917 nodeserver.go:138] NodePublishVolume: volume rg-backups-prod#stpabackupsprod#backups-prod mount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount at /var/lib/kubelet/pods/a406cc56-4df3-432c-ba71-7b95f0786597/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount successfully
I0323 17:24:18.537921 5917 utils.go:82] GRPC response: {}
I0323 17:24:19.741693 5917 utils.go:75] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0323 17:24:19.741710 5917 utils.go:76] GRPC request: {"target_path":"/var/lib/kubelet/pods/43e2afd8-7973-4fc4-9136-17855d409f9b/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0323 17:24:19.741755 5917 nodeserver.go:203] NodeUnpublishVolume: unmounting volume rg-backups-prod#stpabackupsprod#backups-prod on /var/lib/kubelet/pods/43e2afd8-7973-4fc4-9136-17855d409f9b/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0323 17:24:19.741773 5917 mount_helper_common.go:93] unmounting "/var/lib/kubelet/pods/43e2afd8-7973-4fc4-9136-17855d409f9b/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount" (corruptedMount: false, mounterCanSkipMountPointChecks: true)
I0323 17:24:19.741783 5917 mount_linux.go:361] Unmounting /var/lib/kubelet/pods/43e2afd8-7973-4fc4-9136-17855d409f9b/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0323 17:24:19.774871 5917 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/pods/43e2afd8-7973-4fc4-9136-17855d409f9b/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount"
I0323 17:24:19.774964 5917 nodeserver.go:208] NodeUnpublishVolume: unmount volume rg-backups-prod#stpabackupsprod#backups-prod on /var/lib/kubelet/pods/43e2afd8-7973-4fc4-9136-17855d409f9b/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount successfully
I0323 17:24:19.774975 5917 utils.go:82] GRPC response: {}
I0323 17:25:41.801888 5917 utils.go:75] GRPC call: /csi.v1.Node/NodePublishVolume
I0323 17:25:41.801901 5917 utils.go:76] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount","target_path":"/var/lib/kubelet/pods/34a079a8-9e45-40b8-a38b-47cf2cd673fb/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["-o allow_other","--file-cache-timeout-in-seconds=120","--use-attr-cache=true","--cancel-list-on-mount-seconds=5","-o attr_timeout=120","-o entry_timeout=120","-o negative_timeout=120","--log-level=LOG_DEBUG","--cache-size-mb=1000","--virtual-directory=true"]}},"access_mode":{"mode":3}},"volume_context":{"containerName":"backups-prod","csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"orientdbservice-58c79ffb79-cbssg","csi.storage.k8s.io/pod.namespace":"pa-data-services","csi.storage.k8s.io/pod.uid":"34a079a8-9e45-40b8-a38b-47cf2cd673fb","csi.storage.k8s.io/serviceAccount.name":"default","protocol":"fuse2","resourceGroup":"rg-backups-prod","storageAccount":"stpabackupsprod"},"volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0323 17:25:41.802281 5917 nodeserver.go:122] NodePublishVolume: volume rg-backups-prod#stpabackupsprod#backups-prod mounting /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount at /var/lib/kubelet/pods/34a079a8-9e45-40b8-a38b-47cf2cd673fb/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount with mountOptions: [bind]
I0323 17:25:41.802297 5917 mount_linux.go:219] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount /var/lib/kubelet/pods/34a079a8-9e45-40b8-a38b-47cf2cd673fb/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount)
I0323 17:25:41.803521 5917 mount_linux.go:219] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount /var/lib/kubelet/pods/34a079a8-9e45-40b8-a38b-47cf2cd673fb/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount)
I0323 17:25:41.804503 5917 nodeserver.go:138] NodePublishVolume: volume rg-backups-prod#stpabackupsprod#backups-prod mount /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/a827afe57ea9da516011d9c85fceef41165a4bda0ace1d794dbb683b93ddd556/globalmount at /var/lib/kubelet/pods/34a079a8-9e45-40b8-a38b-47cf2cd673fb/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount successfully
I0323 17:25:41.804515 5917 utils.go:82] GRPC response: {}
I0323 17:25:43.107920 5917 utils.go:75] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0323 17:25:43.108075 5917 utils.go:76] GRPC request: {"target_path":"/var/lib/kubelet/pods/a406cc56-4df3-432c-ba71-7b95f0786597/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount","volume_id":"rg-backups-prod#stpabackupsprod#backups-prod"}
I0323 17:25:43.108167 5917 nodeserver.go:203] NodeUnpublishVolume: unmounting volume rg-backups-prod#stpabackupsprod#backups-prod on /var/lib/kubelet/pods/a406cc56-4df3-432c-ba71-7b95f0786597/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0323 17:25:43.108216 5917 mount_helper_common.go:93] unmounting "/var/lib/kubelet/pods/a406cc56-4df3-432c-ba71-7b95f0786597/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount" (corruptedMount: false, mounterCanSkipMountPointChecks: true)
I0323 17:25:43.108247 5917 mount_linux.go:361] Unmounting /var/lib/kubelet/pods/a406cc56-4df3-432c-ba71-7b95f0786597/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount
I0323 17:25:43.114662 5917 mount_helper_common.go:150] Warning: deleting path "/var/lib/kubelet/pods/a406cc56-4df3-432c-ba71-7b95f0786597/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount"
I0323 17:25:43.114755 5917 nodeserver.go:208] NodeUnpublishVolume: unmount volume rg-backups-prod#stpabackupsprod#backups-prod on /var/lib/kubelet/pods/a406cc56-4df3-432c-ba71-7b95f0786597/volumes/kubernetes.io~csi/pv-stpabackupsprod-blob-container/mount successfully
I0323 17:25:43.114769 5917 utils.go:82] GRPC response: {} |
|
Great !!! |
|
Thank you for all your help. |
|
Hi. Our team is experiencing the same issue with blobfuse2 on AKS 1.24.10. We have enabled managed blob-csi driver and its version is v1.19.2. The following error is produced during unmount: From the discussion above I understood that the fix landed in v1.19.2 and deployed to managed blob-csi drivers with the 0319 release (which is deployed to our location |
@kvet this could be different issue, during the unmount, you application would receive SIGTERM signal, and then application should stop writing data to blobfuse volume first, and then unmount would succeed. You could set preStop hook to terminiate the application when it receives SIGTERM signal from kubernetes |
|
@kvet to make sure whether your blobfuse2 mount is affected since the bug is in v1.19.1, and if your blobfuse2 mount was using v1.19.1 (happened before v1.19.2 rollout), then it may have issue. I think you could schedule your pods with blobfuse2 volume to new node or node without any blobfuse2 volume mount, that could make sure the original blobfuse2 mount is always using v1.19.2, hope that answers your question. |
|
@andyzhangx I don't think it's related to the writing as I am mounting the PV with read-only access... But even in the case of an ongoing write to the volume, it still makes sense that the driver manages to properly handle it. With blobfuse1 we are not experiencing similar issues. About what is exactly happening. On a fresh node, the driver allows mounting the volume once. I can list connected drives with Logs from the first mount. During the period the volume was mounted, there was access to the storage container and `df -h` returned a `blobfuse2` mount.Logs from the second mount. Notice that they are almost the same, but the volume wasn't properly mounted to the container.blobfuse2 logs |
|
@kvet the blobfuse2 mount took 18s to mount complete, not sure why it took so long time, is your agent node very busy? that means the original We have published a new v1.19.3 release that could fix this issue ultimately, pls share me with your aks cluster fqdn via mail (you could find my mail address from github), and I could upgrade the version to v1.19.3 from backend directly. |
|
@andyzhangx the cluster received the update. Thank you! I run into a different issue... The driver fails to mount a volume. Even on the first mount. (I run it on a node that no longer had a mounted blob volume and received the update as well as on a node that didn't have a mounted volume before.) The results are the same. The node itself is not busy. There are a number of running pods, but the resource usage is under 15% for CPU and 30% for RAM. Logs |
@kvet I could imagine where the problem is, in v1.19.3, we set 3s as mount timeout, usually 3s is enough, while in your case, it costs 18s for blobfuse2 mount. |
4 tasks
|
If this is new blobfuse version, then the fix to wait for the child mount to complete is embedded and now onwards we may see little more delay in mount completion. However, it shall be few seconds only and not in 10s. For any higher timeouts we shall look at the logs. We try to validate the connection with storage account if the response from server is slow for some reason, then mount will take that much time to complete as well. |
|
@andyzhangx I found something suspicious in the blobfuse2 logs. Attaching them... blobfuse2 logs |
|
Is allow_other enabled here ? |
|
@vibhansa-msft I am mounting the ABS container with the following mount options: Here is the content of `/etc/fuse.conf` on the node from which I sent the logsI've also checked the node where the current blobfuse1 deployment is running (almost the same options but Moreover, with the csi driver 1.19.2 on a fresh node (the one I am sending you the contents of Can you please clarify what exact information you need me to provide in order to mitigate the issue? We are eager to use blobfuse2 as it performs way better in our use case. |
|
We will need blobfuse logs to understand what took time during the mount operation. If you can enable log_debug and retry that might be useful. |
|
When I started the new testing I noticed the new driver version had been deployed to our cluster. For which huge thank you, guys! And during my testing, I haven't experienced any issue with mounting and I can classify the original request as resolved. Hope it will work reliably. Here are the logs from the new driver version and from a fresh node (to which no one blobfuse2 mount was done before). Hope it will help you to find out the root cause. csi logsblobfuse2 logs |
|
@kvet great to hear that. |
|
Blobfuse logs shared above looks clean to me. |
|
@kvet csi driver logs shows it's costing 27s to mount complete, and in v1.19.4, we set mount path timeout as 1min, that's why it works now: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Which version of blobfuse was used?
blobfuse2 version 2.0.2
Which OS distribution and version are you using?
AKS kernel version 1.23.12
Docker: FROM python:3.10.4
root@mydocker:/home# cat /proc/version
Linux version 5.4.0-1091-azure (buildd@lcy02-amd64-023) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1
18.04)) #9618.04.1-Ubuntu SMP Tue Aug 30 19:15:32 UTC 2022root@mydocker:/home# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
If relevant, please share your mount command.
install blobfuse2
wget https://packages.microsoft.com/config/debian/11/packages-microsoft-prod.deb
dpkg -i packages-microsoft-prod.deb
apt-get update
apt-get install libfuse3-dev fuse3 -y
apt-get install blobfuse2
mount command
blobfuse2 mount /home/blobfmount --config-file=/home/blobfuse2/config.yaml --container-name=mycontainer --log-level=log_debug --log-file-path=./bobfuse2b.log
here is config.yaml below:
Refer ./setup/baseConfig.yaml for full set of config parameters
allow-other: true
logging:
type: syslog
level: log_debug
components:
libfuse:
attribute-expiration-sec: 120
entry-expiration-sec: 120
negative-entry-expiration-sec: 240
file_cache:
path: /home/blobfuse/tempcache
timeout-sec: 120
max-size-mb: 4096
attr_cache:
timeout-sec: 7200
azstorage:
type: block
account-name: ***
account-key: ***
endpoint:***
container: ***
What was the issue encountered?
After I ran the blobfuse2 mount command, I cannot check the files in the destination blob or create test file and check in blob.
Have you found a mitigation/solution?
Not yet.
Please share logs if available.
root@mydocker:/home/blobfmount# cat bobfuse2b.log
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_CRIT [mount.go (384)]: Starting Blobfuse2 Mount : 2.0.2 on [Debian GNU/Linux 11 (bullseye)]
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_CRIT [mount.go (385)]: Logging level set to : LOG_DEBUG
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_TRACE [libfuse.go (220)]: Libfuse::Configure : libfuse
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_INFO [libfuse.go (260)]: Libfuse::Configure : read-only false, allow-other true, default-perm 511, entry-timeout 120, attr-time 120, negative-timeout 240, ignore-open-flags: true, nonempty false
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_TRACE [file_cache.go (197)]: FileCache::Configure : file_cache
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_INFO [file_cache.go (272)]: FileCache::Configure : Using default eviction policy
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_INFO [file_cache.go (291)]: FileCache::Configure : create-empty false, cache-timeout 120, tmp-path /home/blobfuse/tempcache, max-size-mb 4096, high-mark 80, low-mark 60
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_TRACE [attr_cache.go (121)]: AttrCache::Configure : attr_cache
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_INFO [attr_cache.go (145)]: AttrCache::Configure : cache-timeout 7200, symlink false, cache-on-list true
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_TRACE [azstorage.go (83)]: AzStorage::Configure : azstorage
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_TRACE [config.go (270)]: ParseAndValidateConfig : Parsing config
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_INFO [config.go (372)]: ParseAndValidateConfig : using the following proxy address from the config file:
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_INFO [config.go (376)]: ParseAndValidateConfig : sdk logging from the config file: false
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_TRACE [config.go (479)]: ParseAndReadDynamicConfig : Reparsing config
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_DEBUG [config.go (383)]: ParseAndValidateConfig : Getting auth type
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_INFO [config.go (467)]: ParseAndValidateConfig : Account: mystorageaccount, Container: mycontainer, AccountType: BLOCK, Auth: KEY, Prefix: , Endpoint: https://mystorageaccount.blob.core.chinacloudapi.cn/, ListBlock: 0, MD5 : false false, Virtual Directory: true
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_INFO [config.go (471)]: ParseAndValidateConfig : Retry Config: Retry count 5, Max Timeout 900, BackOff Time 4, Max Delay 60
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_TRACE [block_blob.go (173)]: BlockBlob::SetupPipeline : Setting up
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_TRACE [block_blob.go (135)]: BlockBlob::getCredential : Getting credential
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_DEBUG [azauth.go (79)]: azAuth::getAzAuth : Account: mystorageaccount, AccountType: BLOCK, Protocol: https, Endpoint: https://mystorageaccount.blob.core.chinacloudapi.cn/
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_TRACE [block_blob.go (260)]: BlockBlob::SetPrefixPath : path
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_TRACE [block_blob.go (209)]: BlockBlob::TestPipeline : Validating
Wed Mar 8 06:31:11 UTC 2023 : blobfuse2[607] : LOG_INFO [mount.go (392)]: mount: Mounting blobfuse2 on /home/blobfmount
root@mydocker:/home/blobfmount# cd ..
I tried to do an unmount command and returned this:
root@mydocker:/home# blobfuse2 unmount all
Error: failed to list mount points [open /etc/mtab: no such file or directory]
root@mydocker:/home# blobfuse2 mount /home/blobfmount --config-file=/home/blobfuse2/config.yaml --container-name=mycontainer --log-level=log_debug --log-file-path=./bobfuse2b.log
Error: mount directory is not empty
root@mydocker:/home# rm -f ./blobfuse2/*
rm: cannot remove './blobfuse2/config.yaml': Device or resource busy
root@mydocker:/home#
The text was updated successfully, but these errors were encountered: