[ AKS / dysk / csi ] MountVolume.WaitForAttach failed

[saykumar]

I'm using AKS v1.10.8. I followed the instructions on the CSI -> Dysk page and am trying the Dynamic Provisioning Example.

I've applied the fix described workaround described here to the csi-dysk-{attacher|provisioner} stateful sets to reliably talk to the master (otherwise there were a bunch of stream errors): Azure/AKS#676

The nginx dynamic pvc example keeps failing with this:

MountVolume.WaitForAttach failed for volume "kubernetes-dynamic-pv-7fbf8d63cbea11e8" : Get https://aks123456mgmt-87ad19cb.hcp.eastus2.azmk8s.io:443/apis/storage.k8s.io/v1beta1/volumeattachments/csi-8df6028b81d46cf0bd5c263a489ea800a7dc6791ecb58fab53664876d37e8929: read tcp 10.240.0.5:46142->137.116.66.84:443: use of closed network connection

Any help would be much appreciated! Thanks!

[saykumar]

Update: I deleted and retried the test pod, and while the volume attachment succeeds, the pod never comes up, their status shows NodeLost and the output of kubectl get nodes shows the node on which mounting was attempted as NotReady.

Would you have some additional documentation for getting this to work for AKS?

[andyzhangx]

@saykumar can you ssh to that node and check kubelet status on that node? And also get kubelet logs on that node by:

sudo journalctl -u kubelet -l > kubelet.log

[saykumar]

@andyzhangx

I've pasted the kubelet logs from the 2 nodes in our cluster here. Since the kublet logs were pretty chatty, I filtered it to the 10 minute window when one of these nodes went into NotReady:

Wrong nodes - let me get the right ones. Sorry about that.

[saykumar]

Unfortunately, the node's SSH daemon isn't responding either. I can potentially reproduce this again and share those steps. Would that help?

[andyzhangx]

@saykumar this could be related to khenidak/dysk#57
@khenidak

[andyzhangx]

close this since dysk driver is deprecated