mount failure in Azure File CSI migration #96508
Comments
k8s-ci-robot
added
priority/important-soon
|
cc @msau42 |
|
@andyzhangx can you paste what the in-tree PV object looks like vs CSI PV object? |
|
@kubernetes/sig-storage-bugs |
|
cc @mattcary |
|
current issue:
if PVC and pod are not in namespace
$ k get pv pvc-01bd1d0b-b965-4d37-8331-817a296c8998 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
kubernetes.io/azure-file-resource-group: andy-1191
pv.kubernetes.io/provisioned-by: file.csi.azure.com
spec:
accessModes:
- ReadWriteMany
azureFile:
secretName: azure-storage-account-f97cecd00f4d4433ebd6809-secret
secretNamespace: null
shareName: pvc-01bd1d0b-b965-4d37-8331-817a296c8998
capacity:
storage: 100Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: pvc-azurefile
namespace: test
persistentVolumeReclaimPolicy: Delete
storageClassName: azurefile-456-kubernetes.io-azure-file-dynamic-sc-4hv88
volumeMode: Filesystem
status:
phase: Bound
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-azurefile
namespace: test
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 100Gi
storageClassName: azurefile-456-kubernetes.io-azure-file-dynamic-sc-4hv88
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: nginx
name: deployment-azurefile
namespace: test
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
name: deployment-azurefile
spec:
nodeSelector:
"kubernetes.io/os": linux
containers:
- name: deployment-azurefile
image: mcr.microsoft.com/oss/nginx/nginx:1.17.3-alpine
command:
- "/bin/sh"
- "-c"
- while true; do echo $(date) >> /mnt/azurefile/outfile; sleep 1; done
volumeMounts:
- name: azurefile
mountPath: "/mnt/azurefile"
readOnly: false
volumes:
- name: azurefile
persistentVolumeClaim:
claimName: pvc-azurefile
strategy:
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
type: RollingUpdate |
|
I have tried multiple times, some symptoms:
|
|
Does it work if pvc and pod and secret are all in the same namespace (not default)? Looking at azure translation code, the inline to csi translation seems problematic, as it's hardcoding default namespace for the secret: The PV to csi translation function looks ok though: |
About above code, inline to csi translation code should be ok since there is no namespace field in inline sturct, so it's always in @msau42 I have verified that if pvc and pod and secret are in the same namespace (not default), it works. And I found one funny thing, only statefulset does not work, and deployment always works even pod and secret are not in same namespace, in below example, statefulset and deployment are using same PVC, only statefulset does not work. Not sure whether there is special handling in statefulset. apiVersion: apps/v1
kind: StatefulSet
metadata:
name: statefulset-azurefile
namespace: test
labels:
app: nginx
spec:
serviceName: statefulset-azurefile
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
nodeSelector:
"kubernetes.io/os": linux
containers:
- name: statefulset-azurefile
image: mcr.microsoft.com/oss/nginx/nginx:1.17.3-alpine
volumeMounts:
- name: azurefile
mountPath: /mnt/azurefile
volumes:
- name: azurefile
persistentVolumeClaim:
claimName: persistent-storage-statefulset-azurefile-0
selector:
matchLabels:
app: nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: nginx
name: deployment-azurefile
namespace: test
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
name: deployment-azurefile
spec:
nodeSelector:
"kubernetes.io/os": linux
containers:
- name: deployment-azurefile
image: mcr.microsoft.com/oss/nginx/nginx:1.17.3-alpine
volumeMounts:
- name: azurefile
mountPath: "/mnt/azurefile"
readOnly: false
volumes:
- name: azurefile
persistentVolumeClaim:
claimName: persistent-storage-statefulset-azurefile-0 |
That's really odd. The Node authorizer should only care about Pods, not the higher controllers. Do the pod specs look the same? |
|
In Azure file CSI migration scenario, the secret creation happens in CSI driver, kubelet does not set relationship between node and secret. PR(#97877) pass the There could be no straightforward fix(as I tried in last two months), #97877 would be a workaround fix, anyway, it would also fix this failure. |
What happened:
after turning on
CSIMigration=true,CSIMigrationAzureFile=trueon 1.19, Azure File CSI migration e2e test would failure, main error is like following:fetching NodeStageSecretRef default/azure-storage-account-f5ffebf7f32ce4953afa073-secret failed: kubernetes.io/csi: failed to find the secret azure-storage-account-f5ffebf7f32ce4953afa073-secret in the namespace default with error: secrets "azure-storage-account-f5ffebf7f32ce4953afa073-secret" is forbidden: User "system:node:k8s-agentpool1-29483490-0" cannot get resource "secrets" in API group "" in the namespace "default": no relationship found between node 'k8s-agentpool1-29483490-0' and this objecthttps://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_azurefile-csi-driver/430/pull-azurefile-csi-driver-e2e-migration/1324965466602475520/build-log.txt
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:
kubectl version): v1.19.0cat /etc/os-release): Ubuntu 16.04uname -a):/priority important-soon
/sig cloud-provider
/area provider/azure
/triage accepted
/sig storage
The text was updated successfully, but these errors were encountered: