PersistentVolumeClaims fail to provision

[nicks]

Steps to reproduce the issue:

1. minikube start
2. kubectl apply -f kubectl apply -f https://download.elastic.co/downloads/eck/1.1.2/all-in-one.yaml
3. kubectl apply -f es.yaml

es.yaml:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.8.0
  nodeSets:
  - name: default
    count: 1
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 100M
        storageClassName: standard

4. kubectl get pvc

Expected result:
A provisioned volume

Actual result:

$ kubectl get pvc
NAME                                         STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
elasticsearch-data-quickstart-es-default-0   Pending                                      standard       24s

Full output of failed command:

N/A

Full output of minikube start command used, if not already included:

$ minikube start
😄 minikube v1.11.0 on Ubuntu 20.04
✨ Automatically selected the kvm2 driver
👍 Starting control plane node minikube in cluster minikube
🔥 Creating kvm2 VM (CPUs=2, Memory=3900MB, Disk=20000MB) ...
🐳 Preparing Kubernetes v1.18.3 on Docker 19.03.8 ...
🔎 Verifying Kubernetes components...
🌟 Enabled addons: default-storageclass, storage-provisioner
🏄 Done! kubectl is now configured to use "minikube"

Optional: Full output of minikube logs command:

This error from minikube logs seems suspicious:

E0707 18:22:59.414554 1 reflector.go:205] k8s.io/minikube/vendor/github.com/r2d4/external-storage/lib/controller/controller.go:411: Failed to list *v1.PersistentVolumeClaim: v1.PersistentVolumeClaimList: Items: []v1.PersistentVolumeClaim: v1.PersistentVolumeClaim: ObjectMeta: v1.ObjectMeta: readObjectFieldAsBytes: expect : after object field, parsing 2938 ...:{},"k:{"... at {"kind":"PersistentVolumeClaimList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/persistentvolumeclaims","resourceVersion":"1101"},"items":[{"metadata":{"name":"my-pvc","namespace":"default","selfLink":"/api/v1/namespaces/default/persistentvolumeclaims/my-pvc","uid":"fc02c10c-bc7b-491f-a074-505e2b1b7600","resourceVersion":"1101","creationTimestamp":"2020-07-07T18:20:17Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"my-pvc","namespace":"default"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"100M"}},"storageClassName":"standard","volumeMode":"Filesystem"}}\n","volume.beta.kubernetes.io/storage-provisioner":"k8s.io/minikube-hostpath"},"finalizers":["kubernetes.io/pvc-protection"],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2020-07-07T18:20:17Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:volume.beta.kubernetes.io/storage-provisioner":{}}}}},{"manager":"kubectl","operation":"Update","apiVersion":"v1","time":"2020-07-07T18:20:17Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}},"f:status":{"f:phase":{}}}}]},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"100M"}},"storageClassName":"standard","volumeMode":"Filesystem"},"status":{"phase":"Pending"}},{"metadata":{"name":"elasticsearch-data-quickstart-es-default-0","namespace":"default","selfLink":"/api/v1/namespaces/default/persistentvolumeclaims/elasticsearch-data-quickstart-es-default-0","uid":"7b55ee14-1699-491d-8ff1-a40d60f8d133","resourceVersion":"612","creationTimestamp":"2020-07-07T18:16:52Z","labels":{"common.k8s.elastic.co/type":"elasticsearch","elasticsearch.k8s.elastic.co/cluster-name":"quickstart","elasticsearch.k8s.elastic.co/statefulset-name":"quickstart-es-default"},"annotations":{"volume.beta.kubernetes.io/storage-provisioner":"k8s.io/minikube-hostpath"},"ownerReferences":[{"apiVersion":"elasticsearch.k8s.elastic.co/v1","kind":"Elasticsearch","name":"quickstart","uid":"08b36181-105c-4cda-9f2b-9856a7963e33","controller":true,"blockOwnerDeletion":false}],"finalizers":["kubernetes.io/pvc-protection"],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2020-07-07T18:16:52Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:volume.beta.kubernetes.io/storage-provisioner":{}},"f:labels":{".":{},"f:common.k8s.elastic.co/type":{},"f:elasticsearch.k8s.elastic.co/cluster-name":{},"f:elasticsearch.k8s.elastic.co/statefulset-name":{}},"f:ownerReferences":{".":{},"k:{"uid":"08b36181-105c-4cda-9f2b-9856a7963e33"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}},"f:status":{"f:phase":{}}}}]},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"100M"}},"storageClassName":"standard","volumeMode":"Filesystem"},"status":{"phase":"Pending"}}]}

It looks like the storage provisioner is failing to parse to persistent volume claim

[medyagh]

@nicks which driver do u sue ?

[nicks]

@medyagh kvm2

[priyawadhwa]

Hey @nicks thanks for providing such clear reproduction instructions, I was able to repro this issue.

We've been having issues with the storage provisioner for a while now and plan on fixing it soon. A dupe of this issue exists here: #8414 so I'm going to close this one so that we can consolidate the issues.

Please comment any new findings there, or feel free to reopen this issue if you think it's different.

[nicks]

@priyawadhwa that looks right to me, thanks!