MountVolume.SetUp failed for volume "test" : invalid character '/' looking for beginning of value #14

asjddd · 2019-03-01T00:19:37Z

What happened:
A mount that had worked just days before failed to mount.

To investigate we simplified to the test example on https://github.com/Azure/kubernetes-volume-drivers/tree/master/flexvolume/blobfuse

This exact test example has worked previously just days ago (clean copy-paste from our docs right to the machine, no chance of human error). This morning it broke.

We tried to mount a test container using the below secret and yaml.

We investigated using:
kubectl describe pod nginx-flex-blobfuse

We found this error message

[3:59 PM] Prajwal Muralidhara
 Warning  FailedMount            6s (x7 over 38s)  kubelet, k8s-agentpool1-35095709-5  MountVolume.SetUp failed for volume "test" : invalid character '/' looking for beginning of value

Daemonset start:
kubectl create -f https://raw.githubusercontent.com/Azure/kubernetes-volume-drivers/master/flexvolume/blobfuse/deployment/blobfuse-flexvol-installer-1.8.yaml

Secret setup:
kubectl create secret generic blobfusecreds --from-literal accountname=test4anjohansen47msft --from-literal accountkey="v0+dpbq9kpd5oOvab7s9/ywe6fhzMzHrxQPgu7BTm+I8e7AUWRlHCtU0RCdqdyvtyFWe9+PpN/oZIFsW/Xf3GA==" --type="azure/blobfuse"

YAML for the pod:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-flex-blobfuse
spec:
  containers:
  - name: nginx-flex-blobfuse
    image: nginx
    volumeMounts:
    - name: test
      mountPath: /data
  volumes:
  - name: test
    flexVolume:
      driver: "azure/blobfuse"
      readOnly: false
      secretRef:
        name: blobfusecreds
      options:
        container: test-container
        tmppath: /tmp/blobfuse
        mountoptions: "--file-cache-timeout-in-seconds=120 --use-https=true"

What you expected to happen:

How to reproduce it:

Anything else we need to know?:
Here's the /var/log/blobfuse-driver.log log from the node:

Enable succeeded: 
[stdout]
d3a06ebd3/volumes/azure~blobfuse/builderartifactstorage --container-name=b7dlfku7s518 --tmp-path=/tmp/blobfuse -o allow_other -o ro --file-cache-timeout-in-seconds=120 --use-https=true
Thu Feb 28 22:55:37 UTC 2019 ERROR: { "status": "Failure", "message": "Failed to mount device /dev/ at /var/lib/kubelet/pods/8ec354c7-3ba4-11e9-b035-000d3a06ebd3/volumes/azure~blobfuse/builderartifactstorage, accountname:builderartifactstorage, error log:Thu Feb 28 22:55:37 UTC 2019 EXEC: /usr/bin/blobfuse /var/lib/kubelet/pods/8ec354c7-3ba4-11e9-b035-000d3a06ebd3/volumes/azure~blobfuse/builderartifactstorage --container-name=b7dlfku7s518 --tmp-path=/tmp/blobfuse -o allow_other -o ro --file-cache-timeout-in-seconds=120 --use-https=true" }
Thu Feb 28 22:57:39 UTC 2019 EXEC: mkdir -p /var/lib/kubelet/pods/8ec354c7-3ba4-11e9-b035-000d3a06ebd3/volumes/azure~blobfuse/builderartifactstorage
Thu Feb 28 22:57:39 UTC 2019 INF: export storage account - export AZURE_STORAGE_ACCOUNT=builderartifactstorage AZURE_STORAGE_ACCESS_KEY=... 
Thu Feb 28 22:57:39 UTC 2019 EXEC: /usr/bin/blobfuse /var/lib/kubelet/pods/8ec354c7-3ba4-11e9-b035-000d3a06ebd3/volumes/azure~blobfuse/builderartifactstorage --container-name=b7dlfku7s518 --tmp-path=/tmp/blobfuse -o allow_other -o ro --file-cache-timeout-in-seconds=120 --use-https=true
Thu Feb 28 22:57:39 UTC 2019 ERROR: { "status": "Failure", "message": "Failed to mount device /dev/ at /var/lib/kubelet/pods/8ec354c7-3ba4-11e9-b035-000d3a06ebd3/volumes/azure~blobfuse/builderartifactstorage, accountname:builderartifactstorage, error log:Thu Feb 28 22:57:39 UTC 2019 EXEC: /usr/bin/blobfuse /var/lib/kubelet/pods/8ec354c7-3ba4-11e9-b035-000d3a06ebd3/volumes/azure~blobfuse/builderartifactstorage --container-name=b7dlfku7s518 --tmp-path=/tmp/blobfuse -o allow_other -o ro --file-cache-timeout-in-seconds=120 --use-https=true" }
Thu Feb 28 22:59:41 UTC 2019 EXEC: mkdir -p /var/lib/kubelet/pods/8ec354c7-3ba4-11e9-b035-000d3a06ebd3/volumes/azure~blobfuse/builderartifactstorage
Thu Feb 28 22:59:41 UTC 2019 INF: export storage account - export AZURE_STORAGE_ACCOUNT=builderartifactstorage AZURE_STORAGE_ACCESS_KEY=... 
Thu Feb 28 22:59:41 UTC 2019 EXEC: /usr/bin/blobfuse /var/lib/kubelet/pods/8ec354c7-3ba4-11e9-b035-000d3a06ebd3/volumes/azure~blobfuse/builderartifactstorage --container-name=b7dlfku7s518 --tmp-path=/tmp/blobfuse -o allow_other -o ro --file-cache-timeout-in-seconds=120 --use-https=true
Thu Feb 28 22:59:41 UTC 2019 ERROR: { "status": "Failure", "message": "Failed to mount device /dev/ at /var/lib/kubelet/pods/8ec354c7-3ba4-11e9-b035-000d3a06ebd3/volumes/azure~blobfuse/builderartifactstorage, accountname:builderartifactstorage, error log:Thu Feb 28 22:59:41 UTC 2019 EXEC: /usr/bin/blobfuse /var/lib/kubelet/pods/8ec354c7-3ba4-11e9-b035-000d3a06ebd3/volumes/azure~blobfuse/builderartifactstorage --container-name=b7dlfku7s518 --tmp-path=/tmp/blobfuse -o allow_other -o ro --file-cache-timeout-in-seconds=120 --use-https=true" }
Thu Feb 28 23:01:43 UTC 2019 EXEC: mkdir -p /var/lib/kubelet/pods/8ec354c7-3ba4-11e9-b035-000d3a06ebd3/volumes/azure~blobfuse/builderartifactstorage
Thu Feb 28 23:01:43 UTC 2019 INF: export storage account - export AZURE_STORAGE_ACCOUNT=builderartifactstorage AZURE_STORAGE_ACCESS_KEY=... 
Thu Feb 28 23:01:43 UTC 2019 EXEC: /usr/bin/blobfuse /var/lib/kubelet/pods/8ec354c7-3ba4-11e9-b035-000d3a06ebd3/volumes/azure~blobfuse/builderartifactstorage --container-name=b7dlfku7s518 --tmp-path=/tmp/blobfuse -o allow_other -o ro --file-cache-timeout-in-seconds=120 --use-https=true
Thu Feb 28 23:01:43 UTC 2019 ERROR: { "status": "Failure", "message": "Failed to mount device /dev/ at /var/lib/kubelet/pods/8ec354c7-3ba4-11e9-b035-000d3a06ebd3/volumes/azure~blobfuse/builderartifactstorage, accountname:builderartifactstorage, error log:Thu Feb 28 23:01:43 UTC 2019 EXEC: /usr/bin/blobfuse /var/lib/kubelet/pods/8ec354c7-3ba4-11e9-b035-000d3a06ebd3/volumes/azure~blobfuse/builderartifactstorage --container-name=b7dlfku7s518 --tmp-path=/tmp/blobfuse -o allow_other -o ro --file-cache-timeout-in-seconds=120 --use-https=true" }

[stderr]

Log from the pod:

Enable succeeded: 
[stdout]
6fe3818b-3bad-11e9-9f23-000d3a06e777/volumes/azure~blobfuse/test
Thu Feb 28 23:55:07 UTC 2019 INF: export storage account - export AZURE_STORAGE_ACCOUNT=test4anjohansen47msft AZURE_STORAGE_ACCESS_KEY=... 
Thu Feb 28 23:55:07 UTC 2019 EXEC: /usr/bin/blobfuse /var/lib/kubelet/pods/6fe3818b-3bad-11e9-9f23-000d3a06e777/volumes/azure~blobfuse/test --container-name=test-container --tmp-path=/tmp/blobfuse -o allow_other  --file-cache-timeout-in-seconds=120 --use-https=true
Thu Feb 28 23:55:07 UTC 2019 ERROR: { "status": "Failure", "message": "Failed to mount device /dev/ at /var/lib/kubelet/pods/6fe3818b-3bad-11e9-9f23-000d3a06e777/volumes/azure~blobfuse/test, accountname:test4anjohansen47msft, error log:Thu Feb 28 23:55:07 UTC 2019 EXEC: /usr/bin/blobfuse /var/lib/kubelet/pods/6fe3818b-3bad-11e9-9f23-000d3a06e777/volumes/azure~blobfuse/test --container-name=test-container --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120 --use-https=true" }
Thu Feb 28 23:57:09 UTC 2019 EXEC: mkdir -p /var/lib/kubelet/pods/6fe3818b-3bad-11e9-9f23-000d3a06e777/volumes/azure~blobfuse/test
Thu Feb 28 23:57:09 UTC 2019 INF: export storage account - export AZURE_STORAGE_ACCOUNT=test4anjohansen47msft AZURE_STORAGE_ACCESS_KEY=... 
Thu Feb 28 23:57:09 UTC 2019 EXEC: /usr/bin/blobfuse /var/lib/kubelet/pods/6fe3818b-3bad-11e9-9f23-000d3a06e777/volumes/azure~blobfuse/test --container-name=test-container --tmp-path=/tmp/blobfuse -o allow_other  --file-cache-timeout-in-seconds=120 --use-https=true
Thu Feb 28 23:57:09 UTC 2019 ERROR: { "status": "Failure", "message": "Failed to mount device /dev/ at /var/lib/kubelet/pods/6fe3818b-3bad-11e9-9f23-000d3a06e777/volumes/azure~blobfuse/test, accountname:test4anjohansen47msft, error log:Thu Feb 28 23:57:09 UTC 2019 EXEC: /usr/bin/blobfuse /var/lib/kubelet/pods/6fe3818b-3bad-11e9-9f23-000d3a06e777/volumes/azure~blobfuse/test --container-name=test-container --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120 --use-https=true" }
Thu Feb 28 23:59:11 UTC 2019 EXEC: mkdir -p /var/lib/kubelet/pods/6fe3818b-3bad-11e9-9f23-000d3a06e777/volumes/azure~blobfuse/test
Thu Feb 28 23:59:11 UTC 2019 INF: export storage account - export AZURE_STORAGE_ACCOUNT=test4anjohansen47msft AZURE_STORAGE_ACCESS_KEY=... 
Thu Feb 28 23:59:11 UTC 2019 EXEC: /usr/bin/blobfuse /var/lib/kubelet/pods/6fe3818b-3bad-11e9-9f23-000d3a06e777/volumes/azure~blobfuse/test --container-name=test-container --tmp-path=/tmp/blobfuse -o allow_other  --file-cache-timeout-in-seconds=120 --use-https=true
Thu Feb 28 23:59:11 UTC 2019 ERROR: { "status": "Failure", "message": "Failed to mount device /dev/ at /var/lib/kubelet/pods/6fe3818b-3bad-11e9-9f23-000d3a06e777/volumes/azure~blobfuse/test, accountname:test4anjohansen47msft, error log:Thu Feb 28 23:59:11 UTC 2019 EXEC: /usr/bin/blobfuse /var/lib/kubelet/pods/6fe3818b-3bad-11e9-9f23-000d3a06e777/volumes/azure~blobfuse/test --container-name=test-container --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120 --use-https=true" }
Fri Mar  1 00:01:13 UTC 2019 EXEC: mkdir -p /var/lib/kubelet/pods/6fe3818b-3bad-11e9-9f23-000d3a06e777/volumes/azure~blobfuse/test
Fri Mar  1 00:01:13 UTC 2019 INF: export storage account - export AZURE_STORAGE_ACCOUNT=test4anjohansen47msft AZURE_STORAGE_ACCESS_KEY=... 
Fri Mar  1 00:01:13 UTC 2019 EXEC: /usr/bin/blobfuse /var/lib/kubelet/pods/6fe3818b-3bad-11e9-9f23-000d3a06e777/volumes/azure~blobfuse/test --container-name=test-container --tmp-path=/tmp/blobfuse -o allow_other  --file-cache-timeout-in-seconds=120 --use-https=true
Fri Mar 1 00:01:13 UTC 2019 ERROR: { "status": "Failure", "message": "Failed to mount device /dev/ at /var/lib/kubelet/pods/6fe3818b-3bad-11e9-9f23-000d3a06e777/volumes/azure~blobfuse/test, accountname:test4anjohansen47msft, error log:Fri Mar 1 00:01:13 UTC 2019 EXEC: /usr/bin/blobfuse /var/lib/kubelet/pods/6fe3818b-3bad-11e9-9f23-000d3a06e777/volumes/azure~blobfuse/test --container-name=test-container --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120 --use-https=true" }

[stderr]

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.11", GitCommit:"637c7e288581ee40ab4ca210618a89a555b6e7e9", GitTreeState:"clean", BuildDate:"2018-11-26T14:38:32Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.7", GitCommit:"b30876a5539f09684ff9fde266fda10b37738c9c", GitTreeState:"clean", BuildDate:"2018-01-16T21:52:38Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.4 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

Kernel (e.g. uname -a):

Enable succeeded: 
Linux k8s-agentpool1-35095709-5 4.15.0-1039-azure #43-Ubuntu SMP Fri Feb 15 17:43:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

The text was updated successfully, but these errors were encountered:

andyzhangx · 2019-03-01T03:00:23Z

hi, could you try this locally by below commands first, seems there is sth wrong with the account key:

mkdir /tmp/abc
export AZURE_STORAGE_ACCOUNT=builderartifactstorage 
export AZURE_STORAGE_ACCESS_KEY="v0+dpbq9kpd5oOvab7s9/ywe6fhzMzHrxQPgu7BTm+I8e7AUWRlHCtU0RCdqdyvtyFWe9+PpN/oZIFsW/Xf3GA=="

# /usr/bin/blobfuse /tmp/abc --container-name=b7dlfku7s518 --tmp-path=/tmp/blobfuse -o allow_other -o ro --file-cache-timeout-in-seconds=120 --use-https=true
Failed to connect to the storage container. There might be something wrong about the storage config, please double check the storage account name, account key and container name. errno = 403

asjddd · 2019-03-04T21:40:51Z

Well here's the thing...

The exact same setup - secrets, container etc. - still works perfectly on another cluster with a slightly older version of the BlobFuse driver. So we're pretty sure it must be a regression that has crept in since then.

What's the best way to help determine exactly which version of the driver that is working for us, so we can zero in on the potentially damaging changes in the codebase?

Here's the logs from a (successful) startup on that cluster for reference.

root@k8s-agentpool1-37198164-vmss00000G:/# cat /var/log/blobfuse-driver.log
Tue Feb 12 17:25:43 UTC 2019 INFO: {"status": "Success", "capabilities": {"attach": false}}
Fri Feb 22 07:01:52 UTC 2019 INFO: {"status": "Success", "capabilities": {"attach": false}}
Fri Feb 22 07:02:40 UTC 2019 INFO: {"status": "Success", "capabilities": {"attach": false}}
Fri Mar 1 07:20:52 UTC 2019 INFO: {"status": "Success", "capabilities": {"attach": false}}
Fri Mar 1 07:21:36 UTC 2019 INFO: {"status": "Success", "capabilities": {"attach": false}}
Mon Mar  4 19:24:03 UTC 2019 EXEC: mkdir -p /var/lib/kubelet/pods/0e72025f-3eb3-11e9-bac1-000d3af9eacf/volumes/azure~blobfuse/builderartifactstorage
Mon Mar  4 19:24:03 UTC 2019 INF: export storage account - export AZURE_STORAGE_ACCOUNT=builderartifactstorage AZURE_STORAGE_ACCESS_KEY=...
Mon Mar  4 19:24:03 UTC 2019 EXEC: /usr/bin/blobfuse /var/lib/kubelet/pods/0e72025f-3eb3-11e9-bac1-000d3af9eacf/volumes/azure~blobfuse/builderartifactstorage --container-name=2u0i5bds8zyzn --tmp-path=/tmp/blobfuse -o allow_other -o ro --file-cache-timeout-in-seconds=120 --use-https=true
Mon Mar 4 19:24:04 UTC 2019 INFO: {"status": "Success"}

andyzhangx · 2019-03-05T03:52:05Z

Here is the way to get the version of driver:

kubectl get po -n kube-system | grep blobfuse
kubectl describe po blobfuse-flexvol-installer-c6f7d -n kube-system | grep blobfuse-flexvolume

the latest k8s verison is 1.0.8

and could you also try your account key in a Linux VM, it does not work now:

mkdir /tmp/abc
export AZURE_STORAGE_ACCOUNT=builderartifactstorage 
export AZURE_STORAGE_ACCESS_KEY="v0+dpbq9kpd5oOvab7s9/ywe6fhzMzHrxQPgu7BTm+I8e7AUWRlHCtU0RCdqdyvtyFWe9+PpN/oZIFsW/Xf3GA=="

# /usr/bin/blobfuse /tmp/abc --container-name=b7dlfku7s518 --tmp-path=/tmp/blobfuse -o allow_other -o ro --file-cache-timeout-in-seconds=120 --use-https=true
Failed to connect to the storage container. There might be something wrong about the storage config, please double check the storage account name, account key and container name. errno = 403

You may install install blobfuse first, refer to https://github.com/Azure/azure-storage-fuse/wiki/1.-Installation

andyzhangx · 2019-03-05T13:00:43Z

also, you k8s cluster is 1.8.7 which is very old, I cannot set up such old cluster now, cannot repro in my k8s cluster. I would suggest use a newer k8s version, e.g. 1.12.x to try again.

asjddd · 2019-03-05T16:43:58Z

root@k8s-agentpool1-37198164-vmss00000G:/etc/kubernetes/volumeplugins/azure~blobfuse# ls -la
total 12
drwxr-xr-x 2 root root 4096 Mar  1 07:21 .
drwxr-xr-x 4 root root 4096 Feb 12 17:25 ..
-rwxr-xr-x 1 root root 3814 Mar  1 07:21 blobfuse
root@k8s-agentpool1-37198164-vmss00000G:/etc/kubernetes/volumeplugins/azure~blobfuse# blobfuse --version
blobfuse 1.0.2
root@k8s-agentpool1-37198164-vmss00000G:/etc/kubernetes/volumeplugins/azure~blobfuse# md5sum blobfuse
96aedf8a661117666d13314927791cdb  blobfuse

asjddd · 2019-03-05T16:47:04Z

You can contact Prajwal who is trying to get this working directly at [email protected] for more details.

IvanovOleg · 2019-03-05T18:49:16Z

I have the same issue on CoreOS. Blobfuse is installed and works in /run/torcx/bin/blobfuse but flexvolume expects it to be in /usr/bin/blobfuse due to the hardcoded path:

kubernetes-volume-drivers/flexvolume/blobfuse/deployment/blobfuse-flexvol-installer/blobfuse

Line 5 in ec19f44

BLOBFUSE="/usr/bin/blobfuse"

Tue Mar 5 18:30:19 UTC 2019 ERROR: { "status": "Failure", "message": "Failed to mount device /dev/ at /var/lib/kubelet/pods/01e692db-3f73-11e9-884c-000d3a4793c1/volumes/azure~blobfuse/pgdatabase-pv-blobfuse-flexvol, accountname:xxx, error log:Tue Mar 5 18:30:19 UTC 2019 EXEC: /usr/bin/blobfuse /var/lib/kubelet/pods/01e692db-3f73-11e9-884c-000d3a4793c1/volumes/azure~blobfuse/pgdatabase-pv-blobfuse-flexvol --container-name=postgresql-backup --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120" }

andyzhangx · 2019-03-06T09:41:09Z

@IvanovOleg thanks for reporting, could your try edit /etc/kubernetes/volumeplugins/azure~blobfuse/blobfuse and check whether that works, I would then remove such hardcoded path in next version.

IvanovOleg · 2019-03-07T06:11:06Z

@andyzhangx
Hi, I've changed the path as you described and got the different error:
Thu Mar 7 02:00:09 UTC 2019 EXEC: mkdir -p /var/lib/kubelet/pods/b71df11f-407c-11e9-b871-000d3a44a2ce/volumes/azure~blobfuse/pgdatabase-pv-blobfuse-flexvol Thu Mar 7 02:00:09 UTC 2019 INF: export storage account - export AZURE_STORAGE_ACCOUNT=xxxxxxx AZURE_STORAGE_ACCESS_KEY=... Thu Mar 7 02:00:09 UTC 2019 EXEC: /run/torcx/bin/blobfuse /var/lib/kubelet/pods/b71df11f-407c-11e9-b871-000d3a44a2ce/volumes/azure~blobfuse/pgdatabase-pv-blobfuse-flexvol --container-name=postgresql-backup --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120 Thu Mar 7 02:00:09 UTC 2019 ERROR: { "status": "Failure", "message": "Failed to mount device /dev/ at /var/lib/kubelet/pods/b71df11f-407c-11e9-b871-000d3a44a2ce/volumes/azure~blobfuse/pgdatabase-pv-blobfuse-flexvol, accountname:xxxxxxxx, error log:Thu Mar 7 02:00:09 UTC 2019 EXEC: /run/torcx/bin/blobfuse /var/lib/kubelet/pods/b71df11f-407c-11e9-b871-000d3a44a2ce/volumes/azure~blobfuse/pgdatabase-pv-blobfuse-flexvol --container-name=postgresql-backup --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=120" }

MountVolume.SetUp failed for volume "pgdatabase-pv-blobfuse-flexvol" : invalid character 'U' looking for beginning of value

andyzhangx · 2019-03-07T06:20:59Z

@IvanovOleg could you try below commands on that VM directly, I am not sure whether blobfuse works on coreos:

export AZURE_STORAGE_ACCOUNT=xxx
export AZURE_STORAGE_ACCESS_KEY="xxx"

mkdir /tmp/abc
# /run/torcx/bin/blobfuse /tmp/abc --container-name=xxx --tmp-path=/tmp/blobfuse -o allow_other -o ro --file-cache-timeout-in-seconds=120 --use-https=true

IvanovOleg · 2019-03-07T06:24:58Z

@andyzhangx It works

/run/torcx/bin/blobfuse /tmp/abc --container-name=postgresql-backup --tmp-path=/tmp/blobfuse -o allow_other  --file-cache-timeout-in-seconds=120
# cd /tmp/abc
# ls
2019-03-06-devices.dump  2019-03-06-schedules.dump  2019-03-06-users.dump

I also tried to reproduce command from log and found that this directory doesn't exist:
/var/lib/kubelet/pods/b71df11f-407c-11e9-b871-000d3a44a2ce/volumes/azure~blobfuse/pgdatabase-pv-blobfuse-flexvol

this exists:
/var/lib/kubelet/pods/b71df11f-407c-11e9-b871-000d3a44a2ce/volumes/azure~blobfuse/

andyzhangx · 2019-03-07T06:29:24Z

@IvanovOleg mkdir does not work?

mkdir -p /var/lib/kubelet/pods/b71df11f-407c-11e9-b871-000d3a44a2ce/volumes/azure~blobfuse/pgdatabase-pv-blobfuse-flexvol

what is your kubelet? is it a container or daemon process? and your k8s version?

IvanovOleg · 2019-03-07T06:30:46Z

Kubelet is a systemd service. K8S 1.13.4

andyzhangx · 2019-03-07T06:32:46Z

@IvanovOleg How can I set up a k8s cluster on coreos?

IvanovOleg · 2019-03-07T06:33:56Z

It's painful) I did it using a custom terraform configuration + torcx images.

andyzhangx · 2019-03-07T06:35:52Z

@IvanovOleg Not sure whether your kubelet is a container or not, it's quite different, you could check by:
https://github.com/andyzhangx/demo/blob/master/debug/README.md#q-check-whether-kubelet-is-containerized-or-running-as-native-daemon

IvanovOleg · 2019-03-07T06:36:12Z

@andyzhangx It's a binary file

andyzhangx · 2019-03-07T06:52:38Z

@IvanovOleg then why mkdir -p /var/lib/kubelet/pods/b71df11f-407c-11e9-b871-000d3a44a2ce/volumes/azure~blobfuse/pgdatabase-pv-blobfuse-flexvol does not work? I cannot test that on coreos, would you contribute on coreos? Thanks.

IvanovOleg · 2019-03-07T06:56:47Z

@andyzhangx I will investigate it on CoreOs anyway, because I need this feature working. I will let you know as soon as I get results. Btw I have no issues with the other persistent volume types (azure disks).

IvanovOleg · 2019-03-07T07:55:25Z

@andyzhangx Here is what I've found on the agent node by executing sudo systemctl -xe:
Mar 07 07:46:15 agent1 kubelet[1474]: I0307 07:46:15.435340 1474 util.go:52] found flex volume secret info: accountkey Mar 07 07:46:15 agent1 kubelet[1474]: I0307 07:46:15.435360 1474 util.go:52] found flex volume secret info: accountname Mar 07 07:46:15 agent1 blobfuse[59748]: Unable to start blobfuse. If no config file is specified, exactly one of the environment variables AZURE_STORAGE_ACCESS_KEY or AZURE_STORAGE_SAS_TOKEN must be set. Mar 07 07:46:15 agent1 blobfuse[59748]: Unable to start blobfuse. Creating local blob client failed: errno = 1200.

I think I know the reason. There are those vars set in the kubelet env file in my configuration, because I use blobfuse on the node itself as well. I will try to move them to the different env file.

andyzhangx · 2019-03-07T08:02:58Z

export does not work on coreos?

kubernetes-volume-drivers/flexvolume/blobfuse/deployment/blobfuse-flexvol-installer/blobfuse

Line 78 in ec19f44

    
           export AZURE_STORAGE_ACCOUNT=${ACCOUNTNAME} AZURE_STORAGE_ACCESS_KEY=${ACCOUNTKEY}

IvanovOleg · 2019-03-07T08:10:24Z

@andyzhangx it works, but I have AZURE_STORAGE_SAS_TOKEN set and looks like we can't have key and token at the same time

andyzhangx · 2019-03-07T08:17:23Z

@IvanovOleg Yes, looks like I need to support AZURE_STORAGE_SAS_TOKEN also:

[OPTIONAL] --container-name=container : Required if no configuration file is specified. Also set account name and key/SAS via the environment variables AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_ACCESS_KEY/AZURE_STORAGE_SAS_TOKEN

IvanovOleg · 2019-03-08T07:56:23Z

@andyzhangx When can I expect a new release of the blobfuse-flexvolume with custom binary path and sas token support?

odedia · 2019-04-17T20:38:55Z

I got the same error while trying to mount an SMB share with flexvolume smb. Note that I didn't deploy to azure but to an on-premise PKS cluster.

andyzhangx · 2019-04-18T01:12:09Z

@odedia could you provide following info:

smb driver version
/var/log/smb-driver.log on the agent node

https://github.com/Azure/kubernetes-volume-drivers/tree/master/flexvolume/smb#debugging-skills

odedia · 2019-04-18T06:27:22Z

Information below.
We have concluded that the issue might be that jq is not installed on the worker nodes. However, in enterprise environments like PKS the worker nodes are managed and therefore any installation on them would be deleted on the next upgrade.
We're trying to think of a way to add jq itself to the demon set installation.

SMB driver version:

    Image:          mcr.microsoft.com/k8s/flexvolume/smb-flexvolume
    Image ID:       docker-pullable://mcr.microsoft.com/k8s/flexvolume/smb-flexvolume@sha256:570e6585d84735409b22487ea24dccda358321e79df722a25f6057d9e48812c3

Pod describe:

Events:
  Type     Reason       Age               From                                           Message
  ----     ------       ----              ----                                           -------
  Normal   Scheduled    12s               default-scheduler                              Successfully assigned default/oded-smb-test to 6b137daa-4276-4977-a09e-add6ef593906
  Warning  FailedMount  3s (x5 over 11s)  kubelet, 6b137daa-4276-4977-a09e-add6ef593906  MountVolume.SetUp failed for volume "test" : invalid character '/' looking for beginning of value

/var/log/smb-driver.log:

Thu Apr 18 06:18:02 UTC 2019 ERROR: { "status": "Failure", "message": "Failed to mount device at /var/lib/kubelet/pods/1542fb35-615c-11e9-9522-005056bb2735/volumes/microsoft.com~smb/test, user:, , error log: dmesg | tail or so." }
Thu Apr 18 06:20:04 UTC 2019 EXEC: mkdir -p /var/lib/kubelet/pods/1542fb35-615c-11e9-9522-005056bb2735/volumes/microsoft.com~smb/test
Thu Apr 18 06:20:04 UTC 2019 EXEC: /bin/mount -t cifs  /var/lib/kubelet/pods/1542fb35-615c-11e9-9522-005056bb2735/volumes/microsoft.com~smb/test
mount: wrong fs type, bad option, bad superblock on ,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<type> helper program)

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
Thu Apr 18 06:20:04 UTC 2019 ERROR: { "status": "Failure", "message": "Failed to mount device at /var/lib/kubelet/pods/1542fb35-615c-11e9-9522-005056bb2735/volumes/microsoft.com~smb/test, user:, , error log: dmesg | tail or so." }
Thu Apr 18 06:22:06 UTC 2019 EXEC: mkdir -p /var/lib/kubelet/pods/1542fb35-615c-11e9-9522-005056bb2735/volumes/microsoft.com~smb/test
Thu Apr 18 06:22:06 UTC 2019 EXEC: /bin/mount -t cifs  /var/lib/kubelet/pods/1542fb35-615c-11e9-9522-005056bb2735/volumes/microsoft.com~smb/test
mount: wrong fs type, bad option, bad superblock on ,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<type> helper program)

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
Thu Apr 18 06:22:06 UTC 2019 ERROR: { "status": "Failure", "message": "Failed to mount device at /var/lib/kubelet/pods/1542fb35-615c-11e9-9522-005056bb2735/volumes/microsoft.com~smb/test, user:, , error log: dmesg | tail or so." }

Output from dmesg | tail:

[257594.137416] CIFS VFS: Malformed UNC in devname.

andyzhangx · 2019-04-18T06:32:35Z

@odedia you need to install jq, and cifs-utils on work node:

apt-get install jq cifs-utils -y

what is PKS?

odedia · 2019-04-18T06:36:34Z

It's an Enterprise Kubernetes offering from Pivotal and VMware.

Thanks for the information, i'll try to get this working from the daemon set.

odedia · 2019-04-18T21:26:41Z

I forked the repo to let jq be installed as part of the daemon set automatically. This solves two problems:

worker nodes can come and go - be resized, repaved, patched etc. so manual installation of binaries is not ideal because the installation would be gone on the next upgrade when a VM is replaced.
in air-gapped environments, it is not possible to go out to the public web to install jq using apt-get.

I didn't do a pull request since I made the changes specifically to the smb version and removed the rest.
Thanks again!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MountVolume.SetUp failed for volume "test" : invalid character '/' looking for beginning of value #14

MountVolume.SetUp failed for volume "test" : invalid character '/' looking for beginning of value #14

asjddd commented Mar 1, 2019

andyzhangx commented Mar 1, 2019

asjddd commented Mar 4, 2019

andyzhangx commented Mar 5, 2019 •

edited

Loading

andyzhangx commented Mar 5, 2019

asjddd commented Mar 5, 2019

asjddd commented Mar 5, 2019

IvanovOleg commented Mar 5, 2019 •

edited

Loading

andyzhangx commented Mar 6, 2019

IvanovOleg commented Mar 7, 2019 •

edited

Loading

andyzhangx commented Mar 7, 2019

IvanovOleg commented Mar 7, 2019 •

edited

Loading

andyzhangx commented Mar 7, 2019

IvanovOleg commented Mar 7, 2019

andyzhangx commented Mar 7, 2019

IvanovOleg commented Mar 7, 2019

andyzhangx commented Mar 7, 2019

IvanovOleg commented Mar 7, 2019

andyzhangx commented Mar 7, 2019 •

edited

Loading

IvanovOleg commented Mar 7, 2019

IvanovOleg commented Mar 7, 2019 •

edited

Loading

andyzhangx commented Mar 7, 2019

IvanovOleg commented Mar 7, 2019 •

edited

Loading

andyzhangx commented Mar 7, 2019

IvanovOleg commented Mar 8, 2019

odedia commented Apr 17, 2019

andyzhangx commented Apr 18, 2019 •

edited

Loading

odedia commented Apr 18, 2019

andyzhangx commented Apr 18, 2019

odedia commented Apr 18, 2019

odedia commented Apr 18, 2019

MountVolume.SetUp failed for volume "test" : invalid character '/' looking for beginning of value #14

MountVolume.SetUp failed for volume "test" : invalid character '/' looking for beginning of value #14

Comments

asjddd commented Mar 1, 2019

andyzhangx commented Mar 1, 2019

asjddd commented Mar 4, 2019

andyzhangx commented Mar 5, 2019 • edited Loading

andyzhangx commented Mar 5, 2019

asjddd commented Mar 5, 2019

asjddd commented Mar 5, 2019

IvanovOleg commented Mar 5, 2019 • edited Loading

andyzhangx commented Mar 6, 2019

IvanovOleg commented Mar 7, 2019 • edited Loading

andyzhangx commented Mar 7, 2019

IvanovOleg commented Mar 7, 2019 • edited Loading

andyzhangx commented Mar 7, 2019

IvanovOleg commented Mar 7, 2019

andyzhangx commented Mar 7, 2019

IvanovOleg commented Mar 7, 2019

andyzhangx commented Mar 7, 2019

IvanovOleg commented Mar 7, 2019

andyzhangx commented Mar 7, 2019 • edited Loading

IvanovOleg commented Mar 7, 2019

IvanovOleg commented Mar 7, 2019 • edited Loading

andyzhangx commented Mar 7, 2019

IvanovOleg commented Mar 7, 2019 • edited Loading

andyzhangx commented Mar 7, 2019

IvanovOleg commented Mar 8, 2019

odedia commented Apr 17, 2019

andyzhangx commented Apr 18, 2019 • edited Loading

odedia commented Apr 18, 2019

andyzhangx commented Apr 18, 2019

odedia commented Apr 18, 2019

odedia commented Apr 18, 2019

andyzhangx commented Mar 5, 2019 •

edited

Loading

IvanovOleg commented Mar 5, 2019 •

edited

Loading

IvanovOleg commented Mar 7, 2019 •

edited

Loading

IvanovOleg commented Mar 7, 2019 •

edited

Loading

andyzhangx commented Mar 7, 2019 •

edited

Loading

IvanovOleg commented Mar 7, 2019 •

edited

Loading

IvanovOleg commented Mar 7, 2019 •

edited

Loading

andyzhangx commented Apr 18, 2019 •

edited

Loading