losetup in buildroot VM does not support -j: MapBlockVolume failed: exit status 32

[anencore94]

Steps to reproduce the issue:

1. Create pv, pvc, pod with spec.VolumeMode:Block

• the same yaml file succeed with another k8s clusters installed with kubespray and kubeadm.
• also, with the same yaml only different with spec.VolumeMode:Filesystem was succeed

2. However, creating pod failed in minikube:
   pod stuck in containerCreating status, but the error message was different depends on minikube version (error message in describe pod and journalctl -u kubelet and minikube log was similar, so added only describing pod events log)

• in minikube v1.10.1 (k8s v1.18.2)

Events:
  Type     Reason           Age    From               Message
  ----     ------           ----   ----               -------
  Normal   Scheduled        5m25s  default-scheduler  Successfully assigned default/busybox-vol to minikube
  Warning  FailedMapVolume  5m25s  kubelet, minikube  MapVolume.MapBlockVolume failed for volume "example-pv-volume" : blkUtil.MapDevice failed. devicePath: /home/docker/aaa, globalMapPath:/var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/example-pv-volume, podUID: dd860442-0b3f-4dd3-bf13-7994f0d93b80, bindMount: true: failed to bind mount devicePath: /home/docker/aaa to linkPath /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/example-pv-volume/dd860442-0b3f-4dd3-bf13-7994f0d93b80: mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/example-pv-volume/dd860442-0b3f-4dd3-bf13-7994f0d93b80 --scope -- mount  -o bind /home/docker/aaa /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/example-pv-volume/dd860442-0b3f-4dd3-bf13-7994f0d93b80
Output: Running scope as unit: run-r3be8748ee1704f7d9b575e971e6acc8b.scope
mount: /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/example-pv-volume/dd860442-0b3f-4dd3-bf13-7994f0d93b80: mount point is not a directory.
  Warning  FailedMapVolume  5m24s  kubelet, minikube  MapVolume.MapBlockVolume failed for volume "example-pv-volume" : blkUtil.MapDevice failed. devicePath: /home/docker/aaa, globalMapPath:/var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/example-pv-volume, podUID: dd860442-0b3f-4dd3-bf13-7994f0d93b80, bindMount: true: failed to bind mount devicePath: /home/docker/aaa to linkPath /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/example-pv-volume/dd860442-0b3f-4dd3-bf13-7994f0d93b80: mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/example-pv-volume/dd860442-0b3f-4dd3-bf13-7994f0d93b80 --scope -- mount  -o bind /home/docker/aaa /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/example-pv-volume/dd860442-0b3f-4dd3-bf13-7994f0d93b80
Output: Running scope as unit: run-re84bf69ccf79445dbbc71358e5b23c43.scope
mount: /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/example-pv-volume/dd860442-0b3f-4dd3-bf13-7994f0d93b80: mount point is not a directory.
  Warning  FailedMapVolume  5m24s  kubelet, minikube  MapVolume.MapBlockVolume failed for volume "example-pv-volume" : blkUtil.MapDevice failed. devicePath: /home/docker/aaa, globalMapPath:/var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/example-pv-volume, podUID: dd860442-0b3f-4dd3-bf13-7994f0d93b80, bindMount: true: failed to bind mount devicePath: /home/docker/aaa to linkPath /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/example-pv-volume/dd860442-0b3f-4dd3-bf13-7994f0d93b80: mount failed: exit status 32

• in minikube v1.8.2 (k8s v1.17.1)
  (tried with hostname : m01, not minikube)

Events:
  Type     Reason           Age              From               Message
  ----     ------           ----             ----               -------
  Normal   Scheduled        5s               default-scheduler  Successfully assigned default/busybox-vol to m01
  Warning  FailedMapVolume  1s (x4 over 5s)  kubelet, m01       MapVolume.MapBlockVolume failed for volume "example-pv-volume" : blkUtil.AttachFileDevice failed. globalMapPath:/var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/example-pv-volume, podUID: 43ef6e11-853f-46fa-a55b-99305d629cbd: GetLoopDevice failed for path /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/example-pv-volume/43ef6e11-853f-46fa-a55b-99305d629cbd: losetup -j /var/lib/kubelet/plugins/kubernetes.io~local-volume/volumeDevices/example-pv-volume/43ef6e11-853f-46fa-a55b-99305d629cbd failed: exit status 1

• in minikube v1.10.1-beta2 (k8s v1.15.3 and v1.18.1)
  • v1.15.3 says nothing, just say failedMapVolume
  • v1.18.1 says same as minikube v1.10.1 (k8s v1.18.2)

Full output of failed command:

• I guess it happens since losetup is invalid in minikube, hint from the error message in minikube v1.8.2
  • related issues in kubernetes/kubernetes :
    • issue-2

1. output of losetup in minikube

$ sudo ls /usr/sbin -al | grep losetup
lrwxrwxrwx 1 root root      14 May 11 21:37 losetup -> ../bin/busybox

$ losetup -j
losetup: invalid option -- 'j'
BusyBox v1.29.3 (2020-05-11 14:37:18 PDT) multi-call binary.

Usage: losetup [-r] [-o OFS] {-f|LOOPDEV} FILE - associate loop devices
	losetup -d LOOPDEV - disassociate
	losetup -a - show status
	losetup -f - show next free loop device

	-o OFS	Start OFS bytes into FILE
	-r	Read-only
	-f	Show/use next free loop device

2. expected output of losetup as in ubuntu (5.3.0-51-generic)

$ losetup --help | grep j
 -j, --associated <file>       list all devices associated with <file>

• it seems the losetup in minikube is not valid.

Full output of minikube start command used, if not already included:

• Ubuntu 18.04
• minikube v1.10.1, v1.10.1-beta2, v1.8.2
• kubernetes v1.18.2, etc.
• virtualbox
• Docker 19.03.8
• Enabled addons: default-storageclass, storage-provisioner
  • also tried with disabling storage-provisioner

[tstromberg]

The root cause here is that there are different versions of losetup. It seems that this functionality is expecting one that supports -j. Just curious - does this work if you use minikube start --driver=docker?

The Docker driver uses an Ubuntu based environment rather than the buildroot VM.

[anencore94]

@tstromberg Yeap, the minikube with docker driver works well. And I can see the losetup in docker-driver-minikube has -j option, I guess your comment was right. Thanks.

Then the virtualbox-driver-minikube base image need to be changed to support that losetup version ?

[afbjorklund]

I think that we can change from busybox to util-linux, by setting BR2_PACKAGE_UTIL_LINUX_LOSETUP=y

That's the one that kicbase uses:

docker@docker:~$ losetup --version
losetup from util-linux 2.34

This is in the builldroot config file: deploy/iso/minikube-iso/configs/minikube_defconfig

[nixpanic]

I just hit this problem on minikube 1.12.1 and kvm2 too. Copying the /sbin/losetup executable from the CentOS 8 host into the minikube VM makes it work (but is a very ugly workaround).

I'll try to build an ISO with BR2_PACKAGE_UTIL_LINUX_LOSETUP=y and see if that works.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.