macOS mount: mounted directory is empty

[zhaytee]

Is this a BUG REPORT or FEATURE REQUEST?
Bug report

Please provide the following details:

Environment:

Minikube version: v0.25.0

• OS: macOS 10.13.2
• VM Driver: hyperkit
• ISO version: v0.25.1
• Others: kubernetes v1.7.5

What happened:
Tried to mount a host directory into a container. The directory inside the container is empty.

What you expected to happen:
The directory inside the container should give access to the mounted directory on the host. At least, that was the behavior when I was using minikube and VirtualBox!

How to reproduce it:

$ eval $(minikube docker-env)
$ mkdir test; cd test; echo 'hello' > test.txt
$ docker run --rm -it -v $(pwd):/testmount alpine
(now inside alpoine container)
/ # cd testmount
/testmount # ls -l
total 0

Anything else do we need to know:
Is this not possible when using hyperkit? If not, what is the alternative? I noticed some options relating to NFS, but I didn't dig too deeply. I'd prefer to just use the standard Docker volume mount mechanism if possible.

Thank you!

[parasyte]

Workaround this issue by using scp to copy the files into the VM:

scp -ri "$(minikube ssh-key)" "$PWD" docker@$(minikube ip):/tmp
docker run --rm -it -v /tmp:/testmount alpine

[sshilko]

With xhyve being deprecated would be great to get it fixed, virtualbox is slower for local environments. (i.e. minikube delete deletes all images cache and need to refetch again, while hyperkit caches them on host machine)

[fenos]

I'm suffering this issue as well 😞

[alphabt]

+1 same issue here

[zhaytee]

This still appears to be an issue with minikube v0.28.0, k8s v1.10.0, and macOS 10.13.5. The hyperkit driver in general seems to be pretty solid now (although kubeadm doesn't work at all; localkube must be used), however without a mounting mechanism, minikube isn't useful for local development. Which kind of defeats the purpose.

Back to Virtualbox!

[ekimia]

any update on when a fix for this will be released? Agree with @zhaytee that this is a big blocker to use hyperkit. We are using xhye for now.

[dlorenc]

AFAIK the xhyve driver supports 9p or NFS mounts. Can you explain which one you're trying to use with hyperkit?

[ekimia]

@dlorenc just trying to do the standard mounts using hostPath. For xhyve and virtualbox, this is all thats needed since my mount point is already in Users. Below is a snippet of my deployment.yaml template:

{{ define "txechart.volumes"}}
      volumes:
      - name: txe-v
        hostPath:
          path: /Users/ek/myproject/code
{{- end }}

{{ define "txechart.volumeMounts"}}
          volumeMounts:
          - mountPath: /code
            name: txe-v
{{- end}}

[bw2]

Also seeing this issue in minikube v0.28.1

[jpswade]

Can replicate this issue in latest...

$ minikube version
minikube version: v0.28.2

[tstromberg]

This definitely seems like a poor user experience. We need to do a deep dive to make mounts more reliable.

As a workaround, I suspect using VirtualBox as a a vm-driver may work, as it has a more robust mounting mechanism.

[itaysk]

still on v0.30.0

[sshilko]

just give up on windows

[jamescarson]

Still on version: v0.32.0

VirtualBox is working fine but poor performance.

[kevinkirkup]

Still an issue on v0.33.1

[mkempster]

It's a poor workaround to be sure, but this command left running in a terminal

minikube mount /Users:/Users

Seems to make the above testcase work as expected

docker run --rm -it -v $(pwd):/testmount alpine

[AnthonyNixon]

Still an issue on v1.0.0

[tstromberg]

I'm curious, for those of you who are seeing this - is this occurring with a directory that contains more than 600 files?

If so, it may be related to #1753

[MaurGi]

the Mac /Users folder is not getting mounted, this still happens in 1.2.0 and on Mac OS 10.14.5:

> docker-machine-driver-hyperkit version
This is a Docker Machine plugin binary.
Plugin binaries are not intended to be invoked directly.
Please use this plugin through the main 'docker-machine' binary.
(API version: 1)
> hyperkit -v
hyperkit: v0.20180403-41-g64bbfb

Homepage: https://github.com/docker/hyperkit
License: BSD
> minikube version          
minikube version: v1.2.0
> minikube start  
😄  minikube v1.2.0 on darwin (amd64)
👍  minikube will upgrade the local cluster from Kubernetes 1.14.0 to 1.15.0
💿  Downloading Minikube ISO ...
 129.33 MB / 129.33 MB [============================================] 100.00% 0s

⚠️  Ignoring --vm-driver=virtualbox, as the existing "minikube" VM was created using the hyperkit driver.
⚠️  To switch drivers, you may create a new VM using `minikube start -p <name> --vm-driver=virtualbox`
⚠️  Alternatively, you may delete the existing VM using `minikube delete -p minikube`

🔄  Restarting existing hyperkit VM for "minikube" ...
⌛  Waiting for SSH access ...
🐳  Configuring environment for Kubernetes v1.15.0 on Docker 18.06.2-ce
💾  Downloading kubelet v1.15.0
💾  Downloading kubeadm v1.15.0
🚜  Pulling images ...
🔄  Relaunching Kubernetes v1.15.0 using kubeadm ... 
⌛  Verifying: apiserver proxy etcd scheduler controller dns
🏄  Done! kubectl is now configured to use "minikube"
> minikube ssh
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ ls
$ ls /
bin  data  dev  etc  home  init  lib  lib64  linuxrc  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
$ exit
logout

[riklopfer]

I do not have the same problem when running with the virtualbox driver. Maybe virtual box should be flagged as the preferred driver instead of Docker: https://minikube.sigs.k8s.io/docs/drivers/

$ minikube start --driver=virtualbox --cpus 4 --memory 8196
😄  minikube v1.18.1 on Darwin 11.2.3
    ▪ MINIKUBE_ACTIVE_DOCKERD=minikube
✨  Using the virtualbox driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating virtualbox VM (CPUs=4, Memory=8196MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.3 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v4
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

$ eval $(minikube docker-env)
$ mkdir test; cd test; echo 'hello' > test.txt
$ docker run --rm -it -v $(pwd):/testmount alpine
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
ba3557a56b15: Pull complete
Digest: sha256:a75afd8b57e7f34e4dad8d65e2c7ba2e1975c795ce1ee22fa34f8cf46f96a3be
Status: Downloaded newer image for alpine:latest
/ # ls -l /testmount/
total 4
-rw-r--r--    1 1000     1000             6 Mar 23 19:17 test.txt
/ #

[sharifelgamal]

@riklopfer This issue is about using the minikube mount command to mount files from your local filesystem into the minikube VM or container. What you are doing is not the same thing. I believe minikube mount now works for macOS on all drivers.

[riklopfer]

@sharifelgamal I am running the commands which were originally reported to reproduce this issue (see #2481 (comment)) . Perhaps this issue changed subject at some point. If this should be tracked elsewhere, #10851 can be reopened.

[legend-zeratul]

Issue happens on arch - docker driver as well. Mounting the folder manually works, able to see the contents, mount at start doesnt..

[legend-zeratul]

My Minikube version is: v1.18.1
Docker version: 20.10.5
OS: Endeavour (Arch base)
Linux Kernel: 5.4.101-1-lts54

Currently working by manually running the mount command after start, but shouldnt this be at start?

[sharifelgamal]

@legend-zeratul to be clear, it's the --mount and --mount-string parameters for minikube start that's not working for you?

[legend-zeratul]

OK, let me add a more detailed description..

For development, say I'm building appX and as part of containerising the app, I've moved out all persistent stores into a set of folders, say, dataX.. As I've understood, the base dataX folder needs to be mounted into the minikube's /data directory (for example), and the app's persistent volumes refer to this /data directory in their descriptors.. As part of normal dev cycles, anyone would need the ability to mount, unmount this directory and undeploy/deploy the containers independently.. The expectation would also be that if the local machine is restarted, the folder is automatically re-mounted when minikube starts..

While I understand local folder mounts are a security risk in integration/staging/production type setups, a local dev setup (minikube's focus domain) should be fine with local folder mounts, so the expectation would be to make this as seamless as possible to ease development time activities..

Problems:

• Mount folders need to be specified at cluster creation time, in order for the mount to work.. Trying to add a mount folder after the cluster is created doesnt work (both with/without the kill=true flag, if I remember correctly).. (I lost 3 days trying to understand why it worked one day and it didnt work the next)
• Dev time activities would not expect a terminal process to be kept running - so these will most likely use the kill=true flag..
• Restarting minikube should auto-mount the folder specified to /data.. (currently it doesnt). This adds an extra layer of complexity for the data-db-init folder that sql databases use because now when minikube starts - the /data folder is empty, the containers that depend on this folder now assume that the DB needs to be reinitialized, leading to interesting scenarios..
• I found the cluster config json's location later - so I dont know if editing the cluster config itself works or not.. (in other words - is the bug in the parameter processing, or is it in the mount part of the code.. ..need to test)

Docker stack does the expected behaviour very very nicely.. Restarting brings everything up to where it was..

The sequence that does work:

• Specify the mount folders at cluster creation time
• Use minikube mount once without the kill=true flag, then Ctrl+C to kill the process..
• Re-mount with the kill=true flag..

(looks like a weird issue - if the second step is skipped and we attempt a minikube mount with the kill=true flag directly, the mount process exits successfully, but the /data folder is empty (not correctly mounted))

The above sequence gets a working cluster with the folders mounted, however bringing down the cluster needs the entire sequence to be repeated (without kill=true and then with kill=true) for the data folders to get mounted.. You can imagine the fun that we will now have when service containers -> need persistent volume claims -> need persistent volumes -> need /data folder mounted: restarting the cluster or the machine as a result requires us to undeploy/deploy everything - this is the point when this stopped being workable..

Hope that helps - open to helping out with testing a patch if needed.. Do let me know.. (dont hv a local source->compile build setup, but will figure that out when needed).

Thanks.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[d3vpasha]

This issue is still active

[sylvain-combe-sonarsource]

Very unreliable 9P mounts on Hyperkit, MacOS 11.6 and minikube 1.23.1
Some files present and with 444 permissions in my host file system are unseen from the container mounting the dir, while some of those files are seen as directories...
File flags and attributes are a mess I would say.

I was planning to replace docker desktop with minikube but migration of existing volumes has been very difficult so far, because of these minikube bugs.

[lubumbax]

More than three years since this issue was opened...
it is still reproducible under Minikube v1.24.0 and Hyperkit 0.20200908

[alokarora0702]

Its still an issue even on HyperV, you mount at start and then it simply stops working after some time, while you are updating some PODs using the same volume

[DmytroLopushanskyy]

I had a similar problem on M1 Mac and ended up solving it using this command:

minikube mount /Users:/Users

[scottiedog45]

One thing to note, after running minikube mount command, it started working for me when I changed the referenced paths in the manifest to be the target dir of the minikube mount command, like in this example (notice /host is in the manifest, vs the path of the host)