Unable to get inotify events inside container using kubernetes

[christopherL91]

Expected behavior

Should be able to get inotify events inside a container in kubernetes mounted with a host directory.

Actual behavior

Not able to get inotify events if there isn't another container outside kubernetes that mounts the same directory directly using docker run -v ...

Information

• Full output of the diagnostics from "Diagnose & Feedback" in the menu
• A reproducible case if this is a bug, Dockerfiles FTW
• Page URL if this is a docs issue or the name of a man page

Docker for Mac: version: 17.12.0-ce-mac45 (a61e84b8bca06b1ae6ce058cdd7beab1520ad622)
macOS: version 10.13.2 (build: 17C88)
logs: /tmp/E73BC492-DDF5-4CBB-B43D-7F7874CF3802/20180109-011402.tar.gz
[OK] db.git
[OK] vmnetd
[OK] dns
[OK] driver.amd64-linux
[OK] virtualization VT-X
[OK] app
[OK] moby
[OK] system
[OK] moby-syslog
[OK] kubernetes
[OK] env
[OK] virtualization kern.hv_support
[OK] slirp
[OK] osxfs
[OK] moby-console
[OK] logs
[OK] docker-cli
[OK] menubar
[OK] disk

Steps to reproduce the behavior

1. run docker run -it -v "some directory" alpine and install inotify apk --no-cache add inotify-tools and then run inotifywait -r -m --format "%e %f" 'some directory' in order to watch for inotify events

apiVersion: v1
kind: Service
metadata:
  name: alpine
spec:
  selector:
    app: alpine
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  type: LoadBalancer
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: alpine
spec:
  replicas: 1
  minReadySeconds: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: alpine
    spec:
      containers:
      - name: alpine
        image: alpine
        imagePullPolicy: Always
        ports:
        - name: http
          containerPort: 8080
        volumeMounts:
        - name: data
          mountPath: /data
      volumes:
      - name: data
        hostPath:
          path: data

create the following service and deployment using kubectl create -f ./alpine.yaml and exec into the container. There you run the same command as with the other container "apk --no-cache ..."

Write and modify the directory that is now mounted inside the two containers, and see that both containers receive inotify events.

Now remove the container that is running outside kubernetes and watch what happens inside the container that is running inside kubernetes. The container no longer receives any inotify events

[thaJeztah]

Possible duplicate of #2216
/cc @djs55

edit: whoops wrong link pasted

[majelbstoat]

Came here to report exactly the same thing. Thanks @christopherL91 for the info that events are passed if it's mounted in two places - for anyone else, I've confirmed that that does work, although for me, I had to specify both the local and remote directory verbosely:

docker run -it --rm -v ~/path/to/code:/usr/src/service alpine

Once I ran that command, containers inside Kubernetes that mounted the same directory started receiving notifications.

That workaround aside, the inability for webpack --watch to receive notifications pretty much makes this a non-starter for fast iterative development in the local kubernetes cluster. I hope a fix for this can be made in an upcoming edge release.

[fdanielsen]

Came across this issue when trying to solve my container running Webpack devserver not detecting changes and re-compiling. I'm not using Kubernetes, but a stack deployed with a compose file where my local directory is mounted as a volume for the Webpack service. Randomly I tried the suggestion of @majelbstoat. While the Webpack service was running, I just mounted the same directory into an alpine container (copied @majelbstoat's example just replacing the source path). Just letting that container run, any changes to my source files was picked up inside the Webpack container too…

I don't know anything about Kubernetes, but it seems these issues might be related then. Is there any changes in how the directory is mounted in the compose file that could solve it for now? Otherwise, having a "ghost" container with the directory mounted will be the solution for now.

Running Docker 18.05.0-ce-rc1-mac63 (24246) on the edge channel, on macOS 10.13.4 with APFS and Docker raw format (default).

[djs55]

Thanks for the report. I think what's happening is:

• kubernetes creates a container with a bind mount
• a controller sees the event and tells com.docker.osxfs to start sending events
• a periodic timer (with a 5 minute interval) fires
• a GC collects unused event registrations, where "unused" means "does not correspond to a running docker container". Since the kubernetes container names are not the same as the docker container names, the garbage collector deletes the kubernetes event registration. The reason docker run -v ... makes it work is that the GC sees the event registration is still in use and doesn't delete it.

I've got a prototype fix for this if you'd like to try it: https://download-stage.docker.com/mac/bysha1/98d31563898720b294dd40349228266a0dc2f969/Docker.dmg

Note that

• the build is a development build, only suitable for testing. It's not suitable for production.
• if it offers an auto-update, don't accept as this may upgrade to a build without the fix

If you get a chance to try it, let me know what happens!

[davidthornton]

I’m at Korean bbq rn but when I’m not drank I will test this.

[fdanielsen]

Hi @djs55,

I'm not sure how relevant this is since I'm not using Kubernetes (but docker stack), but as mentioned the issue seems to act in the same way.

Today in a slight lapse of reasoned thought I upgraded to the latest edge version of Docker on my work computer too, and started seeing inotify events stopping after a few minutes. So, I did a test run with your development build and it sadly had no effect for me. In the first few minutes changes were picked up, but then they stopped after about 5 minutes. Booting up a separate container with -v "restartet" events again.

Anything that would help you in further debugging and fixing this issue?

[djs55]

Hi @fdanielsen -- that does sound like a very similar scenario. I'll investigate locally, but meanwhile if you could try starting a stack and then running:

/Applications/Docker.app/Contents/MacOS/com.docker.osxfs  state

It should display the active event registrations. Hopefully there's an entry for the host path mentioned in the compose file. When the events stop I suspect the entry will have been deleted.

[djs55]

@fdanielsen I created a very simple image:

$ cat build/Dockerfile
FROM alpine

RUN apk --no-cache add inotify-tools

CMD ["/bin/sh", "-c", "sleep 3000"]

and a very simple docker-compose.yml:

version: "3"
services:

  test:
    build: build
    image: test
    volumes:
      - /path:/foo

If I launch this with docker-compose up it seems to work as expected: com.docker.osxfs state shows the event registration. If I launch this with docker stack deploy test -c docker-compose.yml then the event registration is missing, which means no events at all :(

[majelbstoat]

I don't know if this is relevant, but when i ran /Applications/Docker.app/Contents/MacOS/com.docker.osxfs state, I saw about a hundred mounts into different pod versions of the same service. These are old versions of pods that apparently weren't fully cleaned up even though kubernetes took them out of rotation. Restarting docker and kubernetes removed them.

[fdanielsen]

@djs55 That's interesting. Indeed that's how we deploy our stack and when I check the state as you suggest after deploying it this is all I see:

Exported directories:
 - /Users to /Users (nodes_/Users table size: 10006)
 - /Volumes to /Volumes (nodes_/Volumes table size: 0)
 - /private to /private (nodes_/private table size: 0)
 - /tmp to /tmp (nodes_/private/tmp table size: 6)
Container-mounted directories:

If I then start a separate container as mentioned earlier, this gets added in the second section:

 - /Users/… into 8ee07d94b402e8ac10fae67e8c2799f82ca347895e966af3926078a23a9fd364 (state=default)

So there's some issue with Docker not being aware of container-mounted directories for volumes in a deployed stack?

[djs55]

@fdanielsen thanks for your comment -- you're correct, I've dug into it and can confirm the current code only handles the docker run -v case and it misses the docker stack deploy case (and the docker run --mount case). I've got a prototype fix for this which I'm testing now.

@majelbstoat thanks for your comment also -- I've seen a similar phenomenon which seemed to be triggered by the use of a symlink in the path I was sharing with the container. Could this explain what you saw? If so, I have a candidate fix for this also. If symlinks aren't involved then I can't explain it :/

[majelbstoat]

@djs55, yes there were symlinks involved!

[YRM64]

In response to this issue I'm going to provide you a possible explanation from djs55. "I think what's happening is: 1) Kubernetes creates a container with a bind mount 2) A controller sees the event and tells com.docker.osxfs to start sending events 3) A periodic timer (with a 5 minute interval) fires, and 4) A GC collects unused event registrations, where "unused" means "does not correspond to a running Docker container". Since the Kubernetes container names are not the same as the docker container names, the garbage collector deletes the Kubernetes event registration. The reason docker run -v ... makes it work is that the GC sees the event registration is still in use and doesn't delete it".

Djs55 offers a prototype fix for this if you would like to try it at: https://download-stage.docker.com/mac/byshal/98d31563898720b294dd40349228266a0dc2f969/Docker.dmg. He warns that the build is only for developmental purposes; suited for testing only, and not for production. Several other developers have offered their own personal experiences, so there might be a little something in this one for everyone. If you decide to try the prototype djs55 wants feedback. I too am interested in whether it works for you. Happy notifying events inside your container ✌️

[djs55]

There's an even newer development build available: https://download-stage.docker.com/mac/bysha1/7fc56519ef8858a58fd0eb61d10c958688ff8767/Docker.dmg -- this one should have event fixes for

• docker run --mount type=bind (previously only docker run -v would work
• named volumes
• swarm services
• docker stack deploy

Let me know if you get a chance to try it!

[fdanielsen]

Hi @djs55, I've tested this for an hour or so now on my stack setup. It looked very promising at first, but the events suddenly stopped. The state command to see the list of container mounted directories showed the source directory mounted into 7 service containers right from the start, and still lists them even after events have stopped propagating. After events died I tried starting a plain alpine container with a -v mount as before, but events did not start propagating again.

I don't know if there's any other debug info I can get you to figure out why the events stopped? All the container mounts show state=default, as they have since starting up.

[fdanielsen]

An update to my previous comment. After it stopped working I recreated the stack (docker stack rm … and docker stack deploy …), and after that it worked throughout the rest of the work day. But then I left Docker running (usually do) and after leaving my laptop closed overnight, when I opened it the next day I got an error message from Docker about internal services failing. Ran the diagnostics which responded with this:

Docker for Mac: version: 18.05.0-ce-mac65 (07d57837a9c0ad813a1042c9dc39bbc5961d031d)
macOS: version 10.13.4 (build: 17E202)
logs: /tmp/270A2A95-162B-4B1E-A0A0-3D468FD6E41D/20180513-210107.tar.gz
failure: Unexpected error connecting to /Users/frode/Library/Containers/com.docker.docker/Data/s51: (Failure
  "Error connecting socket to 9p endpoint unix:/Users/frode/Library/Containers/com.docker.docker/Data/s51: Unix.Unix_error(Unix.ENOENT, \"connect\", \"\")")
[ERROR]  vpnkit
         com.docker.vpnkit is not running
         Unexpected error connecting to /Users/frode/Library/Containers/com.docker.docker/Data/s51: (Failure
  "Error connecting socket to 9p endpoint unix:/Users/frode/Library/Containers/com.docker.docker/Data/s51: Unix.Unix_error(Unix.ENOENT, \"connect\", \"\")")
[OK]     virtualization hypervisor
[OK]     vmnetd
[OK]     dns
[ERROR]  driver.amd64-linux
         com.docker.driver.amd64-linux is not running
[OK]     virtualization VT-X
[OK]     app
[OK]     moby
[OK]     system
[OK]     moby-syslog
[OK]     kubernetes
[OK]     files
[OK]     env
[OK]     virtualization kern.hv_support
[ERROR]  osxfs
         com.docker.osxfs is not running
[OK]     moby-console
[OK]     logs
[ERROR]  docker-cli
         Connection refused (ECONNREFUSED) connecting to /var/run/docker.sock: check if service is running
         Connection refused (ECONNREFUSED) connecting to /Users/frode/Library/Containers/com.docker.docker/Data/docker.sock: check if service is running
[OK]     disk

Also got Diagnostic ID 270A2A95-162B-4B1E-A0A0-3D468FD6E41D/20180513-210107. I've never seen anything like this happen before, so wonder if something in the patched version crashed hard?

[docker-robott]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

[djs55]

@fdanielsen sorry for the delay getting back to you. Unfortunately I couldn't find the uploaded diagnostic -- perhaps the upload failed. Are you still experiencing the same problem? The latest edge and stable has the fixes incorporated -- could you try again with one of those builds and upload a fresh diagnostic?

Thanks!

[wvl]

@djs55 I am seeing this exact same issue.
Diagnostic: E1A60D95-3A00-4A0F-9FAF-C6BCB69AD186/20181025205053

• Docker for mac, 2.0.0.0-beta1-mac75
• Running a service with kubernetes

The filesystem is mounted:
osxfs on /core type fuse.osxfs (rw,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)

It works as expected, however no filesystem events are being sent to the container.

When I run /Applications/Docker.app/Contents/MacOS/com.docker.osxfs state, the directory does not show up as container mounted.

When I run the image with docker run -it -v dir:/core bash, then the directory shows up in com.docker.osxfs state, and filesystem events are propagated as expected.

If I can provide anymore information, please let me know.

[mhart]

FWIW I think I'm seeing this, or something similar, on Catalina (10.15.1) with Docker 19.03.5.

Starting docker from /Users/michael/test with:

docker run -it -v $PWD:/app alpine sh -c \
  'apk --no-cache add inotify-tools; inotifywait -m --format "%e %f" /app/file'
# ...
Setting up watches.
Watches established.

Running:

/Applications/Docker.app/Contents/MacOS/com.docker.osxfs state

Shows the mount ok:

Container-mounted directories:
 - /Users/michael/test into docker/4b65d1da9fc7e7f305df1ba38dcc21243e15736636d0473309e89b02111a8c1b (state=default)

But it doesn't register any changes to the file (/Users/michael/test/file) if I modify it from the host. No events fire at all.

Running this (modifying from within the container) works fine:

docker run -it -v $PWD:/app alpine sh -c \
  'apk --no-cache add inotify-tools; inotifywait -m --format "%e %f" /app/file & sleep 1; echo hello > /app/file'
# ...
Setting up watches.
Watches established.
MODIFY 
OPEN 
MODIFY 

Has something changed here, or do I need to specify something extra when mounting the volume?

[mhart]

Hmmm, interesting. I restarted Docker and now the above scenario is working. Not sure if I can reproduce now 🤔

[mateusmedeiros]

@mhart I actually had a very very similar experience, and was already typing my comment here describing how I could reproduce the problem and how I tried restarting docker and even uninstalling and installing different versions of docker to see if it was a recent regression.

I was trying different things with the shared directories in the configuration and trying to mount things in different directories in my file system, mounting a directory directly, then mounting it's parent and pointing inotifywait to the subdir, then mounting a file and modifying, just to be sure I could describe everything and maybe discover something related to the root of the apparent bug. Then as a last thing to try, I tried copy and pasting your example, and it magically worked. Then my attempts also worked, everything just started working magically.

I'm not quite sure if there's a bug involved, if it is related to the original issue that the OP had, how to reproduce, if it's related to the latest version of macOS nor if this comment will help in any way since at the end of the day I couldn't really have a reproducible example made, but yeah, something weird with the inotify events definitely happened here.

In my case restarting Docker did NOT work. I have no idea what actually did, but I'd bet that rebooting does the trick (though I'll have to test that specifially the next time I encounter this problem).

[d-franklin]

I'm having the exact same issue on a windows machine, so it's not limited to mac.

[infostreams]

Why is this closed? It's not solved.

[djs55]

@infostreams see this comment where the bot explained what it planned to do. There's normally a 90 day delay between the comment and the close (if no-one tells it to stop) but due to a bug in the bot it was almost 2 years! I'll reopen this issue for now.

[infostreams]

It’s still not solved

/lifecycle frozen

[azoff]

+1 would like to understand how to better debug filesystem events over the osxfs bridge.

for a case I'm working with:

• fsevents API appears to work on the host MacOS installation (i.e. tested via fswatch)
• inotify API appears to work on the containerized Ubuntu installation (i.e. tested via inotifytools)
• osxfs does not appear to bridge the fsevent over to inotify upon file change (i.e. tested using touch on MacOS)

To workaround this, I wrote a script that binds to fsevents on the host machine, and delegates changes manually using docker exec [container_id] touch [file_path] in the target container. Not my favorite solution, but it beats having to restart docker or the host machine itself.

As I understand it, this is still considered the only known workaround to the issue in question here.

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked