Error creating vxlan interface: file exists

[discotroy]

Previous related threads:

• network sandbox join failed: error creating vxlan interface: file exists #562
• What do I do with "subnet sandbox join failed for "10.0.0.0/24": error creating vxlan interface: file exists" #751
• Cannot start container: subnet sandbox join failed for "10.0.0.0/24": error creating vxlan interface: file exists #945
• Containers on overlay network and "error creating vxlan interface: file exists" moby#21482
• [1.13.0-rc1][Intermittent] docker: Error response from daemon: subnet sandbox join failed for "10.0.0.0/16": error creating vxlan interface: file exists. moby#28559

Comment at the current tail-end of #945 recommends opening a new ticket. I couldn't find one opened by the original poster, so here we go.

I've been using swarm for the past couple of months, and frequently hit upon this problem. I have a modest swarm (~8-9 nodes) all running Ubuntu 16.04, now with Docker 17.05-ce on. There is not a great amount of container churn, but I do use a stack yaml file to deploy ~20 services across ~20 encrypted overlay networks.

I tend to find that after a couple of stack deploy / stack rm cycles, my containers get killed at startup with the "Error creating vxlan: file exists" error. This prevents the containers coming up on a host and forces them to attempt to relocate, which may / may not work.

I have noted in the above issues that the problems are, several times over, thought to have been rectified, but yet always creep back in for various users.

To rectify the issue, I have tried rebooting the node, restarting iptables, removing the stack and re-creating, all of which work to varying degrees but are most definitely workarounds and not solutions.

I cannot think how I can attempt to reproduce this error, but if anyone wants to suggest ways to debug, I am at your service.

[mpepping]

We're suffering from the same issue on RHEL7 /w Docker 17.03-ee and are able to reproduce the issue by adding a Service on a swarm-node where the overlay-network isn't active yet.
Tried about the same level of troubleshooting as @discotroy and can confirm the rebooting or restarting docker-engine fixes the issue up to some level, with fluctuating results. Also open for suggestions on how to debug this issue.

[fcrisciani]

Do you guys have some logs to share? would be super helpful to have a way to reproduce and grab logs with the engine in debug mode

Engine in debug:
echo '{"debug": true}' > /etc/docker/daemon.json
then: sudo kill -HUP <pid of dockerd>

[mpepping]

Will collect more logging. Here's some debug /w the error-message: https://gist.github.com/mpepping/50cb9b71b5535b318c6a548d4e8ba97b

[fcrisciani]

@mpepping thanks, the error message is clear. The current suspect that I have is a race condition during the deletion of the sandbox that leak the vxlan interface behind it. When a new container comes up tries to create the vxlan interface again and instead finds that there is already one and errors out. The more interesting part now of the logs would be the block where there is suppose to be the interface deletion and figure out why that is not happening properly.

[fcrisciani]

I'm also already trying to reproduce it locally, but if you guys narrow down a specific set of steps that are able to reproduce with high probability let me know

[mpepping]

@fcrisciani indeed it seems a race condition running into a locking issue. A breakdown of the steps, with debug output, is available at https://gist.github.com/mpepping/739e9a486b6c3266093a8af712869e90 .

Basically, the command-set for us to reproduce the following .. but the gist provides more detail:

docker swarm init
docker network create -d overlay  ucp-hrm
docker stack deploy -c stack.yml test
docker service ls #OK
docker stack rm test
docker service ls
docker stack deploy -c stack.yml test
docker service ls #NOK

Also, we're running into this issue on RHEL7 /w Docker 1703-ee on VMware vSphere virtuals. We were thus far unable to reproduce the issue on Virtualbox or VMware Fusion, using the same stack. Our next steps would be to run an other OS on VMware vSphere to reproduce the issue, and debug the vxlan config.

[pjutard]

Same problem here. Same scenario: multiple stacks deployed, each with its own network, after some docker stack rm and docker stack deploy, we get the "Error creating vxlan: file exists" error msg.
We have a swarm in this state right now...

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 21:43:09 2017
 OS/Arch:      linux/amd64
 Experimental: false

Using Docker4Azure

[mpepping]

Allright, some extensive tests led to interesting results. We're running RHEL7 /w docker-1703-ee.
The issue was direct reproducible when running the 3.10.0-327.10.1.el7.x86_64 kernel with iptables (firewalld removed). A docker swarm deploy/rm/deploy combo fails every test run on this setup.
After bumping the kernel (3.10.0-514.6.1.el7.x86_64) and installing+enabling the firewalld service, the results are much more reliable .. but still can break after 200+ or 800+ docker swarm deploy/rm/deploy runs .. after which rebooting the host is the only reliable way to fix this. Note that just bumping the kernel, or enabling firewalld isn't sufficient .. the combination of both made the difference in our use case.

[jgeyser]

As per #562

You can correct this by running:

sudo umount /var/run/docker/netns/*
sudo rm /var/run/docker/netns/*

Not sure if this is a long term solution.

[mavenugo]

@jgeyser thats a workaround to get out of issue. But that is not a solution. We have to narrow down the RC and fix it in the code.

[dcrystalj]

@jgeyser this is not working for me. sometimes i also get issue Unable to complete atomic operation, key modified

I have tried removing docker-engine and leaving docker swarm, but it didn't work.

update:
Needed full machine restart as well

[dang3r]

We are encountering the same problem with the following configuration:

Linux hostname 4.9.0-0.bpo.2-amd64 #1 SMP Debian 4.9.13-1~bpo8+1 (2017-02-27) x86_64 GNU/Linux

Client:
 Version:      17.03.0-ce
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 10:53:29 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 10:53:29 2017
 OS/Arch:      linux/amd64
 Experimental: false

[lnshi]

Exactly same problem experienced, quite randomly.