port is already allocated without running containers

[oliverhr]

Expected behavior

Container creation

Actual behavior

Error: port is already allocated

Information

Diagnostic ID: 269F8ACE-C09C-4FB4-856C-8208C92B0020
Docker for Mac: 1.12.0-beta21 (Build 11019)
macOS: Version 10.11.6 (Build 15G31)
[OK] docker-cli
[OK] app
[OK] moby-syslog
[OK] disk
[OK] virtualization
[OK] system
[OK] menubar
[OK] osxfs
[OK] db
[OK] slirp
[OK] moby-console
[OK] logs
[OK] vmnetd
[OK] env
[OK] moby
[OK] driver.amd64-linux

Steps to reproduce

1. Run a container using -p with any port - in my case two containers nginx (80, 443) and a node app running on 10011
2. Remove the container
3. Create the same or another container on the same port

---

Result for command docker ps -a:

Portscan on 0.0.0.0 with docker running - But without running any container

Portscan on 0.0.0.0 without docker:

[oliverhr]

Problem remains even after uninstall beta version and install stable.

[oliverhr]

Solution:

I don't like but – It is what it is

[jamesdixon]

Also seeing this when my mac sleeps and this is woken up; containers are killed (issue #85) and then when trying to restart containers using docker-compose up, I'm seeing the above error. Only fix is to restart my machine :(

[qxhy123]

same here, reset to factory defaults would fix that.

[jamesdixon]

In addition to this and #85, it's making development very difficult. Any additional information/assistance I can provide to help?

[sergeyklay]

The same issue on OSX 10.11.6 with Docker 1.12.1 stable

[djs55]

@sergeyklay could you upload diagnostics from a reproduction on 1.12.1? The diagnostics in 1.12.1 are much better than in 1.12.0, in particular they include a process list from inside the VM which would show ports being held open from there. (The port opening is requested by the docker engine by running a proxy process, which communicates with a server on the Mac which actually opens the port. When the process inside the VM quits, the ports are closed on the Mac)

It would also really help if someone could help me reproduce this. Has anyone got a simple script or compose file they could share? Every time I try to repro this it always seems to work as expected :(

For example if I run an nginx in one terminal:

$ docker run -it -p 80:80 -p 443:443 nginx

and then in another terminal

$ curl http://localhost
... lots of html ...

I can see the ports bound via netstat:

$ netstat -an | less
Active Internet connections (including servers)
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)    
tcp4       0      0  *.80                   *.*                    LISTEN     
tcp4       0      0  *.443                  *.*                    LISTEN  
...

I can then Control+C the docker run and the ports go away. If I run docker run again it all comes back as expected :(

For a more complicated example, if I clone the example-voting-app:

$ git clone git://github.com/docker/example-voting-app
Cloning into 'example-voting-app'...
remote: Counting objects: 238, done.
remote: Total 238 (delta 0), reused 0 (delta 0), pack-reused 238
Receiving objects: 100% (238/238), 182.09 KiB | 0 bytes/s, done.
Resolving deltas: 100% (71/71), done.

I can repeatedly bring it up and down with

$ docker-compose up -d
# manually verify that ports 5000 and 5001 are bound
$ docker-compose down
# manually verify that ports 5000 and 5001 are free

with no error :(

[counterbeing]

Just wanted to add, I ran into this as well, but on different ports.

➜  lsof | grep 16380
com.docke 14092 cory   20u    IPv4 0xc6baa1d6e0a73c6d        0t0      TCP *:16380 (LISTEN)
com.docke 14092 cory   21u    IPv6 0xc6baa1d6d43e7c75        0t0      TCP localhost:16380 (LISTEN) (LISTEN)

This was with no containers running, and even after removing all containers, restarting docker, and then finally reinstalling docker for mac.

Restoring factory settings and setting everything up again now. Fortunately, that part is all automated... but it's a very odd problem.

[counterbeing]

I don't know what that was, but restoring to factory defaults did the trick!

[sergeyklay]

did the trick

Ya, and removes all images, containers, volumes, networks, etc

[oliverhr]

This happen again after shutdown computer while closing docker.

On linux users have reported that after remove local-kv.db with all containers stoped the problem gets solved.

The thing is on Mac I can't find this file. On linux the path is /var/lib/docker/network/files/ but not in OsX or macOS.

[roelvanhintum]

Just ran into the same issue. After trying out some things, i found it. I did not want to reset my machine.

• run docker-compose down (or similar when not using compose) to remove the network
• stop docker and start again
• start container without issues (or test with lsof -i tcp:PORTNUMBER)

[oliverhr]

I already tried by stopping all the containers, even removing all of them. The reset is a desperate measure but is the only thing that kinda works, as I say before by far is not the best but It is what it is at the moment.

[roelvanhintum]

@oliverhr In my case stopping containers was not sufficient. Cleaning up the remaining networks and after that stopping and starting docker fixed it. The containers are fine, it's just the lingering networks that keep blocking the ports.

[ghost]

Similar problem here. Container using port 3000/3001 cannot be started: "Bind for 0.0.0.0:3001 failed: port is already allocated". When I remove port 3001, I get the same message for port 3000.
Happened this morning, no problems the days before, no system updates in between.
Stopped/removed container, restarted docker daemon ("sudo service docker restart"), but no result.

Netstat shows this:
tcp6 0 0 :::3000 :::* LISTEN 7039/docker-proxy
tcp6 0 0 :::3001 :::* LISTEN 7031/docker-proxy

I start container with "-p 0.0.0.0:3000:3000 -p 0.0.0.0:3001:3001" options.

I am on a slow internet connection, so removing docker folder and recreating all images is not really an option at the moment.

The only way I can get around this is using a different port: "-p 0.0.0.0:4000:3000 -p 0.0.0.0:4001:3001".

There must be some way to release these ports??

docker version:
Client:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 05:22:43 2016
OS/Arch: linux/amd64

Server:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 05:22:43 2016
OS/Arch: linux/amd64

[samoht]

@oliverhr what's your new diagnostic ID? I can only see the logs related to your initial issue, where you hit an osxfs bug which was fixed after Beta21:

Detected symptom of problem 'invariant-violated' in 269F8ACE-C09C-4FB4-856C-8208C92B0020/20160809-155640.

An invariant within the code of one of the components was not held. Docker needs to be restarted.

May be related to #89

The following log matches:

Aug  2 17:48:53 Mac-Mini Docker[com.docker.driver.amd64-linux][59184] <Notice>: VM: 2016-08-02 22:48:53 +0000 UTC kern.info kernel: br-70b5fe7d8033: port 4(vethe7beb37) entered disabled state
Aug  2 17:49:45 Mac-Mini Docker[com.docker.osxfs][59181] <Notice>: Volume.stop be109ea08951fa4a1b43320d0ee14f1f069c935bfad8b30787c8d0ae78bd7e26 (paths = [/Users/oliver/Projects/dagm8/server/docker/messagesapi//../../../messagesapi])
Aug  2 17:49:45 Mac-Mini Docker[com.docker.osxfs][59181] <Error>: WATCH TREE REMOVAL EXISTENCE INVARIANT VIOLATED
Aug  2 17:49:45 Mac-Mini Docker[com.docker.driver.amd64-linux][59184] <Notice>: VM: 2016-08-02 22:49:45 +0000 UTC kern.info kernel: br-70b5fe7d8033: port 3(veth2d92fed) entered disabled state
Aug  2 17:49:45 Mac-Mini Docker[com.docker.driver.amd64-linux][59184] <Notice>: VM: 2016-08-02 22:49:45 +0000 UTC kern.info kernel: veth3c3979f: renamed from eth0
Aug  2 17:49:45 Mac-Mini Docker[com.docker.driver.amd64-linux][59184] <Notice>: VM: 2016-08-02 22:49:45 +0000 UTC kern.info kernel: br-70b5fe7d8033: port 3(veth2d92fed) entered disabled state

[ZherebcovSergey]

The same issue on OSX 10.12 (build: 16A323) with Docker version: 1.12.1 (2d5b4d9). After rebooting MAC.
I've got one of my services declared in docker-compose.yml (webpack) like that:
ports: - "3000:3000"

docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

docker-compose up
ERROR: for webpack  Cannot start service webpack: driver failed programming external connectivity on endpoint webpack_1 (456a452bfa9bcbc6a296265c23cb57bd11c203430aee4b01b6eb9e9312ead2b0): Bind for 0.0.0.0:3000 failed: port is already allocated
ERROR: Encountered errors while bringing up the project.

logs: /tmp/79ABB4D6-AC97-498B-9C28-BFA79AE6C99E/20161018-121008.tar.gz
[OK] docker-cli
[OK] virtualization kern.hv_support
[OK] menubar
[OK] moby-syslog
[OK] dns
[OK] disk
[OK] system
[OK] app
[OK] osxfs
[OK] virtualization VT-X
[OK] db
[OK] slirp
[OK] logs
[OK] env
[OK] vmnetd
[OK] moby-console
[OK] moby
[OK] driver.amd64-linux

Suggestions of roelvanhintum no result.

I checked it again, here is what i did to solve it:
remove the network listed in docker network ls
stop docker to release the port (after this the port should be unused on your mac)
start docker (the port should not be in use before starting any containers)

Restoring to factory defaults did the trick! But this solution is in the forehead, as the situation is repeated often enough, including other ports which are specified in the docker-compose.yml file

[vortec]

We've been facing the same issue for weeks and it's seriously slowing us down.

Here's a diagnostic ID from one of my co-workers: 8B5B573A-EEA7-447B-8DCD-3BC115AFF40A

We can provide many more if needed :-)

Is there anything else we can do to help you reproduce this issue?

[djs55]

A small update: this could be related to the issue [moby/moby#25981] -- under some circumstances the docker engine keeps a port allocated even though there is no associated container. Hopefully the upcoming 1.12.3 will have some fixes in this area.

On the linked ticket some people reported success with a workaround: (NB I've had to tweak this to remove mention of systemctl)

First get a shell inside the VM:

docker run --rm --net=host --pid=host --privileged -it justincormack/debian nsenter -m -t 1 sh

Next delete the kv store which contains the network information inside the VM:

rm /var/lib/docker/network/files/local-kv.db

Next logout of the shell in the VM and restart the app. With luck this avoids the need to perform a factory reset. Let me know if this helps (or not). I'll let you know when there's an update with some relevant patches in.

[ZherebcovSergey]

ERROR: Cannot start service webpack: driver failed programming external connectivity on endpoint webpack_1 (11daa633feb41ea0cad4086d45b66f526580bc0b1524eb4467962019cb8ffe2c): Bind for 0.0.0.0:3000 failed: port is already allocated
ERROR: Encountered errors while bringing up the project.

First get a shell inside the VM:
docker run --rm --net=host --pid=host --privileged -it justincormack/debian nsenter -m -t 1 sh
Next delete the kv store which contains the network information inside the VM:
rm /var/lib/docker/network/files/local-kv.db

Hooray! Correct! There is no need to throw off to the factory settings ... Thank you!
If you suddenly need a diagnosis...

BA3EF875-A019-4578-9D7C-C0212AA758F2

Docker for Mac: version: 1.12.1 (2d5b4d9)
OS X: version 10.12 (build: 16A323)
logs: /tmp/BA3EF875-A019-4578-9D7C-C0212AA758F2/20161024-150942.tar.gz
[OK] docker-cli
[OK] virtualization kern.hv_support
[OK] menubar
[OK] moby-syslog
[OK] dns
[OK] disk
[OK] system
[OK] app
[OK] osxfs
[OK] virtualization VT-X
[OK] db
[OK] slirp
[OK] logs
[OK] env
[OK] vmnetd
[OK] moby-console
[OK] moby
[OK] driver.amd64-linux

[pgcp]

After putting together pieces from various sources, this is the whole process that worked for me:

1. Get a shell to the VM by connecting via screen:
   screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
2. Hit Enter to see the shell prompt.
3. Run rm /var/lib/docker/network/files/local-kv.db to clear network information, including the allocated ports.
4. Exit the VM shell by killing the screen session with the key sequence: Ctrl-A Ctrl-K. Answery when prompted with Really kill this window [y/n]
5. Restart Docker

[oliverhr]

@pgcp I think that does not work for Docker native for Mac

[oliverhr]

Same problem with Version 1.12.3-beta29.2 (13499)
Diagnostic ID: B9ACA12E-2617-41AD-9A22-9737F2739253

Docker for Mac: version: 1.12.3-beta29.2 (902414d)
OS X: version 10.12.1 (build: 16B2555)
logs: /tmp/B9ACA12E-2617-41AD-9A22-9737F2739253/20161028-014626.tar.gz
[OK]     vmnetd
[OK]     dns
[OK]     driver.amd64-linux
[OK]     virtualization VT-X
[OK]     app
[OK]     moby
[OK]     system
[OK]     moby-syslog
[OK]     db
[OK]     env
[OK]     virtualization kern.hv_support
[OK]     slirp
[OK]     osxfs
[OK]     moby-console
[OK]     logs
[OK]     docker-cli
[OK]     menubar
[OK]     disk

[oliverhr]

Update:

With Version 1.12.3-beta29.2 I was able to fix doing this:

• Force remove all containers from network
  • docker network disconnect -f network-name container-name
• Remove network
  • docker network rm network-name
• Create again the network
  • docker network create network-name
• Delete and Re-Run the containers

Hope this help.

[cloutiertyler]

I am having this issue on a Mac with docker 1.13.0.

I ran diagnostics and uploaded my tar. My diagnostic ID is 44F38709-1B25-4D2F-A458-C8C5D72184E7.

Also @oliverhr I was unable to do step one of your work around. In particular the command worked, but it didn't seem to have any effect on the state.

Edit: Reseting to Default did NOT resolve the issue.

[samoht]

@cloutiertyler can you give a bit more details on the things you were trying to do and the error that you got? I cannot see anything obvious in your logs so any hint would be appreciated.

[freneticmonkey]

I'm running Docker for Mac Version 1.13.0 (15072) on Sierra 10.12.2 and I'm having this problem as well. I've reset docker and have the issue running the following container.

docker run -d --name db -p 8091-8094:8091-8094 -p 11210:11210 couchbase

With the output

docker: Error response from daemon: driver failed programming external connectivity on endpoint db (8a5c3df508167f520b681e9f796cf5303120eca1e4857f253e9e198138da675d): Error starting userland proxy: Bind for 0.0.0.0:8091 failed: port is already allocated.

[samoht]

@cloutiertyler actually your docker logs contain an interesting stack trace:

time="2017-01-10T02:25:30.844552039Z" level=debug msg="Calling GET /v1.22/info"
unexpected fault address 0x420580
fatal error: fault
[signal 0xb code=0x2 addr=0x420580 pc=0x7663ff]

goroutine 156126 [running]:
runtime.throw(0x1e1c070, 0x5)
        /usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc82222fd58 sp=0xc82222fd40
runtime.sigpanic()
        /usr/local/go/src/runtime/sigpanic_unix.go:27 +0x2ab fp=0xc82222fda8 sp=0xc82222fd58
encoding/json.(*decodeState).scanWhile(0x420568, 0x9, 0xc822372168)
        /usr/local/go/src/encoding/json/decode.go:313 +0x4f fp=0xc82222fde0 sp=0xc82222fda8
encoding/json.(*decodeState).object(0x420568, 0x0, 0x0, 0x0)
        /usr/local/go/src/encoding/json/decode.go:695 +0xe2e fp=0xc822230188 sp=0xc82222fde0
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2137 +0x44e

It's probably an upstream bug, I will report that upstream.

[samoht]

@freneticmonkey can you please open a new issue with a diagnostic ID? Thanks!

[samoht]

@cloutiertyler can you share a way to reproduce the error that you had?

[samoht]

The go panic has been reported upstream: moby/moby#30641

[cloutiertyler]

@samoht I apologize for the radio silence, I've been quite busy the last few days. Unfortunately I haven't found a way to reliably reproduce the bug. What I do know is that it happened after I sent an interrupt to forcibly stop my docker containers.

[dukeimg]

I'm having the same issue as @cloutiertyler does. Even reset to defaults doesn't help me.

My diagnostics ID: DC6D6138-681A-4F76-B826-A50AD62CF90D

[hkoelewijn]

I am getting the same error. Docker ps -a does not show the reported port as being allocated, but compose still complains: Diagnostics: 17F28038-B039-469E-9BE5-118D85A94A03/2017-12-11_12-46-21

Windows 10, latest update

[anouarIT]

sudo service docker restart

[mfojtik]

I wonder why this issue is closed when this issue persists in the current "stable" release. After starting containers with host ports, and then stopping and removing (!) them, the ports seems to be still allocated and I need to restart the docker daemon in order to start new container with the same host port.

EDIT: This is also not fixed in the edge version.

To reproduce this, just run container with host port set, then kill that container and delete it. Then run another container with this command:

"cat /proc/net/tcp && ( [ -e /proc/net/tcp6 ] && cat /proc/net/tcp6 || true)")

and you will see that host port still allocated in Docker VM.

[steveetm]

We also still have this problem, sometimes even docker restart is not enough, I have to manually kill VPNKit process.

[Rub21]

This is weird, but I've resolved it restarting my machine and killing the pid

netstat -anvf inet | grep 5432

output 👇

** tcp4 0 0 .5432 . LISTEN 131072 131072 552 0*

sudo kill 552

[djs55]

@mfojtik I believe the issue was closed because the specific bug which could cause the symptom was believed fixed.

If this symptom still occurs, there must be a separate problem in the code somewhere. Could you open a fresh issue with a set of diagnostics and set of repro instructions?

[djs55]

I've locked this issue since it's become too noisy and too difficult to disentangle the different reports.

Please do report separate issues and feel free to link them to this one if you think they're related.

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked