Watchtower Fails Recreating Containers that Share a Network Stack #1286
Comments
|
Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏 |
|
I'm having the same issue and cannot get watchtower to stop and start linked containers in the correct order. I've tried:
In each case, watchtower correctly identifies that there are linked containers but still shuts the parent down first, then the children, then starts up the children (which error out because of the missing parent), then the parent and then deletes all the dangling images which now includes the non-running child images. Edit: actually I did have some improvement using no. 2 setup above (depends on parent in each child) and using a leading slash in the container name (so label is com.centurylinklabs.watchtower.depends-on: "/gluetun") results in the correct shutdown and startup order. However, the child containers failed to start with the image not found error. Strangely this occurs before watchtower removes all the dangling images so I'm not entirely sure what the problem is. I'm re-trying with WATCHTOWER_CLEANUP set to false and if that doesn't work, I'll just turn off auto-updates for Gluetun. |
|
Same thing here, and also with gluetun. I disabled watchtower on those containers for now. |
|
Same here. |
|
Maybe there is a reference to the old container in the config somewhere. Could you post a |
I'm facing exactly what @ljo123 described. docker inspect: [
{
"Id": "7f89de80221494c2fdca6dca286b15eedb3c9af975a51039304528345b63cc2b",
"Created": "2022-07-08T09:08:08.548100679Z",
"Path": "/go/bin/all-in-one-linux",
"Args": [],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2243640,
"ExitCode": 0,
"Error": "",
"StartedAt": "2022-07-08T09:08:08.878087076Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:5011eb6cadf176aa8ca70812a17499e132b985bc203b4e5d566976943cd1eca0",
"ResolvConfPath": "/var/lib/docker/containers/02cea43ed47cd6d0ea6ebaecfc889cf01623712fe846ca3c3cad1e94c3ed8ce2/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/02cea43ed47cd6d0ea6ebaecfc889cf01623712fe846ca3c3cad1e94c3ed8ce2/hostname",
"HostsPath": "/var/lib/docker/containers/02cea43ed47cd6d0ea6ebaecfc889cf01623712fe846ca3c3cad1e94c3ed8ce2/hosts",
"LogPath": "/var/lib/docker/containers/7f89de80221494c2fdca6dca286b15eedb3c9af975a51039304528345b63cc2b/7f89de80221494c2fdca6dca286b15eedb3c9af975a51039304528345b63cc2b-json.log",
"Name": "/jaeger",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "docker-default",
"ExecIDs": null,
"HostConfig": {
"Binds": [],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "container:02cea43ed47cd6d0ea6ebaecfc889cf01623712fe846ca3c3cad1e94c3ed8ce2",
"PortBindings": {},
"RestartPolicy": {
"Name": "unless-stopped",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": [],
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "host",
"Dns": null,
"DnsOptions": null,
"DnsSearch": null,
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": null,
"DeviceCgroupRules": null,
"DeviceRequests": null,
"KernelMemory": 0,
"KernelMemoryTCP": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": [
"/proc/asound",
"/proc/acpi",
"/proc/kcore",
"/proc/keys",
"/proc/latency_stats",
"/proc/timer_list",
"/proc/timer_stats",
"/proc/sched_debug",
"/proc/scsi",
"/sys/firmware"
],
"ReadonlyPaths": [
"/proc/bus",
"/proc/fs",
"/proc/irq",
"/proc/sys",
"/proc/sysrq-trigger"
]
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/2db8b3f5fb10b872c5bfbceeaa0c0369649ab716cc5fd26829ad928009ad9d2b-init/diff:/var/lib/docker/overlay2/70a39c423ebe007a3cc2e1a2c3fb1c9d60cfc9c3117ac9e3fd50cedc434d1da0/diff:/var/lib/docker/overlay2/35f1782638eaff2f56e39677c3d11f69841decee009d501d6463f6be8873605f/diff:/var/lib/docker/overlay2/28f4b48752d7a3f77cd689c3390463ffb8cad62e8e24904bbde0faf370b4aa28/diff:/var/lib/docker/overlay2/5eedcaf13c27e39123b0277274d15e0d920d152810ae1a959299a0a874e42e1b/diff:/var/lib/docker/overlay2/f98556203bf805f7592608d96e10d84862bf42840852459af978abcdbcd80cfc/diff",
"MergedDir": "/var/lib/docker/overlay2/2db8b3f5fb10b872c5bfbceeaa0c0369649ab716cc5fd26829ad928009ad9d2b/merged",
"UpperDir": "/var/lib/docker/overlay2/2db8b3f5fb10b872c5bfbceeaa0c0369649ab716cc5fd26829ad928009ad9d2b/diff",
"WorkDir": "/var/lib/docker/overlay2/2db8b3f5fb10b872c5bfbceeaa0c0369649ab716cc5fd26829ad928009ad9d2b/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "volume",
"Name": "5a99f2e145e0e426dae0b6a3d56bce224459cd7c809d24408eb6244c5f75e134",
"Source": "/var/lib/docker/volumes/5a99f2e145e0e426dae0b6a3d56bce224459cd7c809d24408eb6244c5f75e134/_data",
"Destination": "/tmp",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "02cea43ed47c",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"14250/tcp": {},
"14268/tcp": {},
"16686/tcp": {},
"5775/udp": {},
"5778/tcp": {},
"6831/udp": {},
"6832/udp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"SAMPLING_STRATEGIES_FILE=/etc/jaeger/sampling_strategies.json"
],
"Cmd": null,
"Image": "jaegertracing/all-in-one",
"Volumes": {
"/tmp": {}
},
"WorkingDir": "",
"Entrypoint": [
"/go/bin/all-in-one-linux"
],
"OnBuild": null,
"Labels": {
"com.centurylinklabs.watchtower.depends-on": "/cloudflared",
"com.docker.compose.config-hash": "c2b318c7497a806ecdf583ead8cc28d591ad6de32393fb9786f20e7aff6bf188",
"com.docker.compose.container-number": "1",
"com.docker.compose.oneoff": "False",
"com.docker.compose.project": "user",
"com.docker.compose.project.config_files": "docker-compose.yml",
"com.docker.compose.project.working_dir": "/home/user",
"com.docker.compose.service": "jaeger",
"com.docker.compose.version": "1.29.2",
"traefik.enable": "True",
"traefik.http.middlewares.jaegerauth.basicauth.users": "secret:secret",
"traefik.http.routers.jaeger.middlewares": "jaegerauth@docker",
"traefik.http.routers.jaeger.rule": "Host(`jaeger.secret.com`)",
"traefik.http.services.jaeger.loadbalancer.server.port": "16686"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {}
}
}
]docker-compose version: "3.8"
services:
cloudflared:
image: cloudflare/cloudflared
container_name: cloudflared
command:
- tunnel
- --url=http://localhost:80
- run
- --token=secret
extra_hosts:
- host.docker.internal:172.177.0.1
restart: unless-stopped
jaeger:
image: jaegertracing/all-in-one
container_name: jaeger
# ports:
# - 16686:16686
network_mode: service:cloudflared
labels:
com.centurylinklabs.watchtower.depends-on: /cloudflared
traefik.enable: true
traefik.http.middlewares.jaegerauth.basicauth.users: secret:secret
traefik.http.routers.jaeger.middlewares: jaegerauth@docker
traefik.http.routers.jaeger.rule: Host(`jaeger.secret.com`)
traefik.http.services.jaeger.loadbalancer.server.port: 16686
restart: unless-stopped
traefik:
image: traefik
container_name: traefik
command:
- --api.dashboard
- --entrypoints.web.address=:80
- --entryPoints.web.forwardedHeaders.trustedIPs=127.0.0.1/32
- --experimental.hub=true
- --global.checkNewVersion=true
- --hub.tls.insecure=true
# - --log.level=DEBUG
- --metrics.prometheus.addrouterslabels=true
- --providers.docker
- --providers.docker.exposedbydefault=false
- --tracing.jaeger=true
# ports:
# - 8080:8080
volumes:
- /run/docker.sock:/var/run/docker.sock:ro
network_mode: service:cloudflared
depends_on:
- jaeger
labels:
com.centurylinklabs.watchtower.depends-on: /cloudflared
traefik.enable: true
traefik.http.middlewares.traefikauth.basicauth.users: secret:secret
traefik.http.routers.traefik.middlewares: traefikauth@docker
traefik.http.routers.traefik.rule: Host(`traefik.secret.com`)
traefik.http.routers.traefik.service: api@internal
traefik.http.services.traefik.loadbalancer.server.port: 8080
restart: unless-stopped
watchtower:
image: containrrr/watchtower
container_name: watchtower
command:
- --cleanup
- --debug
- --include-restarting
- --include-stopped
- --remove-volumes
- --trace
volumes:
- /etc/localtime:/etc/localtime:ro
- /run/docker.sock:/var/run/docker.sock
restart: unless-stopped
networks:
default:
ipam:
config:
- subnet: 172.177.0.0/16 |
|
Yeah, that's exactly what I suspected:
That means that Note: |
|
Any workaround for this short of adding monitor-only to the parent container and manually updating the stack periodically? |
|
The only workaround is to use another networking mode afaik. It would be nice, a PR is welcomed! |
Hello! I have a VPN container and torrent container that share a network stack (torrent container has --network container:PIA). I'm noticing that whenever something happens to the "parent" VPN container that the torrent container errors out. This can come in a few different ways:
In this case, it seems that the torrent container is created before the VPN container:
In this case the containers are recreated but the child container looses the connection to the parent container (presumably because the container hash changes).
I understand that a similar issue used to happen with the --link command but that was resolved back in March. This seems to be the same issue just with the new command (--network container:<> replaced --link).
The text was updated successfully, but these errors were encountered: