None-scoped instances clean up scoped ones

[piksel]

I am noticing that on a restart of my watchtower with scope none it is cleaning up my other watchtower instances that have defined scopes.

Originally posted by @owine in #1800 (comment)

[piksel]

I cannot reproduce this problem locally:

docker-compose.yaml:

version: '3.7'

services:
  watchtower-foo:
    container_name: watchtower-foo
    image: containrrr/watchtower:latest-dev
    volumes: [ /var/run/docker.sock:/var/run/docker.sock:ro ]
    command: --debug -lpretty --scope foo
    labels:
      com.centurylinklabs.watchtower.scope: foo
  watchtower-none:
    container_name: watchtower-none
    image: containrrr/watchtower:latest-dev
    volumes: [ /var/run/docker.sock:/var/run/docker.sock:ro ]
    command: --debug -lpretty --scope none
    labels:
      com.centurylinklabs.watchtower.scope: none
    depends_on: [ watchtower-foo ]  # To make sure this starts after `foo`

log output:

watchtower-foo   | DEBU[0000] Using scope "foo"
watchtower-foo   | DEBU[0000] Sleeping for a second to ensure the docker api client has been properly initialized.
watchtower-none  | DEBU[0000] Using scope "none"
watchtower-none  | DEBU[0000] Sleeping for a second to ensure the docker api client has been properly initialized.
watchtower-foo   | DEBU[0001] Making sure everything is sane before starting
watchtower-foo   | DEBU[0001] Retrieving running containers
watchtower-foo   | DEBU[0001] There are no additional watchtower containers
watchtower-foo   | DEBU[0001] Watchtower HTTP API skipped.
watchtower-foo   | INFO[0001] Watchtower v1.6.0-18-g097df11
watchtower-foo   | INFO[0001] Using no notifications
watchtower-foo   | INFO[0001] Only checking containers in scope "foo"
watchtower-foo   | INFO[0001] Scheduling first run: 2023-11-12 14:25:14 +0000 UTC
watchtower-foo   | INFO[0001] Note that the first check will be performed in 23 hours, 59 minutes, 59 seconds
watchtower-none  | DEBU[0001] Making sure everything is sane before starting
watchtower-none  | DEBU[0001] Retrieving running containers
watchtower-none  | DEBU[0001] There are no additional watchtower containers
watchtower-none  | DEBU[0001] Watchtower HTTP API skipped.
watchtower-none  | INFO[0001] Watchtower v1.6.0-18-g097df11
watchtower-none  | INFO[0001] Using no notifications
watchtower-none  | INFO[0001] Only checking containers without a scope,
watchtower-none  | INFO[0001] Scheduling first run: 2023-11-12 14:25:14 +0000 UTC
watchtower-none  | INFO[0001] Note that the first check will be performed in 23 hours, 59 minutes, 59 seconds

Both instances filter out each other:

watchtower-foo   | DEBU[0001] There are no additional watchtower containers
watchtower-none  | DEBU[0001] There are no additional watchtower containers

@owine Perhaps the problem was due to the scoped watchtower instances not having labels set?
Considering this to be closed until I can reproduce the issue.

[owine]

Hi I am still experiencing this and will collect additional logs. My containers are deployed by ansible with the proper labels (startup notifications describe scopes as expected) but when I have tasks that stop start my containers, I end up with the none scoped instance cleaned up. I will provide some logs and additional detail for you. Thanks for investigating

[piksel]

The startup notifications only show the label that the watchtower instance is looking for, not what label they have on themselves.
You can see the labels using:

$ docker inspect watchtower | jq '.[].Config.Labels["com.centurylinklabs.watchtower.scope"]'

(where watchtower is the name of the container)

[owine]

Thank you for clarifying. All instances did not have this label set on them. I will redeploy with this set and monitor.

[owine]

Just wanted to close the loop. The labels on the watchtower instances was indeed the issue. Thanks for the point in the right direction and clarifications.

[piksel]

Thanks for the follow-up. I suspected that was the case because it's not the most user friendly configuration. Hopefully we can improve it at some point...