Customizable Verbosity, Duplicate "found new" and "removing" alerts.

[mrmachine]

We have many projects that run several services with the same images within their stacks. E.g. Django projects that run separate celery, celerybeat, celeryflower, gunicorn, etc. all from the same image.

When Watchtower detects a new image, it sends a duplicate "Found new ... image..." notification (we're using the slack integration, but Watchtower also sends to stdout logs) for every service, even though they are all using the same image.

Likewise, if we run Watchtower with --cleanup to avoid running out of disk space, we get a duplicate "Removing image ..." notification for every service.

Watchtower should send a single "Found new ... image..." notification (per distinct image), followed by however many "Stopping ..." and "Creating ..." notifications as required, followed by a single "Removing image ..." notification.

An easy fix might be to just debounce or deduplicate the log handler to filter out duplicate events within the last 30 seconds or something?

Update:

Just to be clear, the effect of the current behaviour is to DOUBLE the logging/notification output and reduces the "signal to noise" ratio to 50%. I only care about my services being stopped and started by watchtower.

It might also be even worth sending a new notification when restarted services are marked healthy by Docker (assuming they have health checks).

[simskij]

I'm not sure whether I think this would be an improvement or not compared to the current implementation. What would happen if watchtower finds an image, updates two of the containers and then crashes/panics? Wouldn't this mean we risk losing notifications that would actually be of relevance?

[mrmachine]

Not when we never even read the torrent of duplicate notifications because it's way too noisy.

…

On Sun, 12 May 2019 at 5:40 pm, Simon Aronsson ***@***.***> wrote: I'm not sure whether I think this would be an improvement or not compared to the current implementation. What would happen if watchtower finds an image, updates two of the containers and then crashes/panics? Wouldn't this mean we risk losing notifications that would actually be of relevance? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#255 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAMVYZ5EIN5YUVOYRW3PQ3PU7CYFANCNFSM4GU2CGOQ> .

[simskij]

Feel free to have a stab at it and submit a pull request, however this isn't something I'll prioritize atm, especially not until the new notification layer is in place.

You could also experiment with changing the notification log level from Info, which is the default, to only showing warnings and above.

Thanks 🙏

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[zyberspace]

@simskij This problem is still relevant. Could you maybe disable the stale bot for this issue?

Changing the notification level to warnings doesn't help in our case because we use the notifications to check if a container got updated.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[zyberspace]

See last comment:

@simskij This problem is still relevant. Could you maybe disable the stale bot for this issue?

Changing the notification level to warnings doesn't help in our case because we use the notifications to check if a container got updated.

[piksel]

Shoutrrr does have ServiceRouter.Enqueue and ServiceRouter.Flush that is meant to be used for this purpose. Still needs to rewrite parts of the log/notification handling to make use of them, but the foundation is there at least.

[scottjl]

would love to see this implemented as well. just set up shoutrrr/pushover notifications and they're way too verbose. i'd just like to see the container name, updated, restarted. i don't need the hashes.

[simskij]

This is available using the templating function which will be part of the next release. Unfortunately, it currently lacks documentation, so for now it's for advanced users only. 😅

[mrmachine]

@simskij any tips on how to use it to achieve this goal, since it is undocumented? 😄

[piksel]

@mrmachine @scottjl This is the default template for the "advanced" notifications:

{{- if .Report -}}
  {{- with .Report -}}
    {{- if ( or .Updated .Failed ) -}}
{{len .Scanned}} Scanned, {{len .Updated}} Updated, {{len .Failed}} Failed
      {{- range .Updated}}
- {{.Name}} ({{.ImageName}}): {{.CurrentImageID.ShortID}} updated to {{.LatestImageID.ShortID}}
      {{- end -}}
      {{- range .Fresh}}
- {{.Name}} ({{.ImageName}}): {{.State}}
	  {{- end -}}
	  {{- range .Skipped}}
- {{.Name}} ({{.ImageName}}): {{.State}}: {{.Error}}
	  {{- end -}}
	  {{- range .Failed}}
- {{.Name}} ({{.ImageName}}): {{.State}}: {{.Error}}
	  {{- end -}}
    {{- end -}}
  {{- end -}}
{{- else -}}
  {{range .Entries -}}{{.Message}}{{"\n"}}{{- end -}}
{{- end -}}

You can enable them by passing --notification-report or setting WATCHTOWER_NOTIFICATION_REPORT=true, and customize the template further if you want to decrease the verbosity.

The first block is for the report that is sent after a "session", containing how many containers where updated/failed etc (but only sends it if there are any Updated or Failed, see line 3). The second block (last three lines) is used for other notifications that are logged outside a session and should normally only be emitted on startup.

I will create some examples for different scenarios and put them in the documentation as soon as I get the time.