add RANDOM_DELAY to spread backup start in swarm

[NeuerUser]

When all backups in a swarm start at the same time, there might be performance issues. Some cron implementations support the RANDOM_DELAY environment variable, which defines the maximum amount of minutes the cron job could be randomly delayed. This could help to reduce strain on the swarm during backup.

[ThomDietrich]

I support this.
If you have the needed bash experience feel free to propose the change as a PR or here in this ticket.

Side note: Turns out this variable is commonly interpreted as minutes. I'm triggered by this as "random delay" could be in any time unit. The name is horrible. Suggest to go with RANDOM_DELAY_MINUTES_MAX.

[ThomDietrich]

To clarify. I don't think the used go cron implementation supports RANDOM_DELAY. We can simply add the delay in bash, which has no downsides and gives us more control (for whichever reason). @djmaze are you personally interested to add this functionality in go-cron?

[djmaze]

To be precise, I had already been playing around with ways to spread backup times in a swarm cluster. As random delays might still collide, I was hoping to achieve deterministic run times for every node. Turns out this is really hard to do when deploying as a global swarm service.

As I have been talking about here and there, I am hoping to have a central scheduler solution with swarm-cronjob someday, but it does not yet work correctly for global mode services.

Using the random delay might be a good interim solution until we have that.

• It could be added to go-cron but ideally to the upstream cron library where a suitable PR has been closed quite some time ago. Maybe it should be looked at again.
• But, even easier, as @ThomDietrich said adding a sleep with a random delay to the backup script could be enough. I think that would be a good solution until we have something more sophisticated.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.