Override unattended-upgrade timers to make updates more closely aligned with cron-apt behaviour

[zenmonkeykstop]

Description

By default, update and upgrade unattended-upgrade actions are triggered by systemd timers with a randomized delay of up t o12 hours. This makes updates occur at erratic intervals. The timing and delay is configurable via systemctl edit <timername> or by adding override.conf files under /etc/systemd/system/... - ideally the timers should be set to trigger just before the nightly server reboot, similarly to the cron-apt behaviour in Xenial

User Research Evidence

Zip. Nada.

User Stories

As an admin, I'd like to ensure that reboots and updates are applied during periods with reduced demand.

[eloquence]

Marked as release stretch goal for now, @zenmonkeykstop defer to you if you want to immediately kick it over to 1.9.0.

[eloquence]

We discussed further in standup today. #5852 is expected to fix issues with the upgrade not being applied; if unattended-upgrades now fully works as expected (i.e. we are able to specify its behavior clearly), we'll most likely defer further improvements to 1.9.0 given the complexity of this logic (note that we want to honor sdconfig settings for reboot time as part of any further changes). Not re-milestoning just yet, pending @zenmonkeykstop's final recommendation as RM.

[rmol]

Right now I think there's at least one significant difference with unattended-upgrades. It might disrupt services twice per update, instead of once with cron-apt. An update to securedrop-app-code will see the package updated when the unattended-upgrade script runs, then the system will reboot at the time specified in securedrop-admin sdconfig and configured in /etc/apt/apt.conf.d/80securedrop. The time between those events could be almost a full day, depending on what the admin selected for the daily reboot time. On my staging server the apt-daily-upgrade.timer is configured to run between 6 and 7 a.m. by default, so not long after our default 4 a.m. reboot.

[eloquence]

Can we apply a preferable configuration state (e.g., where the upgrade time is dependent on the configured daily reboot time) in postinst with a future release? If so, the behavior above is IMO acceptable for 1.8.0 (in an emergency, we can always advise admins to manually update and/or reboot, which we might have done even in the cron-apt days).

[rmol]

Right now, as far as I know (would be glad to be apprised of my ignorance) the only record on the servers of the daily reboot time chosen during sdconfig will be in /etc/apt/apt.conf.d/80securedrop. We could obtain the time from that, create systemd timer override files with calendars based on it, and then comment it out, to revert to the default reboot time of "now". If we get that wrong, all the corrections I can think of are going to require assistance from admins.

So, probably, but I wouldn't enjoy having to try it.

[conorsch]

Yes, 80securedrop is that's the file on disk, or will be as of 1.8.0. Since it's live in the apt config, we could use something like:

apt-config dump --format '%v' Unattended-Upgrade::Automatic-Reboot-Time

to retrieve it programmatically. It would make sense to provide a default value, sanely "4:00" since that's default everywhere, but then we should be able to reuse it anywhere else.

[rmol]

It's also available via apt_pkg.config in Python, which I was thinking I'd rather use to construct the calendars. I hesitated to suggest either because we might end up needing recovery logic that has to parse the commented-out line in the file anyway. 🙂

But maybe better than cleverly constructing the calendars around the upgrade+reboot time is just starting that earlier in the hour than the update timer; then we wouldn't need windows around the upgrade to avoid cache lock problems.