Support for soft reboots

[cgwalters]

This will build on ostreedev/ostree#3420 and we're hoping to land this by at least the Fedora 43 timeframe.

---

TODO:

• update to newer ostree rust APIs
• Expose in status whether we are capable of soft reboots to the staged deployment
• use api to detect soft reboot capable (ostree)
• handle case where kargs are different on the container image than local state /usr/lib/bootc/kargs.d. (bootc)
• Debug prepare-root: boot is kept writable ostreedev/ostree#3365
• Debug Debug: soft-reboot [ 388.410231] systemd[1]: Failed to load new SELinux policy. Continuing with old policy. ostreedev/ostree#3502 (this may be a serious problem!)
• UNDO: bootc upgrade --apply should automatically use soft reboots if capable (detect via new API); though we may need an opt-out flag for unusual cases bootc upgrade --apply --skip-soft-reboot?
• Handle cases (RHEL9) where soft reboot is not supported by systemd (detect missing soft-reboot.target)
• Opt-in and opt-out mechanisms (see below)

Detailed API

We need more sophistication for agent-driven cases that want precise control over the flow of every step in:

• download image (also xref upgrade/switch: Support trickle-fetch #1375 )
• queue for update (xref download without applying (finalization locking) #1320)
• detect when update can or cannot be soft-rebooted (on bare metal, the difference in downtime can be minutes)
• Control whether or not soft reboot is used

Opt-in soft reboots

• systemctl enable bootc-soft-reboot.service
• CLI flag bootc upgrade --queue-soft-reboot? (Problem: tools like flightctl, gnome-software etc. would all need to learn how to opt in to this) - this just calls systemctl enable --runtime bootc-soft-reboot.service

Precedent: rhsm-publish-facts.service which monitors bootc state for modifications

[dustymabe]

To start, probably default to soft reboots for non-apply too.

I was doing some thinking on this. As far as defaulting to soft-reboot it would be interesting to be able to base this on how long the machine takes to boot (we can look at logs for this, like how long did it take for the system to get to default.target or some other target)? My feeling is that if your machine boots quickly you'd almost always prefer hard reboot (at least I would, because that tests every part of process that's needed to bring your machine fully up). In the case where your machine takes a long time to boot, then yeah, give me soft-reboot please.

[jeckersb]

I was doing some thinking on this. As far as defaulting to soft-reboot it would be interesting to be able to base this on how long the machine takes to boot (we can look at logs for this, like how long did it take for the system to get to default.target or some other target)? My feeling is that if your machine boots quickly you'd almost always prefer hard reboot (at least I would, because that tests every part of process that's needed to bring your machine fully up). In the case where your machine takes a long time to boot, then yeah, give me soft-reboot please.

I think the bigger problem is that some hardware just takes forever to initialize itself. So it might be super fast and get from init -> default.target in a second, but it spent 90 seconds in POST sorting itself out before it starts actually booting the operating system.

(Giant disclaimer that I haven't worked with "production" hardware for like ~15 years now, so I am drawing on dated experiences. But back then we definitely had some "big iron" that took forever to boot up. Doing a warm vs. cold boot definitely helped some, but it was still painful)

[dustymabe]

So it might be super fast and get from init -> default.target in a second, but it spent 90 seconds in POST sorting itself out before it starts actually booting the operating system.

Right. I was thinking of a case where it would take a long time to initialize hardware on bringup (i.e. after the kernel started so we could measure from that), but that certainly doesn't account for POST before the kernel.

[jcpunk]

FWIW: my GH200 takes 5 minutes to POST/Self Test.

[cgwalters]

To handle opt-out of soft reboot: Add CLI to undeploy? Then client tool can bootc upgrade [--cached] --disable-soft-reboot ?

Yes this one is important. I suspect we may hit bugs and corner cases. Another option is we expose something in the status (flippable via bootc edit). But the API ultimately has to drive something written as a file...it could look like /run/bootc/soft-reboot-disabled say?

Alternatively...hmm maybe systemctl mask --runtime soft-reboot.target works or could be made to work?

[jmarrero]

Today in a video call, we discussed how systemd checks for /run/nextroot very early when it is called https://github.com/systemd/systemd/blob/bebaaddd41e1aa86662032d9b1e8969e41faf195/src/systemctl/systemctl-logind.c#L99, as Colin found.

This early check means we cannot prepare /run/nextroot (for soft reboot) at that stage, which limits when the preparation logic can be executed. Based on this constraint, we looked at a few design options for how to queue or trigger a soft reboot.

Design Options Considered

Option 1: systemd unit with config file

• A systemd unit writes a configuration file such as /run/bootc/bootc.conf, which includes a key like soft-reboot=1.
• The unit would also trigger bootc internals prepare-soft-reboot in case there is any action to take.
• Then, within the bootc code, we would read this config file during staging or rollback and decide whether to initiate a soft reboot based on its contents.

Option 2: systemd unit without a config file

• Similar to option 1, but instead of writing a config file, the code would check if a systemd service is enabled (for example, bootc-soft-reboot.service).
• The presence or status of the service would determine whether a soft reboot should be triggered.

Option 3: CLI option (preferred for MVP)

• Add a new CLI flag such as --queue-soft-reboot to the bootc rollback, bootc update, and bootc switch commands.
• When used, this flag would directly call the prepare-soft-reboot logic internally.
• This option avoids the need for a systemd unit or config file and is easy to implement and understand.

Option 4: CLI option plus remoting API

• Same as option 3, but also implements the remoting API proposed in issue remoting API #522.
• This would allow users to trigger soft reboots either through the CLI or remotely through the API.
• This provides more flexibility, especially for automated or remote workflows.

Current Direction

We are narrowing consensus in on Option 3 as the MVP.

In future releases, we can revisit the design to support more persistent soft reboot behavior. For example, we could change the internal implementation of the CLI flag to start a systemd unit or write a config file. This would allow users to enable soft reboots by default without needing to pass the flag every time in the future.

[cgwalters]

Agree on option 3 as MVP.