MgsUpdateDriver support for RoT bootloader update

[davepacheco]

I'm not sure yet how the RoT bootloader update process looks different from an SP update. At the very least, this will involve:

• updating the PendingMgsUpdate type to have a variant for the RoT bootloader [reconfigurator] Variant for the RoT bootloader on PendingMgsUpdate #8236
• implementing a new SpComponentUpdateHelper for the RoT bootloader [reconfigurator] Pre-checks and post_update actions for RoT bootloader update #8325
• updating MgsUpdateDriver to create the new update helper when it sees a config for updating the bootloader [reconfigurator] Variant for the RoT bootloader on PendingMgsUpdate #8236
• update reconfigurator-sp-updater to support specifying this, analogous to how it does sp today [reconfigurator] rot-bootloader subcommand for reconfigurator-sp-updater #8330
• test on a raclette

[davepacheco]

I looked into how this works (in Wicket) and found that the update process itself is a bit more work because there are additional post-update steps: we reset the device before activating the new image, make sure that it was correctly verified, then activate the new image, then (I think?) reset the device again. The apply_update() function might need some rework to do these steps in a way that can be interrupted, resumed, entered concurrently by other instances, taken over by other instances, etc.

[davepacheco]

Copying this note from chat.

tl;dr: I think we basically don't need to change the structure of apply_update() after all, though we may want to in the medium term for testability.

I was thinking a bit more about how the existing apply_update() function might need to change to support the RoT bootloader update procedure. If we do the obvious thing and put the multiple resets into the post-update hook... the risks would be:

• if that Nexus crashes and restarts at that point, how does it know to resume the update, and from what step?
• if that Nexus disappears forever at that point, how does another Nexus know to take it over and from what step?
• if another Nexus enters apply_update() at this point, how does it know not to try to upload another image? (for the SP, this is prevented because the SP won't allow another update to start until the reset; but now we have a period where we've done the reset but still don't want another update to start)

The way this works for the SP, the first two are actually the same case -- a restarted Nexus doesn't know or care if it was running one before. It'll just take over an existing one if it's been stuck for too long, just like any other Nexus would. I think this will also work fine for the RoT.

I think it might also work for the RoT bootloader? I'm not positive yet. But I believe if a new update came in while we're in that state (say, having completed the first reset), it would find that the update was not in progress and go through the entire update process again from scratch. This is slightly inefficient but perfectly fine.

That leaves case 3: how do we keep a Nexus that enters apply_update() from starting another update while another Nexus really is in the middle of that post-update (and isn't stuck)? I wonder if we can fix this with a pretty small change: allow precheck() to return a variant like WaitingForOngoingUpdate and have apply_update() loop for up to N minutes while in that state. If we do that I think we might get all the behaviors we want?

• happy path update should work fine
• if a Nexus crashes at any point, permanently or otherwise, it or another Nexus will wind up hitting that timeout (or the similar one that already exists when sending the image) and restarting the whole process
• if another Nexus is concurrently trying to do an update while one is in progress but not stuck, I think this will be fine?
  • if it tries to start an image upload while one is in progress, it will fail
  • it won't try to start an image upload unless the RoT is clearly not in the post-reset phase or it has been there not moving for a few minutes

Some of this is a little racy but I don't think any errors there cause permanent damage -- just maybe an extra lap. We just want to make sure it converges.

I also went down the path of reworking apply_update() and the helper interface to be much more generic so that instead of just a precheck() hook, that hook is more like decide_what_to_do_next() and it returns an enum with generic variants (like "start the upload") and potentially some type-specific variants. Then the body of apply_update() would just call it in a loop until it gets a variant saying it's done. I think this would have a lot of advantages. decide() would basically be a pure function of the current device state so we could test that really exhaustively. And with sim-sp we could test overall update process (the driver function) by sort of single-stepping to await points and injecting either a concurrent update or other sim-sp behaviors (e.g., reset at the "wrong" time).

I think this might be worthwhile in the medium term but it's a fair bit more work and risks breaking SP update, which I think already works and handles edge cases well. So I'm leaning towards doing the simpler thing above for now. Then we can write some regression tests for at least the happy path, and then consider this more ambitious thing if it still seems worthwhile.