update process must work around dendrite#49

[davepacheco]

Currently, Dendrite does not tolerate the switch going away and coming back. More specifically, @rmustacc reported that:

1. Nothing tells dendrite the device is gone.
2. dendrite doesn't survive it going and usually enters maintenance
3. I don't know if we can dynamically add a device into the zone or not via zonecfg
4. We tie the switch zone's existence to the device presence, but my guess is we don't want to tare the zone down if it disappears probably

The net result is that today, if the switch resets (as happens when the SP resets as part of an update), it's necessary to reboot the host OS on the corresponding Scrimlet, too.

Assuming this is not fixable in the R17 timeframe, there are a few obvious approaches to working around this in the update system (and probably others I haven't considered):

1. Have the update process plan Scrimlet SP updates immediately after switch SP updates.
2. Have the update process schedule a Scrimlet reboot (a new kind of operation) immediately after a switch SP update.
3. Have the execution of a switch SP update go and bounce the corresponding Scrimlet.
4. Have the scrimlet itself detect this condition and reboot the box.
5. Power off the scrimlet before doing the switch SP update and power it back on afterwards. Like 1/2 vs. 3, there's a choice here about whether to do this via the planner or executor.

Problem with (1): if the Scrimlet doesn't need any updates, the reboot won't happen.
Downside of (2): if the Scrimlet does need an update, then we're taking an extra bounce for no reason.
Downside of doing: "(1) most of the time, and (2) if it didn't need an update": we almost never test case (2).
Problem with (2) and (3): it's not super obvious how to implement an idempotent "reboot exactly once" operation. It's not impossible -- e.g., it could be "reboot if the system has synchronized its clock and the system-reported boot time is not newer than time T". This does seem doable. It'd probably have to be evaluated by sled agent.
Problem with (1) - (4): all of these assume it's okay to live for a little while in the intermediate state. Is that true? Might this situation prevent us from getting ourselves out of it? (e.g., might a broken dendrite impede the system's ability to carry out one of these options?) Would the customer be impacted in the meantime?

Does any of this change for systems with only one switch?

[davepacheco]

We discussed this on this morning's networking call.

Background facts

Most switch SP updates in practice do not have any real impact here. Only updates that contain a change to any FPGA image.

The behavior can be simulated without such an update by putting the Sidecar into A2 or powering it off via ignition.

What happens today in this situation is:

• the device has been given to the zone when the zone was created
• when the device goes away, the PCIe link goes down
• the Tofino SDE triggers repeated crashes in dendrite and then dendrite goes into maintenance
• the device cannot be torn down because the zone still has a hold on it

Sled agent has code to detect this and deal with it by tearing down the switch zone, and Ry believes it did run at some point in the past, but it's currently conditional on the dev_info_t, which is what can't currently be torn down because the switch zone has a hold on it. So we believe this code cannot ever run today. Instead, this behavior should be conditional on something like: presence of the underlying device or whether the PCIe link is up.

Plan

The basic idea for the short term is to have sled agent identify this condition correctly and tear down the switch zone.

There are two initial action items:

1. We need to identify what OS facilities Sled Agent can use to determine whether to tear down the zone. @rmustacc and @Nieuwejaar were going to look into this.
2. De-risk the existing sled agent mechanism for tearing down the switch zone in this case.
   a. add an internal API for it
   b. add an omdb command to invoke that API
   c. test this a bunch by putting the Sidecar into A2 and/or using ignition to power it off and then using the omdb command to trigger the response that we want

If we find that it's fairly easy for sled agent to identify the correct condition, then we'll implement that behavior. This is ideal because it will cover other cases in which the Sidecar / switch go away.

If we need a stopgap, the update system could potentially invoke that same internal API to trigger a switch zone teardown (and/or reboot) when it knows that it's just updated the switch.

Priority and risk

Recall that we're planning to deliver self-service update in a release that itself must be MUPdated-to. We expect customers can test this by performing an update to a release that in practice will be almost the same release. If we're correct that Sidecar SP updates only trigger this problem when they contain new FPGA images, it's unlikely that this will be a blocker for that initial delivery of self-service upgrade.

Not having fixed this carries the ongoing risk that if we need to deploy a switch SP update with a new FPGA image, then either that will be blocked on fixing this or support will have to be involved in that update.

Choosing not to make it a blocker carries the risk that we're wrong and ordinary SP updates do trigger problems like this. We can mitigate this by testing that.

Longer-term thoughts (Toronto)

Today, it's reasonable for sled agent to tear down the whole switch zone when the Tofino device goes away because that zone can't do anything useful without that device anyway. That may not be true with future switches, where access to the management network could remain available while the main switch is gone or vice versa. In that world, it may be preferable for individual components in the switch zone to better deal with this condition (instead of having sled agent tear down the whole zone). Alternatively, we might split the components in the current switch zone into different zones reflecting the fault domains of the hardware they're talking to.

[davepacheco]

@mkeeter adds that it's possible for a switch SP update to "mess with transceiver state in unpredictable ways [without] bring[ing] down the Tofino switch". That's a different failure mode and wouldn't be detected by the above.

[mkeeter]

More details: an update which changes the front FPGA image will cause that FPGA to reset itself. The front FPGA manages the transceivers on the front of the switch, which are configured by bf_sde. Resetting that FPGA may re-enable ports which had previously been disabled in software, @Aaron-Hartwig can you provide details on how it behaves? My concern is about different parts of the system seeing a different state, e.g. whether bf_sde could have stale information about transceiver state.

It also looks like any SP reset will clear the SP's perspective on which transceivers are disabled, even if the FPGA remains up. @Aaron-Hartwig this may be a Hubris issue to fix; should the SP be reading the FPGA when initially populating its disabled bitmask (or once the front IO board is known to be present)?

[Aaron-Hartwig]

Resetting that FPGA may re-enable ports which had previously been disabled in software, @Aaron-Hartwig can you provide details on how it behaves? My concern is about different parts of the system seeing a different state, e.g. whether bf_sde could have stale information about transceiver state.

An FPGA reset would re-enable the modules in the sense that power would be applied (the reset value of that register) but the module would be held in reset. The notion of "is a module disabled" lives only within the SP. Since the SP tries to talk to the FPGA regularly for thermal monitoring reasons, an uncooperative module would pretty quickly be disabled again.

However, as you state, none of this would happen with the SDE's knowledge. The good thing is that we never write to the memory map (the exception is the page select register). In chat @bnaecker noted that the SDE should handle a change in the power state just fine.

It also looks like any SP reset will clear the SP's perspective on which transceivers are disabled, even if the FPGA remains up. @Aaron-Hartwig this may be a Hubris issue to fix; should the SP be reading the FPGA when initially populating its disabled bitmask (or once the front IO board is known to be present)?

We could have the SP go query the FPGA's enable registers as a means of initially populating that! Otherwise the SP will likely disable the module again in a few seconds since it will try to poll it for sensor data. A query on startup would prevent that up front.