mtcollins1 boot: select the §28.3 chassis action from observed power state, and read the transition back

[gunbai-bot]

Defect. Run 36023602469 never powered mtcollins1 on. The host was off. mtcollins1_power_action = "cycle" sent ipmitool chassis power cycle. IPMI v2.0 §28.3 says a power cycle should do nothing on a host in S4/S5 (D5h recommended). The AMI MegaRAC exited 0 and did nothing, and the handoff minted BootControlIssued on that exit code alone.

Fix at the authority (§6b).

• extdeps.bmc.ipmi_chassis_control (new) models the §28.3 Chassis Control action as a typed choice: IpmiChassisPowerDown | PowerUp | PowerCycle | HardReset, plus its ipmitool verb. That extdeps type stays broad. The boot handoff's power write takes only the narrower OobPowerAction = OobPowerUp | OobPowerCycle, so power-down, hard reset and an already-on power-up can't reach it on this route (review 5310234917).
  • It is also now the one home of the §28.2 System Power decoder and ChassisPowerObservation, moved out of the fan-observation module so the fan lane and the boot handoff share one decoder.
• oob_boot_handoff reads chassis status before choosing a verb:
  • off → power up
  • on → the host's running-restart verb (mtcollins1: cycle, evidenced on its SEL)
  • unread or undecodable → PowerStateReadFailed; no power command is sent
• Read-back after the command: the handoff polls chassis status up to 60 times, 1 s apart.
  • Power-on must be seen on. A cycle must be seen off and then on.
  • A hard reset can't be witnessed by power state, so it always refuses.
  • Otherwise → PowerCommandAcceptedNoTransition { action, observed_state }. Exit 0 alone is never success.
• Outcome vocabulary, per eager-owl-205: PowerStateReadFailed, PowerCommandRefused, PowerCommandAcceptedNoTransition. The success arm BootControlIssued now carries power: PoweredOn | PowerCycled. It is still a control-plane fact, not proof of what booted.
• All callers migrated, and no string power action remains: mtcollins1_actuate, mtcollins1_boot_authorization, host_reset_subject_roster and host_reset_return_run.
  • The intent hash still renders the verb (cycle).
  • The authorized action is now ipmi.chassis.power.on-or-cycle.
• oob_power_readback_allowance() (182 s) joins mtcollins1_boot_step_timeout_minutes. The generated step timeout goes 57 → 60 min.
  • I edited that value by hand: remote gunbc run refused for lack of a cgroup memory limit. The arithmetic reproduces the old 57 min exactly (3380 s → 57; 3562 s → 60). The generated-artifact drift check on this PR is the confirmation.
• mtcollins1 platform model: the PSU2 AC loss now carries a typed SupplyUncabledOperatorReported cause: the 2026-09-24 operator report that only PSU1 is connected. That report explains the recurring PSU2_Status Power Supply AC lost SEL entries.

Controls (test.claim.machine_intake.oob_power_action_witness, 9 tests, all PASS on a remote claim_batch run):

• off host selects power up
• on host selects cycle
• unread or unknown power state selects nothing
• off host + cycle + no transition refuses
• a cycle that never leaves on refuses
• a cycle seen off then on within the bound passes
• power-up seen on passes, and stuck off refuses
• a hard reset is never witnessed
• PSU2 carries the operator report

fan_policy_standing_witness: 36/36 PASS after the decoder move.

Not verified locally: mtcollins1_boot_run_witness (only change: the covered sum includes the read-back allowance). It was OOM-killed (rc=137) during type-checking under a 12 GiB bound, and the larger runner had no writable cgroup, so it never ran. Leaving it to CI.

Known risk: a cycle whose off window is shorter than one poll (about 1 s sleep plus one chassis-status call) would refuse as not observed. That failure is loud, not silent. §28.3 sets a power-off interval of at least 1 s for a cycle.

Split (approved by eager-owl-205). These go in a follow-up after #12101 lands, because all three edit mtcollins1_boot_run:

• start the boot deadline only once power is established on
• boot parameter 5 before/after, and a post-END-marker poweroff counted as expected
• SMpro reads while power is on

🤖 Generated with Claude Code

Review 5310234917 (head 6985674):

• There's no running-restart parameter any more. The handoff's selection picks the action from the chassis read alone: power up only from an off reading, power cycle only from an on reading. With one supported verb, a caller-supplied restart carried no real choice. It also let the full §28.3 type leak back in.
• The mtcollins1 power_action row and the power_action fields on the reset roster and the boot subject are removed.
• The success record carries only PoweredOn | PowerCycled.
• PowerCommandAcceptedNoTransition now carries progress: never left on, reached off but did not come back on, never reached on, or read-back unavailable. An observed off half is no longer lost.
• The comment on the read-back bound now says that an off window of about 1 s can fall between samples, and that the resulting false refusal is accepted.
• New controls:
  • Compile-time refusals (DeclaredTypeNotInhabited) for IpmiChassisPowerDown, IpmiChassisHardReset and IpmiChassisPowerUp passed as the handoff's action type, plus a clean control.
  • An on host selects power cycle only.
  • On → off → unreadable ends as "reached off, not returned".
• Local claim_batch --wet results:
  • oob_power_action_witness 13/13
  • unit_hold_forged_probe 5/5
  • mtcollins1_boot_run 14/14
• main_wet regen: no generated file changed.

[gunbai-bot]

Regenerated with gunbc run --source-root dag --source-root src/v2 --entry dag/gunbc/instruments/generated_artifact_gate.dag --function main_wet on a gunbc built from this head (f33687f, private target dir). It rewrote every committed generated artifact, including .github/workflows/fleet-converge.yml, and the worktree came out clean. The committed bytes (timeout-minutes: 60) are exactly what the derivation produces, so there was nothing to push.

[gunbai-bot]

Merged main (#12101) and resolved the conflicts:

• ResetSubjectCandidate keeps mtcollins1_boot: WIF-trusted recurring effect via auth pattern selection, with an operator maintenance hold #12101's unit_hold alongside the typed power_action.
• mtcollins1_boot: WIF-trusted recurring effect via auth pattern selection, with an operator maintenance hold #12101 removed the boot-authorization scope and intent hash, and I took main's side there.
• The admit_callers seal now covers both new power functions: oob_power_action_after_confirmed_selection admits oob_boot_handoff, and oob_power_actuate_and_read_back admits the former.
• unit_hold_proof_forged_probe calls the handoff with the typed action. It also calls both sealed power functions from outside, and the forged-probe witness requires both calls to be refused.

Regenerated all generated artifacts with main_wet on a gunbc built from this head. The derived change is in fleet-converge.yml: job 84→87 minutes, step 34→37 minutes.

Run locally (claim_batch --wet, head-built), 64 of 64 PASS:

• oob_power_action_witness: 9/9
• mtcollins1_unit_hold_forged_probe_witness: 5/5
• fan_policy_standing_witness: 36/36
• mtcollins1_boot_run_witness: 14/14

— sent from fierce-cat-441

[briansrls]

SOURCE HOLD at exact head 57e74a1a3c944820510652d0203d5afc43eba38c (the requested f33687fa042ff8caf04fe83d946ca807eb72612b was superseded by a main merge and regenerated workflow).

The incident repair is correctly aimed: the Mt. Collins row is cycle; an OFF pre-read selects on; an ON pre-read selects the row's restart action; unreadable/undecodable state refuses before a power write; and success is no longer derived from the ChassisPowerControl exit status alone. Moving the §28.2 decoder to one extdeps authority, deriving the readback allowance, regenerating the workflow, and typing the PSU2 report are all accepted.

Two source seams remain.

1. running_restart still admits actions that are not a running restart

oob_power_action_selection accepts running_restart: IpmiChassisControlAction, the full four-arm §28.3 type. On an already-ON host it passes that value through unchanged. This makes three invalid paths writable:

• running_restart = IpmiChassisPowerUp: chassis power on can exit 0 on an already-on host; the first readback is still ON, ExpectPowerOn immediately succeeds, and the handoff mints BootControlIssued { power: PoweredOn } although no restart or state transition occurred and the one-shot override may remain unconsumed. This recreates the exact accepted-command/pre-existing-state class the PR is meant to eliminate.
• running_restart = IpmiChassisPowerDown: an OFF readback mints BootControlIssued { power: PoweredOff }, so a boot handoff can report success while deliberately leaving the host off.
• running_restart = IpmiChassisHardReset: the code knows power state cannot witness this action, but sends the reset anyway and only then spends the bounded poll returning failure. An effect known to be unverifiable should refuse before the write.

The PR body says the success carrier is PoweredOn | PowerCycled, but the source also exposes PoweredOff | HostReset.

Required repair: give the handoff a narrower running-restart type (currently only the observed PowerCycle policy is justified), or validate the broad action before any power write and refuse every non-restart/unwitnessable arm. PowerDown, already-on PowerUp, and HardReset must be unable to reach ChassisPowerControl through this boot-handoff route. Add discriminating controls for those three cases. The generic extdeps §28.3 action type may remain broad; the workflow policy type must not be.

2. Exhaustion erases a partially observed cycle

The poll carries ExpectPowerOffThenOn { off_seen }, but PowerReadbackExhausted retains only the last ChassisPowerObservation. Thus a sequence such as ON -> OFF -> unreadable... reaches exhaustion after having established the OFF half, yet becomes PowerCommandAcceptedNoTransition { observed_state: ChassisUnreadable }. The renderers then say no transition was observed. That is false: a transition to OFF was observed; only return to ON was unestablished.

This distinction is operationally load-bearing: "never left ON", "reached OFF but did not return ON", and "readback became unavailable" describe different chassis states and retry evidence. Carry the final expectation/progress into the exhausted outcome (or split the arms) and render it accurately. Add a control that observes OFF and then exhausts without ON, proving that the partial transition survives. Also correct the source comment claiming one-second spacing samples every spec-admitted off window: with a one-second sleep plus a status call budgeted at two seconds, a conforming one-second off interval can be missed; that conservative false refusal is acceptable, but it is not guaranteed coverage.

No objection remains to the current Mt. Collins cycle policy, the off->on selection, decoder move, timeout integration, or the accepted-command-without-readback refusal once these state-space holes are closed.

[briansrls]

HOLD — exact current head 5e0bee9, not the rollup's 6985674.

[P1] The carried decoder blocker is still present in extdeps.bmc.ipmi_chassis_control chassis_power_line. The fold replaces its accumulated answer whenever another System Power line appears. Thus System Power : on\nSystem Power : off becomes an admitted off reading and selects OobPowerUp; reversing the lines selects OobPowerCycle. Duplicate identical lines also pass, and System Power : on : garbage is accepted because only the second split field is inspected. A malformed external response is selecting an effect, not refusing.

Repair at this shared decoder: admit exactly one well-formed System Power field with exactly one recognized on/off value. Zero, multiple matching fields (including identical duplicates), conflicting values and malformed/trailing fields must yield ChassisStatusUndecodable. Preserve the offending evidence. Add REDs for both conflict orders, identical duplicates and malformed field cardinality, plus the ordinary on/off controls. The fan consumer and boot action selector must consume that same result, with no power write on undecodable input.

The narrowed OobPowerAction and admit_callers boundary are otherwise a substantial improvement: only the observed-state selector reaches power-up/cycle, and exhausted readback preserves an observed off half. I did not carry forward the obsolete concern about callers selecting power-down or hard reset; those constructors do not inhabit the new route.

Exact-head workflow 36228771813 is completed/success; green CI does not discriminate the malformed decoder population above. No hardware action was performed. Re-review the repaired head before queueing.

[briansrls]

HOLD — exact head 2fc6622f25270672f6748501ad997b8a5077f11e.

The requested repairs are present and accepted: a second System Power line now reaches ChassisStatusUndecodable; ChassisStatus and ChassisBootParamGet have one bounded operation each with caller-supplied deadline/kill-after; the handoff and fan caller provide those bounds; and the derived workflow budget moves 37→41 minutes.

One carried decoder condition from the latest hold remains. chassis_power_line rejects fewer than two colon-separated fields, but then reads only fields.skip(1).first(). Therefore malformed output such as System Power : on : trailing still decodes as ON and selects OobPowerCycle; System Power : off : trailing decodes as OFF and selects OobPowerUp. A malformed external response is still able to select an effect.

Small required repair: admit only exactly one well-formed System Power key/value field with exactly the literal on or off; extra/trailing fields must produce an undecodable/malformed arm, carrying the offending line or fields for diagnosis. Add REDs for trailing-field ON and OFF, alongside the duplicate-line and ordinary ON/OFF controls. The duplicate-line construction and bounded-read changes need no redesign.

At review time clippy was green; compiler, floor, and emit-build were still in progress. This hold is source-scoped.

[briansrls]

APPROVE-MERGE — exact head bd381207dfb8674acf8b125b43bb27796b6c4415.

The residual decoder blocker from review 5326561042 is closed at the shared §28.2 authority. A line whose trimmed key is System Power now admits a power state only when the colon split has exactly two fields and the sole value is literally on or off. Missing delimiters, trailing fields and unrecognized values cannot select a power action; the malformed arm retains the offending line. Multiple System Power lines remain undecodable rather than first/last-wins.

The new controls distinguish on : trailing, off : trailing, and the no-colon System Power line, while the ordinary on/off, repeated-line, narrowed-action and transition-readback controls remain. The in-place timeout bounds for ChassisStatus and ChassisBootParamGet are retained.

At review time clippy, compiler, and emit-build were completed/success; floor was still in progress. Queue once the exact-head required checks are green.