Model durable BMC convergence and quiet fan policy

[briansrls]

Review status

The typed-operation migration is complete and pushed. This remains a draft for review and CI, not because raw fan-control scripts remain: those builders and the generic password-SSH script operation are deleted.

What changed

• Model one fleet-wide BMC valid state: OpenBMC major 3 plus a quiet TEMP_SOC policy that requests 30% through 90C and ramps to 55/70/85/100% at 93/95/97/98C.
• Reuse the observation -> classify -> apply/no-op -> independent readback workflow for BMC firmware and fan convergence.
• Model state lifetime explicitly with named lifecycle boundaries and require capability, commit-barrier, and perturbation/readback evidence for the selected realizer.
• Derive the durability capability from the exact projection identity, lifecycle boundaries, and executable file-publication protocol. It is no longer a separately writable claim.
• Model the literal Linux substrate first: the live srv4 mount is cow / overlay, lowerdir /run/initramfs/ro, upperdir /run/initramfs/rw/cow, with SquashFS and parent JFFS2 backing mounts.
• Replace multiline remote scripts with typed DAG composition for reads, JSON projection, upload, validation, same-directory rename, filesystem synchronization, controller restart, bounded failsafe polling, rollback, and cleanup.
• Keep jq programs at the OpenBMC extdeps realization edge. ObjectMapper and sensor reads use busctl --json=short and typed local jq projection rather than presentation-text parsing.
• Model the OpenSSH transport truth: trailing local arguments are remote-shell interpreted. Typed argv is serialized exactly once with the shared POSIX word renderer; complex jq programs cannot be reinterpreted as shell syntax.
• Keep demand-derived GCP Secret Manager access and password-on-fd-0 SSH. No password or bearer token enters argv or environment.
• Treat installed topology FAN1 through FAN4 and zone0 FailSafe=false as desired state; unpopulated FAN5 is not an aggregate PID input.

Why

srv4 factory policy jumped from 50% at 65C to 75% at 75C, explaining its audible surge near the original 72-77C observation. The first live transaction also exposed a stale FAN5 input that kept the controller in failsafe.

An extant writable file and a currently effective controller do not prove the intended lifetime. Reconciliation now distinguishes literal storage/publication facts, capability across a named boundary, and evidence that the exact realization survived that boundary.

srv3 and srv4 share global firmware intent but not observed state. srv3 remains on OpenBMC 2.07 and refuses version-specific fan mutation. It will not silently update without an approved exact firmware artifact and qualified update realizer.

Live execution

• Previously applied the quiet curve and FAN1-FAN4 topology to srv4; independent readback reported FailSafe=false.
• Monitored 12 samples over six minutes: TEMP_SOC 72-77C, final 73C; guarded DIMMs at most 69C; four healthy tachs.
• Performed a graceful Redfish BMC restart. The BMC was down for 157 seconds while host power and host SSH remained available.
• After restart, OpenBMC remained 3.22.00, config digest was unchanged, controller active, failsafe clear, and convergence took the no-op arm.
• Executed the new typed srv4_bmc_fan_converge entrypoint after this migration. It returned ExitSuccess through the fully observed no-op path, so it made no config write or service restart.
• The typed mutation branch was not forced merely for testing after srv4 was healthy; its publication and refusal behavior is covered by executable witnesses, including missing-sync and incomplete-overlay RED controls.

Remaining boundaries and gaps

• srv4 persistence is qualified only across one graceful BMC control-plane restart. Forced restart, power loss, firmware replacement, and factory reset remain separate unproven boundaries.
• srv3 firmware convergence remains refused until update authorization/artifact/realizer work lands.
• gunbc converge remains HostOs-only; top-level whole-BMC convergence wiring is still explicit follow-up work.
• The correct per-sensor observer currently uses independent SSH operations and is slower than a future connection-reusing transport.

Validation

• 13/13 focused typed fan observer/publication witnesses passed.
• 13/13 generic state-durability witnesses passed.
• 21/21 firmware/thermal/BMC lifecycle witnesses passed.
• 16/16 host-standup spine witnesses passed.
• bmc_fan_converge.dag closure: zero blocking diagnostics.
• host_standup.dag closure: zero blocking diagnostics.
• Live typed srv4 convergence: ExitSuccess no-op.
• git diff --check passed.

[briansrls]

Superseded by the review-driven stacked drafts: #7371 generic durability contract; #7372 capability facts/live receipts; #7373 desired state/reconciliation; #7374 typed operations; #7375 demand-derived access ensure; #7376 live actuator/rollback. The interpreter substring-redaction change is absent from the stack. Exact stack-tip Linux CI is green: https://github.com/gunb-ai/gunbc/actions/runs/30380260457. The original branch remains available for history.