HEALTH-0: an out-of-band, read-only fleet health check

[briansrls]

HEALTH-0: the first fleet health check. It is out-of-band, read-only, and goal-blind, and it reports what it cannot establish rather than passing over it.

Per the ruling on this work (2026-09-18): a separate protocol beside convergence, never an arm of FleetConvergeRequest; the roster comes from fleet_intent_known_hosts; no convergence lease; the observer takes no expectation; a signal the BMC does not expose is a typed gap and never an in-band fallback.

What it does, per host

1. Reads the BMC's live firmware version and derives its telemetry surface from the board catalog, so a flash changes the route rather than leaving a stale row behind.
2. Reads the event log at the window's opening and closing, and refuses the window if the log wrapped past the opening's newest entry or was cleared in between.
3. Samples fans and temperatures 5 times between those reads.
4. Authenticates only as the gunbc_health ReadOnly account, through a private per-run netrc it shreds on every path out.

Assessment is std.goal_assessment: deviations, unknowns and refusals stay apart, and the fleet is green only if every rostered host is. An unreachable host stays in the denominator as a located refusal.

Modules

• product.host_health — pure host and fleet assessment. Memory counters with an explicit baseline seam (NoPriorBaseline today; HEALTH-1 supplies admitted baselines), temperatures with availability and the BMC's own verdict kept apart from the reading, the event window, fans delegated whole to product.fan_tach_health.
• product.fan_tach fan_tach_window_of — the window's one producer from samples; until now every FanTachWindow in the tree was hand-transcribed.
• extdeps.bmc.redfish_telemetry — Sensor, Thermal and LogEntry decoders for both BMC images. A reading is a value, a null or missing: three facts, never coerced.
• extdeps.bmc.openbmc_message_registry — MemoryECC correctable/uncorrectable, the ECC detail record, Ampere DIMM_HOT, and phosphor-sel-logger threshold text. Asserts diverge, deasserts are recoveries, detail records add nothing to the error they accompany.
• gunbc.fleet_health_observe — the collector. Closed FleetHealthRedfishRead vocabulary (five GETs), surface derived from live firmware, sampled windows, event-window continuity.
• gunbc.fleet_health — goals derived from the roster, the receipt, and the entry fleet_health_check.
• gunbc.bmc_custody_read — the custody read, split out of gunbc.tools.bmc_onboard so the health closure imports nothing that can mint or rotate a credential.
• gunbc.fleet_fan_tach_expectation — the board's secondary tach headers FAN1_1..FAN5_1, which both images publish and no host wires, declared not-expected on all four hosts.

Status (wind-down, 2026-09-20)

The lane is winding down to free the system for v1 performance and the v2 migration. This PR is complete and green as it stands; the one remaining item — a live run whose receipt matches the current code (the receipt below predates the event classification and the realization disclosure) — is carried as the roadmap row fleet-health-0-live-receipt in #11859 rather than held in a session. The live evidence below is real and unchanged in what it found.

Evidence

• Live run, 2026-09-19 13:51Z, all four BMCs, read-only. Fleet INDETERMINATE, which is the honest answer while no temperature limit is cited and memory counters are not exposed out-of-band. Findings: srv1 FAN4 rotating in 1 of 5 samples and FAN5 in 2 of 5; srv2 FAN5 in 2 of 5; srv3 logged 14 corrected memory-error events during the ~30-minute window plus DIMM_HOT on channel 2; srv4 DIMM_HOT on channels 2 and 6. Receipt kept at /tmp/fleet-health-receipt.txt on srv1.
• Witnesses (remote claim_batch): host health assessment 16/16 (the ruling's discriminating cases: 0->0, 4->5 with residue kept, regression/reboot/endpoint-set change cannot construct a delta, no-baseline-with-residue, a baseline for another host refuses, a wrapped log is not an empty delta, one zero fan sample is not a failure, missing sensor unknown vs duplicate refusal, 0 °C vs unavailable vs disabled vs absent, uncited limit never green, one unreachable host blocks the fleet without hiding the others, empty roster refuses); collector 5/5; goal 1/1; registry 2/2; Redfish decoders 5/5 on captured srv1 and srv3 responses.
• No mutation reachable (witness 12): call_reachable_decls from the entry over the nine health modules finds no mutating bare call; the enrolled forbidden-program fixture's mutations are found by identity; the walk demonstrably reaches the collector three levels down. Its limit is measured, not promised: neither declaration census carries a service call's operation identity, so a service-call mutation is invisible — as are the collector's own five GETs. That absence is asserted against the fixture, so the claim fails the day the instrument gains service-call identity, which is the trigger to extend it.

Realization

The route is qualified under the recompute realization (GUNBC_EVAL_MEMO=0, extdeps.realization.eval_memo): the seed's eval-frame memo keys every pure call before admission, and the decoders thread whole response bodies through recursive pure calls, so under the memo each call re-hashes the document — the 1.08 MB srv3 event log decoded in ~430 s of CPU against ~27 s. Qualified 2026-09-19 on the captured responses: complete decoded output byte-identical under both realizations (sha256 a5c00479…), SipHash 79% → 4.6% of samples. The receipt's first line discloses the realization actually in effect. The general defect is its own lane (#11741 and its staged rewires).

Known residuals, named

• Memory error counters are not exposed out-of-band on either image, so every host is unknown on that axis; srv1/srv2 additionally have empty event logs on firmware 2.07. Flashing them to 3.22 is what would expose it.
• No temperature limit is cited yet, so every sensor reports its limit as unestablished. The BMC's own threshold verdict is still carried.
• srv3/srv4 event logs wrap at ~1200 entries and are flooded by fan-threshold flapping, so continuity holds only over short windows.
• A killed run can leave its reader credential file behind: recorded as gunbc.recurring_failure_mode.a_killed_run_leaves_its_credential_file_behind, with ceiling and trigger named.

🤖 Generated with Claude Code

[gunbai-bot]

Transition-admission migration for gunbc#11704. No action until the operator sequences #11704's landing.

gunbc#11704 moves namespace transition admissions out of the tree. Once it lands, any file under dag/gunbc/namespace/transition_admission/ refuses at this PR's own gate, and the admission has to be carried in a commit message on this branch instead. The block below was derived mechanically from this PR's 13 row file(s) at its current head. The only edits: deletion_follow_up, owner_pull_request and their now-unused imports are dropped, because those fields no longer exist. All 13 blocks load through the production fold (carried_admissions_from_messages) with no refusal.

To migrate (paste + delete), after #11704 is on main and merged into this branch:

git rm \
  dag/gunbc/namespace/transition_admission/gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_as_admin_bmc_custody_credential.dag \
  dag/gunbc/namespace/transition_admission/gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_as_admin_custodycredentialabsent.dag \
  dag/gunbc/namespace/transition_admission/gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_as_admin_custodycredentialheld.dag \
  dag/gunbc/namespace/transition_admission/gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_as_admin_custodycredentialunreadable.dag \
  dag/gunbc/namespace/transition_admission/gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_bmc_custody_credential.dag \
  dag/gunbc/namespace/transition_admission/gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_custodycredentialabsent.dag \
  dag/gunbc/namespace/transition_admission/gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_custodycredentialheld.dag \
  dag/gunbc/namespace/transition_admission/gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_custodycredentialunreadable.dag \
  dag/gunbc/namespace/transition_admission/gunbc_tools_bmc_onboard_bmc_probe_credential_phase_bmc_custody_credential.dag \
  dag/gunbc/namespace/transition_admission/gunbc_tools_bmc_onboard_bmc_probe_credential_phase_custodycredentialabsent.dag \
  dag/gunbc/namespace/transition_admission/gunbc_tools_bmc_onboard_bmc_probe_credential_phase_custodycredentialheld.dag \
  dag/gunbc/namespace/transition_admission/gunbc_tools_bmc_onboard_bmc_probe_credential_phase_custodycredentialunreadable.dag \
  dag/gunbc/namespace/transition_admission/test_claim_bmc_health_reader_witness_only_a_404_reads_as_absent_custody_custody_read_refusal_cause.dag
git commit -F msg.txt   # msg.txt = the text below, verbatim

-F keeps the lines exactly as they are. The squash merge carries the message into the queue run, and nothing lands in the tree. If a row is wrong later, a later block with the same stem supersedes it.

msg.txt

Move transition admissions into the commit message (gunbc#11704)

```transition-admission
module gunbc.namespace.transition_admission.gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_as_admin_bmc_custody_credential

import std.types { NonEmptyStr, List }
import std.decl_ref { decl_ref }
import gunbc.compiler_frontend_program_interlock { TargetChanged }
import gunbc.namespace.transition_admission { TransitionAdmission, Binding }

data gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_as_admin_bmc_custody_credential: TransitionAdmission = TransitionAdmission {
  label: "BMC custody read moves to its read-only module gunbc.bmc_custody_read (gunbc#11681)" as NonEmptyStr,
  subject: Binding {
    enclosing: decl_ref("gunbc.tools.bmc_health_reader_converge", "bmc_health_reader_converge_as_admin"),
    spelling: "bmc_custody_credential" as NonEmptyStr,
    expected_candidates: [decl_ref("gunbc.bmc_custody_read", "bmc_custody_credential")],
  },
  disposition: TargetChanged,
}
```

```transition-admission
module gunbc.namespace.transition_admission.gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_as_admin_custodycredentialabsent

import std.types { NonEmptyStr, List }
import std.decl_ref { decl_ref }
import gunbc.compiler_frontend_program_interlock { TargetChanged }
import gunbc.namespace.transition_admission { TransitionAdmission, Binding }

data gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_as_admin_custodycredentialabsent: TransitionAdmission = TransitionAdmission {
  label: "BMC custody read moves to its read-only module gunbc.bmc_custody_read (gunbc#11681)" as NonEmptyStr,
  subject: Binding {
    enclosing: decl_ref("gunbc.tools.bmc_health_reader_converge", "bmc_health_reader_converge_as_admin"),
    spelling: "CustodyCredentialAbsent" as NonEmptyStr,
    expected_candidates: [decl_ref("gunbc.bmc_custody_read", "CustodyCredentialAbsent")],
  },
  disposition: TargetChanged,
}
```

```transition-admission
module gunbc.namespace.transition_admission.gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_as_admin_custodycredentialheld

import std.types { NonEmptyStr, List }
import std.decl_ref { decl_ref }
import gunbc.compiler_frontend_program_interlock { TargetChanged }
import gunbc.namespace.transition_admission { TransitionAdmission, Binding }

data gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_as_admin_custodycredentialheld: TransitionAdmission = TransitionAdmission {
  label: "BMC custody read moves to its read-only module gunbc.bmc_custody_read (gunbc#11681)" as NonEmptyStr,
  subject: Binding {
    enclosing: decl_ref("gunbc.tools.bmc_health_reader_converge", "bmc_health_reader_converge_as_admin"),
    spelling: "CustodyCredentialHeld" as NonEmptyStr,
    expected_candidates: [decl_ref("gunbc.bmc_custody_read", "CustodyCredentialHeld")],
  },
  disposition: TargetChanged,
}
```

```transition-admission
module gunbc.namespace.transition_admission.gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_as_admin_custodycredentialunreadable

import std.types { NonEmptyStr, List }
import std.decl_ref { decl_ref }
import gunbc.compiler_frontend_program_interlock { TargetChanged }
import gunbc.namespace.transition_admission { TransitionAdmission, Binding }

data gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_as_admin_custodycredentialunreadable: TransitionAdmission = TransitionAdmission {
  label: "BMC custody read moves to its read-only module gunbc.bmc_custody_read (gunbc#11681)" as NonEmptyStr,
  subject: Binding {
    enclosing: decl_ref("gunbc.tools.bmc_health_reader_converge", "bmc_health_reader_converge_as_admin"),
    spelling: "CustodyCredentialUnreadable" as NonEmptyStr,
    expected_candidates: [decl_ref("gunbc.bmc_custody_read", "CustodyCredentialUnreadable")],
  },
  disposition: TargetChanged,
}
```

```transition-admission
module gunbc.namespace.transition_admission.gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_bmc_custody_credential

import std.types { NonEmptyStr, List }
import std.decl_ref { decl_ref }
import gunbc.compiler_frontend_program_interlock { TargetChanged }
import gunbc.namespace.transition_admission { TransitionAdmission, Binding }

data gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_bmc_custody_credential: TransitionAdmission = TransitionAdmission {
  label: "BMC custody read moves to its read-only module gunbc.bmc_custody_read (gunbc#11681)" as NonEmptyStr,
  subject: Binding {
    enclosing: decl_ref("gunbc.tools.bmc_health_reader_converge", "bmc_health_reader_converge"),
    spelling: "bmc_custody_credential" as NonEmptyStr,
    expected_candidates: [decl_ref("gunbc.bmc_custody_read", "bmc_custody_credential")],
  },
  disposition: TargetChanged,
}
```

```transition-admission
module gunbc.namespace.transition_admission.gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_custodycredentialabsent

import std.types { NonEmptyStr, List }
import std.decl_ref { decl_ref }
import gunbc.compiler_frontend_program_interlock { TargetChanged }
import gunbc.namespace.transition_admission { TransitionAdmission, Binding }

data gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_custodycredentialabsent: TransitionAdmission = TransitionAdmission {
  label: "BMC custody read moves to its read-only module gunbc.bmc_custody_read (gunbc#11681)" as NonEmptyStr,
  subject: Binding {
    enclosing: decl_ref("gunbc.tools.bmc_health_reader_converge", "bmc_health_reader_converge"),
    spelling: "CustodyCredentialAbsent" as NonEmptyStr,
    expected_candidates: [decl_ref("gunbc.bmc_custody_read", "CustodyCredentialAbsent")],
  },
  disposition: TargetChanged,
}
```

```transition-admission
module gunbc.namespace.transition_admission.gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_custodycredentialheld

import std.types { NonEmptyStr, List }
import std.decl_ref { decl_ref }
import gunbc.compiler_frontend_program_interlock { TargetChanged }
import gunbc.namespace.transition_admission { TransitionAdmission, Binding }

data gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_custodycredentialheld: TransitionAdmission = TransitionAdmission {
  label: "BMC custody read moves to its read-only module gunbc.bmc_custody_read (gunbc#11681)" as NonEmptyStr,
  subject: Binding {
    enclosing: decl_ref("gunbc.tools.bmc_health_reader_converge", "bmc_health_reader_converge"),
    spelling: "CustodyCredentialHeld" as NonEmptyStr,
    expected_candidates: [decl_ref("gunbc.bmc_custody_read", "CustodyCredentialHeld")],
  },
  disposition: TargetChanged,
}
```

```transition-admission
module gunbc.namespace.transition_admission.gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_custodycredentialunreadable

import std.types { NonEmptyStr, List }
import std.decl_ref { decl_ref }
import gunbc.compiler_frontend_program_interlock { TargetChanged }
import gunbc.namespace.transition_admission { TransitionAdmission, Binding }

data gunbc_tools_bmc_health_reader_converge_bmc_health_reader_converge_custodycredentialunreadable: TransitionAdmission = TransitionAdmission {
  label: "BMC custody read moves to its read-only module gunbc.bmc_custody_read (gunbc#11681)" as NonEmptyStr,
  subject: Binding {
    enclosing: decl_ref("gunbc.tools.bmc_health_reader_converge", "bmc_health_reader_converge"),
    spelling: "CustodyCredentialUnreadable" as NonEmptyStr,
    expected_candidates: [decl_ref("gunbc.bmc_custody_read", "CustodyCredentialUnreadable")],
  },
  disposition: TargetChanged,
}
```

```transition-admission
module gunbc.namespace.transition_admission.gunbc_tools_bmc_onboard_bmc_probe_credential_phase_bmc_custody_credential

import std.types { NonEmptyStr, List }
import std.decl_ref { decl_ref }
import gunbc.compiler_frontend_program_interlock { TargetChanged }
import gunbc.namespace.transition_admission { TransitionAdmission, Binding }

data gunbc_tools_bmc_onboard_bmc_probe_credential_phase_bmc_custody_credential: TransitionAdmission = TransitionAdmission {
  label: "BMC custody read moves to its read-only module gunbc.bmc_custody_read (gunbc#11681)" as NonEmptyStr,
  subject: Binding {
    enclosing: decl_ref("gunbc.tools.bmc_onboard", "bmc_probe_credential_phase"),
    spelling: "bmc_custody_credential" as NonEmptyStr,
    expected_candidates: [decl_ref("gunbc.bmc_custody_read", "bmc_custody_credential")],
  },
  disposition: TargetChanged,
}
```

```transition-admission
module gunbc.namespace.transition_admission.gunbc_tools_bmc_onboard_bmc_probe_credential_phase_custodycredentialabsent

import std.types { NonEmptyStr, List }
import std.decl_ref { decl_ref }
import gunbc.compiler_frontend_program_interlock { TargetChanged }
import gunbc.namespace.transition_admission { TransitionAdmission, Binding }

data gunbc_tools_bmc_onboard_bmc_probe_credential_phase_custodycredentialabsent: TransitionAdmission = TransitionAdmission {
  label: "BMC custody read moves to its read-only module gunbc.bmc_custody_read (gunbc#11681)" as NonEmptyStr,
  subject: Binding {
    enclosing: decl_ref("gunbc.tools.bmc_onboard", "bmc_probe_credential_phase"),
    spelling: "CustodyCredentialAbsent" as NonEmptyStr,
    expected_candidates: [decl_ref("gunbc.bmc_custody_read", "CustodyCredentialAbsent")],
  },
  disposition: TargetChanged,
}
```

```transition-admission
module gunbc.namespace.transition_admission.gunbc_tools_bmc_onboard_bmc_probe_credential_phase_custodycredentialheld

import std.types { NonEmptyStr, List }
import std.decl_ref { decl_ref }
import gunbc.compiler_frontend_program_interlock { TargetChanged }
import gunbc.namespace.transition_admission { TransitionAdmission, Binding }

data gunbc_tools_bmc_onboard_bmc_probe_credential_phase_custodycredentialheld: TransitionAdmission = TransitionAdmission {
  label: "BMC custody read moves to its read-only module gunbc.bmc_custody_read (gunbc#11681)" as NonEmptyStr,
  subject: Binding {
    enclosing: decl_ref("gunbc.tools.bmc_onboard", "bmc_probe_credential_phase"),
    spelling: "CustodyCredentialHeld" as NonEmptyStr,
    expected_candidates: [decl_ref("gunbc.bmc_custody_read", "CustodyCredentialHeld")],
  },
  disposition: TargetChanged,
}
```

```transition-admission
module gunbc.namespace.transition_admission.gunbc_tools_bmc_onboard_bmc_probe_credential_phase_custodycredentialunreadable

import std.types { NonEmptyStr, List }
import std.decl_ref { decl_ref }
import gunbc.compiler_frontend_program_interlock { TargetChanged }
import gunbc.namespace.transition_admission { TransitionAdmission, Binding }

data gunbc_tools_bmc_onboard_bmc_probe_credential_phase_custodycredentialunreadable: TransitionAdmission = TransitionAdmission {
  label: "BMC custody read moves to its read-only module gunbc.bmc_custody_read (gunbc#11681)" as NonEmptyStr,
  subject: Binding {
    enclosing: decl_ref("gunbc.tools.bmc_onboard", "bmc_probe_credential_phase"),
    spelling: "CustodyCredentialUnreadable" as NonEmptyStr,
    expected_candidates: [decl_ref("gunbc.bmc_custody_read", "CustodyCredentialUnreadable")],
  },
  disposition: TargetChanged,
}
```

```transition-admission
module gunbc.namespace.transition_admission.test_claim_bmc_health_reader_witness_only_a_404_reads_as_absent_custody_custody_read_refusal_cause

import std.types { NonEmptyStr, List }
import std.decl_ref { decl_ref }
import gunbc.compiler_frontend_program_interlock { TargetChanged }
import gunbc.namespace.transition_admission { TransitionAdmission, Binding }

data test_claim_bmc_health_reader_witness_only_a_404_reads_as_absent_custody_custody_read_refusal_cause: TransitionAdmission = TransitionAdmission {
  label: "BMC custody read moves to its read-only module gunbc.bmc_custody_read (gunbc#11681)" as NonEmptyStr,
  subject: Binding {
    enclosing: decl_ref("test.claim.bmc_health_reader_witness", "only_a_404_reads_as_absent_custody"),
    spelling: "custody_read_refusal_cause" as NonEmptyStr,
    expected_candidates: [decl_ref("gunbc.bmc_custody_read", "custody_read_refusal_cause")],
  },
  disposition: TargetChanged,
}
```

[gunbai-bot]

Fixed in the latest push. Review 69184 was right and the finding was the sharp kind: call_reachable_decls cannot follow a callee whose module is absent from the pool, so a nine-module pool while the route calls into gunbc.auth.netrc_binding and gunbc.auth.access_token_source meant the walk never entered the two modules that actually touch the filesystem — with RecursiveForce sitting on the forbidden roster the whole time. The claim was broader than its executed evidence.

• Both modules are now in health_closure_modules and health_pool_files.
• The positive control is renamed and strengthened: it asserts the walk reaches place_netrc_binding_privately, shred_netrc_binding and ensure_access_token by identity, so a pool that loses one of them fails there rather than quietly narrowing the headline claim. The old control checked only two in-pool functions and could not have caught this.
• The module doc now states the pool rule and cites this review as the reason.

4/4 pass remotely with the wider pool: the closure is still clean of mutating bare calls. The service-call blind spot is unchanged and still pinned as an executed absence.

— sent from swift-newt-222