From 69f6048592949bff70ba0f787bd6e5b78e6f65aa Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Sun, 6 Sep 2026 00:16:17 +0000 Subject: [PATCH 01/11] Record the diskless boot, and correct a capability row that had gone false MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Mt. Collins unit booted an aarch64 Linux kernel to systemd on 2026-09-05, from an image the controller fetched itself over NFS and presented as a virtual CD, on a machine with NO BOOT DRIVE. Nothing served PXE and nothing ran on the controller. That happened and existed nowhere in the corpus. THE MARKER IS THE BOOTED PROGRAM'S OWN OUTPUT, not the media status. A nonzero redirection_status proves a session was established; serial-over-LAN carried GRUB 2.12, "EFI stub: Booting Linux Kernel...", Linux 6.8.0-71-generic on aarch64, squashfs 4.0, an overlayfs upper, and systemd 255.4 as PID 1. WHAT MAKES IT DISKLESS rather than merely booted: the kernel command line is BOOT_IMAGE=/casper/vmlinuz and casper mounts the squashfs read-only under a tmpfs overlay, so the running root is in RAM. The unit reported 261972592 KiB available, which is why a RAM-resident root is unremarkable here. THE IMAGE DIGEST AGREED WITH THE MODEL'S PIN BEFORE ANYTHING BOOTED IT. The file measured 2ee2163c..., which is exactly extdeps.provisioning.ubuntu_install_media noble_numbat_2404_3_live_server_arm64 .content_sha256 and upstream's own cdimage SHA256SUMS. That is what makes the boot attributable to a known artifact rather than to whatever was on the share. CapabilityOemRemoteMedia IS ADDED, AND THE ANNOTATION BESIDE IT HAD GONE FALSE. It read that the REST remote-media surface "answered 401 -- present, unauthenticated-to, never exercised" and grounded "no capability whatever (nothing was mounted)". True when written; false once an authenticated session attached an image and the host booted it. The row is corrected in place rather than having the capability quietly appended, because the sentence was the reason the capability was absent. The distinction that text drew is still right and is what makes the addition admissible rather than optimistic: a 401 grounds a SURFACE and no capability, because a capability is a claim about a write taking effect. What changed is that the write was performed and its effect observed FROM THE HOST -- the same standard CapabilityBootSourceOverride and CapabilityPowerControl were held to. CapabilityVirtualMedia stays absent on a real distinction: this controller serves no Redfish, so what was exercised is the OEM REST route specifically. CapabilityAccountManagement stays absent because the operation that would exercise it was REMOVED, so nothing can perform the write that would ground it. THREE CONTROLLER BEHAVIOURS ARE TYPED, because a modeled caller that does not expect them will be wrong in the dangerous direction: PUT /api/settings/media/general answered HTTP 500 body code 16425 AND THE SHARE RE-MOUNTED -- the image list, empty before, enumerated both images 23 seconds later. POST stop-media answered HTTP 500 body code 1411 and the start-media that followed succeeded. The share listing is a CACHE, not the share: after a new file appeared the controller returned an empty list including for the file it had previously enumerated, until mount_cd was re-asserted. An operation whose exit arms treat nonzero as refusal will refuse a write that worked, which is worse than a plain failure -- the caller then retries or aborts against a controller whose state has already moved. The success signal for these must be a READ-BACK, the same shape oob_boot_handoff already uses for the boot override. THE CONTROLLER CLOCK IS OFFSET, NOT DRIFTING, and the distinction decides the remedy. Two readings sixteen hours apart give the same offset to the second: 4h03m35s behind at 07:27Z and again at 23:57Z. A clock losing time shows a growing gap; an identical gap means the oscillator is fine and the clock was set wrong, so the remedy is ONE write rather than an NTP dependency. It matters because every SEL entry carries this stamp: the power-cycle receipts this lane relies on are four hours in the past, which does not corrupt them but makes them unjoinable to anything observed from the fleet. No clock-set operation is added -- grounding one needs the write performed and read back, and SurfaceBmcClock remains recorded as a successful READ surface only. megarac.Media IS MODELED AND HAS NO CALLER YET. extdeps.bmc.megarac carried paths and contract notes with no transport, so nothing in this repository could invoke the media surface and the first attach was driven by a hand-written shell script -- the unmodeled realization DESIGN §6 names. The service makes it reachable; the workflow that calls it is not in this commit, so image procurement is NOT yet end-to-end through .dag and this commit does not claim it is. Session auth is cookie-and-CSRF rather than netrc, which is why these cannot reuse the redfish.Http shape beside them, and the credential reaches argv in the session call only -- stated in place rather than hidden, with the next-rung repair named. WHAT THIS BOOT DOES NOT ESTABLISH, recorded on the receipt itself: it is a STOCK live-server image with nothing per-host injected and no identity bound; the backing share is srv2, a FLEET PEER, so the colo case is unproven; and UEFI HTTP Boot remains UNMEASURED because every boot in this lane was forced to a device over IPMI, so the PXE client observation is not evidence that HTTP boot is unavailable. Corpus: 27 blocking, none in any file this commit touches. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01D7PLjw5FMYdZ2GWn3FeMpR --- artifacts/bmc/mtcollins1-diskless-boot.json | 73 ++++++++++ dag/extdeps/bmc/megarac.dag | 105 +++++++++++++++ .../mtcollins1_access_observation.dag | 41 ++++-- .../mtcollins1_diskless_boot_receipt.dag | 127 ++++++++++++++++++ 4 files changed, 335 insertions(+), 11 deletions(-) create mode 100644 artifacts/bmc/mtcollins1-diskless-boot.json create mode 100644 dag/gunbc/machine_intake/mtcollins1_diskless_boot_receipt.dag diff --git a/artifacts/bmc/mtcollins1-diskless-boot.json b/artifacts/bmc/mtcollins1-diskless-boot.json new file mode 100644 index 00000000000..2f3f0cbc22b --- /dev/null +++ b/artifacts/bmc/mtcollins1-diskless-boot.json @@ -0,0 +1,73 @@ +{ + "attach": { + "configurations_after": { + "redirection_status": 100, + "session_index": 0 + }, + "endpoint": "POST /api/settings/media/remote/start-media", + "image_index_resolved_by": "reading /api/settings/media/remote/images, not assumed" + }, + "boot_drive": { + "performed_by": "gunbc.machine_intake_mtcollins1_actuate mtcollins1_cdrom_handoff", + "precondition": "boot parameter 5 reset to 'No override' before the run" + }, + "capture_subject": "mtcollins1-diskless-boot", + "claim": "the host booted an aarch64 Linux kernel and reached systemd with a squashfs+overlayfs root, from an image delivered entirely out of band, on a machine with no boot drive", + "delivery_chain": [ + "srv2 NFS export /srv/bmc (ro, scoped to 192.168.1.228)", + "MegaRAC remote-media client on the controller fetches the image itself", + "virtual CD presented to the host", + "host UEFI -> GRUB 2.12 -> EFI stub" + ], + "diskless_evidence": { + "kernel_cmdline": "BOOT_IMAGE=/casper/vmlinuz", + "memory_available_kib": 261972592, + "note": "casper runs the squashfs from RAM with a tmpfs overlay; nothing is written to local storage, and this unit has no boot drive to write to", + "root": "squashfs 4.0 mounted read-only with an overlayfs upper" + }, + "firmware_behaviours_that_a_modeled_caller_must_expect": { + "put_media_general": { + "body_code": 16425, + "consequence": "an operation treating nonzero as refusal will refuse a write that worked", + "http": 500, + "observed": "the write reported failure AND the share re-mounted; images re-enumerated 23s later" + }, + "share_rescan": "the controller stopped enumerating the share after a new file appeared, returning an empty image list; re-asserting mount_cd re-scanned it", + "stop_media": { + "body_code": 1411, + "http": 500, + "observed": "reported failure; the subsequent start-media succeeded" + } + }, + "host_side_acceptance_marker": { + "source": "serial-over-LAN console, not redirection_status", + "trace": [ + "[2;30HGNU GRUB version 2.12", + "[5;3H*Try or Install Ubuntu Server ", + "[0;37;40mEFI stub: Booting Linux Kernel...", + "EFI stub: WARNING: Working around broken SetVirtualAddressMap()", + "EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path", + "EFI stub: Generating empty DTB", + "EFI stub: Exiting boot services...", + "[ 0.000000] Booting Linux on physical CPU 0x0000120000 [0x413fd0c1]", + "[ 0.000000] Linux version 6.8.0-71-generic (buildd@bos03-arm64-036) (aarch64-linux-gnu-gcc-13 (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #71-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 22 16:44:45 UTC 2025 (Ubuntu 6.8.0-71.71-generic 6.8.12)", + "[ 0.000000] Kernel command line: BOOT_IMAGE=/casper/vmlinuz --- console=tty0", + "[ 14.925304] squashfs: version 4.0 (2009/01/31) Phillip Lougher", + "[ 25.748415] overlayfs: \"xino\" feature enabled using 3 upper inode bits.", + "[ 26.406227] systemd[1]: systemd 255.4-1ubuntu8.10 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)", + "[ 26.442927] systemd[1]: Detected architecture arm64." + ], + "why_this_is_the_marker": "a nonzero redirection_status proves a media session; only the booted program's own output proves the artifact executed on the host" + }, + "image": { + "file": "ubuntu-24.04.3-live-server-arm64.iso", + "measured_sha256": "2ee2163c9b901ff5926400e80759088ff3b879982a3956c02100495b489fd555", + "pin_agrees_with": "extdeps.provisioning.ubuntu_install_media noble_numbat_2404_3_live_server_arm64.content_sha256, and upstream cdimage SHA256SUMS" + }, + "not_established": [ + "this is a stock live-server image; nothing per-host is injected and no identity is bound", + "the backing share is srv2, a fleet peer, so this is not yet independent of the fleet", + "UEFI HTTP Boot remains unmeasured; every boot so far was forced to a device over IPMI" + ], + "observed_at_utc": "2026-09-05T22:55Z" +} diff --git a/dag/extdeps/bmc/megarac.dag b/dag/extdeps/bmc/megarac.dag index 847043160bc..e57f800ea11 100644 --- a/dag/extdeps/bmc/megarac.dag +++ b/dag/extdeps/bmc/megarac.dag @@ -156,3 +156,108 @@ fn megarac_image_redirection_wire(r: MegaRacImageRedirectionRequired) -> NonEmpt MegaRacImageRedirectionOn => "1" } } + +// THE OPERATIONS ABOVE ARE A DESCRIPTION; THESE ARE THE OPERATIONS. megarac_api_operations records +// paths and contract notes and carries no transport, so nothing in this repository could invoke the +// MegaRAC media surface — the first attach against a Mt. Collins unit was therefore driven by a +// hand-written shell script, which is exactly the unmodeled realization DESIGN §6 names. These rows +// make that surface reachable from a workflow. +// +// SESSION AUTH IS COOKIE-AND-HEADER, NOT netrc, which is why these cannot reuse the redfish.Http +// shape beside them. POST /api/session returns a CSRFToken in its body and sets a session cookie; +// every subsequent request must present BOTH the cookie jar and an X-CSRFTOKEN header. The jar path +// is a parameter rather than a fixed location so an attempt owns its own jar and two concurrent +// attempts cannot share one session. +// +// THE CREDENTIAL IS FORM-ENCODED IN THE SESSION CALL AND NOWHERE ELSE. It reaches argv there, which +// is a real weakness and is stated rather than hidden: curl offers no netrc equivalent for a +// form-encoded POST body, and the alternative spellings (--data @file) put the credential in a file +// whose lifetime this operation does not control. Every operation AFTER the session carries no +// credential at all — the cookie is the bearer — so the exposure is one call, not the whole flow. +// The next-rung repair is a transport that reads a form body from a file the caller materializes +// and removes, which is the same missing mechanism the removed ipmi UserSetPassword operation +// needs. +// +// StartMedia's json_body MUST CARRY image_redirection AND ITS ABSENCE IS OPAQUE. The contract note +// on the operation row above records that omitting it makes the firmware refuse with codes +// 13410/13460, recovered from the firmware's own UI bundle rather than from documentation. The body +// is a caller parameter rather than a baked literal because image_index is resolved by READING the +// images listing, never assumed, and megarac_image_redirection_wire is the authority for the field +// itself. +service megarac.Media { + + operation OpenSession { + input { bmc_host: NonEmptyStr, username: NonEmptyStr, password: NonEmptyStr, cookie_jar: NonEmptyStr } + output { body: String from "stdout", success: Bool from "exit_success" } + transport shell { + argv: ["curl", "--fail-with-body", "-sS", "-k", "-c", "{cookie_jar}", "-X", "POST", "-d", "username={username}", "-d", "password={password}", "https://{bmc_host}/api/session"] + } + exit { + 0 => Unit + nonzero => String "megarac session POST failed" + } + } + + operation GetMediaGeneral { + input { bmc_host: NonEmptyStr, cookie_jar: NonEmptyStr, csrf_token: NonEmptyStr } + output { body: String from "stdout", success: Bool from "exit_success" } + readonly + transport shell { + argv: ["curl", "--fail-with-body", "-sS", "-k", "-b", "{cookie_jar}", "-H", "X-CSRFTOKEN: {csrf_token}", "https://{bmc_host}/api/settings/media/general"] + } + exit { + 0 => Unit + nonzero => String "megarac media general GET failed" + } + } + + operation GetRemoteImages { + input { bmc_host: NonEmptyStr, cookie_jar: NonEmptyStr, csrf_token: NonEmptyStr } + output { body: String from "stdout", success: Bool from "exit_success" } + readonly + transport shell { + argv: ["curl", "--fail-with-body", "-sS", "-k", "-b", "{cookie_jar}", "-H", "X-CSRFTOKEN: {csrf_token}", "https://{bmc_host}/api/settings/media/remote/images"] + } + exit { + 0 => Unit + nonzero => String "megarac remote images GET failed" + } + } + + operation GetRemoteConfigurations { + input { bmc_host: NonEmptyStr, cookie_jar: NonEmptyStr, csrf_token: NonEmptyStr } + output { body: String from "stdout", success: Bool from "exit_success" } + readonly + transport shell { + argv: ["curl", "--fail-with-body", "-sS", "-k", "-b", "{cookie_jar}", "-H", "X-CSRFTOKEN: {csrf_token}", "https://{bmc_host}/api/settings/media/remote/configurations"] + } + exit { + 0 => Unit + nonzero => String "megarac remote configurations GET failed" + } + } + + operation StartMedia { + input { bmc_host: NonEmptyStr, cookie_jar: NonEmptyStr, csrf_token: NonEmptyStr, json_body: NonEmptyStr } + output { body: String from "stdout", success: Bool from "exit_success" } + transport shell { + argv: ["curl", "--fail-with-body", "-sS", "-k", "-b", "{cookie_jar}", "-H", "X-CSRFTOKEN: {csrf_token}", "-H", "Content-Type: application/json", "-X", "POST", "-d", "{json_body}", "https://{bmc_host}/api/settings/media/remote/start-media"] + } + exit { + 0 => Unit + nonzero => String "megarac start-media POST failed" + } + } + + operation StopMedia { + input { bmc_host: NonEmptyStr, cookie_jar: NonEmptyStr, csrf_token: NonEmptyStr, json_body: NonEmptyStr } + output { body: String from "stdout", success: Bool from "exit_success" } + transport shell { + argv: ["curl", "--fail-with-body", "-sS", "-k", "-b", "{cookie_jar}", "-H", "X-CSRFTOKEN: {csrf_token}", "-H", "Content-Type: application/json", "-X", "POST", "-d", "{json_body}", "https://{bmc_host}/api/settings/media/remote/stop-media"] + } + exit { + 0 => Unit + nonzero => String "megarac stop-media POST failed" + } + } +} diff --git a/dag/gunbc/machine_intake/mtcollins1_access_observation.dag b/dag/gunbc/machine_intake/mtcollins1_access_observation.dag index e307947080c..7d7c79e768d 100644 --- a/dag/gunbc/machine_intake/mtcollins1_access_observation.dag +++ b/dag/gunbc/machine_intake/mtcollins1_access_observation.dag @@ -8,6 +8,7 @@ import extdeps.bmc.capability { BmcFirmwareReleaseCapabilityRow, BmcFirmwareReleaseIdentity, CapabilityBootSourceOverride, + CapabilityOemRemoteMedia, CapabilityPowerControl, } import extdeps.bmc.access_profile { @@ -204,21 +205,39 @@ data mtcollins1_access_standing: BmcAccessObservationStanding = BmcAccessObserve // `Power Unit — Power cycle | Asserted` and `System Boot Initiated — Initiated by power up`. The // controller's own event log is a second witness to the effect, not a restatement of the request. // -// WHAT IS DELIBERATELY ABSENT, because absence here is the whole point of the row. CapabilityVirtualMedia, -// CapabilityOemRemoteMedia, CapabilityNbdProxyVirtualMedia, CapabilityAccountManagement, -// CapabilityFirmwareUpdate, CapabilityBiosAttributeConfig and CapabilitySerialConsole are NOT members. -// The REST remote-media surface answered 401 — present, unauthenticated-to, never exercised — and a -// 401 grounds SurfaceOemRestRemoteMedia in discovered_surfaces (the surface exists) while grounding -// no capability whatever (nothing was mounted). That split is the model earning its keep: the delivery -// solver reads CAPABILITIES to choose a transport, so an unexercised media capability authored here -// would send it down a path nothing has shown this controller can walk. SerialConsole is the sharpest -// case — SOL was in fact activated and streamed Ampere checkpoint codes, so the capability is very -// probably real, and it is still omitted because that session was driven by hand from a terminal and -// not through any modeled route. A capability this row asserts must be one the AUTOMATION can perform. +// CapabilityOemRemoteMedia — ADDED 2026-09-05T22:55Z, and this row previously said the opposite. +// The earlier text read that the REST remote-media surface "answered 401 — present, +// unauthenticated-to, never exercised" and grounded "no capability whatever (nothing was +// mounted)". That was true when written and is now false: an authenticated session attached +// ubuntu-24.04.3-live-server-arm64.iso from the NFS share, configurations read back +// redirection_status 100 with session_index 0, and the host BOOTED IT — GRUB 2.12, an EFI-stub +// aarch64 kernel, squashfs and overlayfs, systemd as PID 1. Evidence: +// artifacts/bmc/mtcollins1-diskless-boot.json. +// +// The distinction the old text drew is still the right one and is what makes this addition +// admissible rather than optimistic: a 401 grounds the SURFACE and no capability, because a +// capability is a claim about a WRITE taking effect. What changed is that the write was performed +// and its effect observed from the host, which is the same standard CapabilityBootSourceOverride +// and CapabilityPowerControl were held to above. +// +// WHAT IS STILL DELIBERATELY ABSENT, because absence here is the whole point of the row. +// CapabilityVirtualMedia, CapabilityNbdProxyVirtualMedia, CapabilityAccountManagement, +// CapabilityFirmwareUpdate, CapabilityBiosAttributeConfig and CapabilitySerialConsole are NOT +// members. CapabilityVirtualMedia is the near miss and stays out on a real distinction: this +// controller serves no Redfish, so what was exercised is the OEM REST remote-media route +// specifically, and the generic virtual-media capability names a surface this unit does not have. +// CapabilityAccountManagement is absent because the operation that would exercise it was REMOVED +// (extdeps.bmc.ipmi records why): rotation has no modeled operation at all, so nothing can perform +// the write that would ground it. SerialConsole is the sharpest case — SOL was activated and +// streamed both Ampere checkpoint codes and the entire Ubuntu boot, so the capability is very +// probably real, and it is still omitted because that session was driven by hand from a terminal +// and not through any modeled route. A capability this row asserts must be one the AUTOMATION can +// perform. data mtcollins1_capability_row: BmcFirmwareReleaseCapabilityRow = BmcFirmwareReleaseCapabilityRow { firmware: mtcollins1_firmware, capabilities: [ CapabilityBootSourceOverride, CapabilityPowerControl, + CapabilityOemRemoteMedia, ], } diff --git a/dag/gunbc/machine_intake/mtcollins1_diskless_boot_receipt.dag b/dag/gunbc/machine_intake/mtcollins1_diskless_boot_receipt.dag new file mode 100644 index 00000000000..ce3b8ce8031 --- /dev/null +++ b/dag/gunbc/machine_intake/mtcollins1_diskless_boot_receipt.dag @@ -0,0 +1,127 @@ +module gunbc.machine_intake_mtcollins1_diskless_boot_receipt + +import std.types { Bool, EpochMs, Int, List, NonEmptyStr } +import std.content_hash { ContentHash, Sha256Digest, Sha256Hash } + +// THE HOST BOOTED DISKLESS FROM AN OUT-OF-BAND IMAGE, and this row is what the corpus holds about +// it. On 2026-09-05T22:55Z the Foxconn Mt. Collins unit — which has NO BOOT DRIVE — ran an aarch64 +// Linux kernel to systemd, from an image the controller fetched itself over NFS and presented as a +// virtual CD. Nothing on the network served PXE and nothing ran on the controller. +// +// THE MARKER IS THE BOOTED PROGRAM'S OWN OUTPUT. A nonzero redirection_status proves a media +// session was established; only the host console proves the artifact executed. Serial-over-LAN +// carried GRUB 2.12, "EFI stub: Booting Linux Kernel...", "Linux version 6.8.0-71-generic" on +// aarch64, "squashfs: version 4.0", an overlayfs upper, and systemd 255.4 as PID 1. +// +// WHAT MAKES IT DISKLESS RATHER THAN MERELY BOOTED: the kernel command line is +// BOOT_IMAGE=/casper/vmlinuz, and casper mounts the squashfs read-only under a tmpfs overlay, so +// the running root is in RAM. The unit reported 261972592 KiB available, which is why a +// RAM-resident root is unremarkable here rather than a constraint. +data mtcollins1_diskless_boot_capture_digest: ContentHash = Sha256Hash(Sha256Digest { + hex: "cefdf6c063544a08c937770f9450ce0d0ee15a06bab2bf35af9f5eaadcda7681", +}) + +data mtcollins1_diskless_boot_artifact_path: NonEmptyStr = "artifacts/bmc/mtcollins1-diskless-boot.json" + +data mtcollins1_diskless_boot_byte_count: Int = 4361 + +data mtcollins1_diskless_boot_observed_at: EpochMs = 1788648900000 + +// THE IMAGE DIGEST AGREED WITH THE MODEL'S PIN BEFORE ANYTHING BOOTED IT, which is the one check +// that makes the boot attributable to a known artifact rather than to whatever happened to be on +// the share. The file measured 2ee2163c... and that is the value carried by +// extdeps.provisioning.ubuntu_install_media noble_numbat_2404_3_live_server_arm64.content_sha256, +// and by upstream's own cdimage SHA256SUMS for ubuntu-24.04.3-live-server-arm64.iso. +data mtcollins1_booted_image_sha256: NonEmptyStr = "2ee2163c9b901ff5926400e80759088ff3b879982a3956c02100495b489fd555" + +// THREE CONTROLLER BEHAVIOURS A MODELED CALLER MUST EXPECT, recorded here because they were +// measured and because a modeled operation that does not expect them will be WRONG in the +// dangerous direction. +// +// A WRITE CAN REPORT FAILURE AND TAKE EFFECT. PUT /api/settings/media/general answered HTTP 500 +// with body code 16425 — and the share re-mounted anyway; the image list, empty before, enumerated +// both images 23 seconds later. POST stop-media answered HTTP 500 with body code 1411, and the +// start-media that followed succeeded. An operation whose `exit` arms treat nonzero as refusal will +// refuse a write that worked, which is worse than a plain failure: the caller retries or aborts +// against a controller whose state has already moved. +// +// THAT IS WHY THE SUCCESS SIGNAL FOR THESE OPERATIONS MUST BE A READ-BACK, not an exit status — +// the same shape gunbc.machine_intake_oob_boot_handoff already uses for the boot override, where a +// set command's own return could not distinguish acceptance from indifference. The read-back here +// is the images listing and the configurations listing. +// +// AND THE SHARE LISTING IS NOT STABLE ACROSS FILE ADDITIONS. After a new ISO appeared in the +// exported directory the controller returned an EMPTY image list, including for the file it had +// previously enumerated. Re-asserting mount_cd re-scanned it. A caller that reads an empty listing +// as "the share has no images" will conclude the wrong thing; the listing is a cache, not the +// share. +type ControllerWriteReportingAnomaly { + operation: NonEmptyStr + http_status: Int + body_code: Int + effect_observed: NonEmptyStr +} + +data mtcollins1_controller_write_anomalies: List = [ + ControllerWriteReportingAnomaly { + operation: "PUT /api/settings/media/general", + http_status: 500, + body_code: 16425, + effect_observed: "the share re-mounted and both images enumerated 23s later, having been empty before", + }, + ControllerWriteReportingAnomaly { + operation: "POST /api/settings/media/remote/stop-media", + http_status: 500, + body_code: 1411, + effect_observed: "the subsequent start-media succeeded and attached the requested image", + }, +] + +// THE CONTROLLER CLOCK IS OFFSET, NOT DRIFTING, and the distinction decides the remedy. Two +// readings sixteen hours apart give the SAME offset to the second: +// +// 2026-09-05T07:27:22Z real, controller reported 03:23:47 -> 4h03m35s behind +// 2026-09-05T23:57:54Z real, controller reported 19:54:19 -> 4h03m35s behind +// +// A clock losing time would show a growing gap; an identical gap across sixteen hours means the +// oscillator is fine and the clock was simply set wrong. So the remedy is ONE write (`sel time +// set`), not an NTP client, and the converged property is a bounded difference from real time +// rather than a sync relationship with a server. +// +// WHY IT MATTERS BEYOND TIDINESS: every SEL entry is stamped with this clock. The power-cycle +// receipts this lane already relies on — "Power Unit — Power cycle | Asserted" and "System Boot +// Initiated — Initiated by power up" — carry timestamps four hours in the past, so correlating a +// controller event with anything observed from the fleet requires knowing the offset. An +// uncorrected clock does not corrupt the events; it makes them unjoinable to everything else. +// +// THE WRITE IS NOT MODELED. extdeps.bmc.ipmi has no clock operation, and this row does not add one: +// grounding it needs the write performed and read back, the same standard every other capability in +// gunbc.machine_intake_mtcollins1_access_observation was held to. SurfaceBmcClock is recorded there +// as a successful READ surface only, which remains accurate. +type ControllerClockSkewObservation { + observed_at: EpochMs + controller_behind_seconds: Int + rate_correct: Bool +} + +data mtcollins1_controller_clock_skew: ControllerClockSkewObservation = ControllerClockSkewObservation { + observed_at: 1788652674000, + controller_behind_seconds: 14615, + rate_correct: true, +} + +// WHAT THIS BOOT DOES NOT ESTABLISH, stated here so the receipt cannot be read as more than it is. +// +// It is a STOCK live-server image. Nothing per-host is injected, no identity is bound, and the +// running system is the installer's live environment rather than a converged host. Customising the +// root needs a squashfs builder, which extdeps.tools.squashfs_tools deliberately does not model. +// +// The backing share is srv2, a FLEET PEER, so this is not yet a boot path independent of the +// fleet's own operating systems. Moving that endpoint off-fleet is a change of location and not of +// mechanism, and until it happens the colo case is unproven. +// +// UEFI HTTP Boot remains UNMEASURED. Every boot in this lane was forced to a device over IPMI, so +// the firmware has never been given the chance to advertise HTTPClient. The PXE client observation +// in gunbc.machine_intake_mtcollins1_netboot_client_observation is therefore not evidence that HTTP +// boot is unavailable — only that PXE is what a forced PXE boot produced. +data mtcollins1_diskless_boot_receipt_scope: NonEmptyStr = "stock live-server image, fleet-peer backing share, UEFI HTTP Boot unmeasured" From 1070027ef3acf5165eae8d0059cdf2a2de8252a5 Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Sun, 6 Sep 2026 00:48:41 +0000 Subject: [PATCH 02/11] Attach remote media through the model, not a shell script MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous commit landed service megarac.Media and said plainly that it had no caller, so image procurement was not end-to-end through .dag. This is the caller. EXECUTED AGAINST THE UNIT, from a genuinely detached start rather than a no-op. The image was stopped first (configurations read image_name "" with redirection_status 0), then: ◐ megarac.Media.OpenSession 208ms ◐ megarac.Media.GetRemoteImages 148ms ◐ megarac.Media.StartMedia 104ms ◐ megarac.Media.GetRemoteConfigurations 166ms and an independent read afterwards showed the ISO attached. The shell script that performed the first attach is now redundant. THREE THINGS THE MODELED VERSION DOES THAT THE SCRIPT DID NOT, each one a defect the script had rather than a refinement: THE CSRF TOKEN COMES FROM THE PARSED DOCUMENT. The script matched a regex over the response body — a second reading of JSON beside the one this repository already owns. extdeps.languages.json.parse is the single recognition authority, and a document it cannot parse is now UNREADABLE rather than partially believed: MegaRacSessionTokenUnreadable carries the body that defeated it. THE IMAGE INDEX IS RESOLVED BY NAME, NEVER ASSUMED. start-media takes an image_index and the controller assigns indices by its own enumeration of the share; the same file was index 0 on one scan and index 1 after another image appeared. A baked index attaches whatever now occupies that slot, which on a share holding a memtest ISO and an installer ISO is a SILENT WRONG BOOT rather than a failure. SUCCESS IS A READ-BACK, NEVER AN EXIT STATUS, and this firmware forces it. The previous commit recorded two measured cases of it answering HTTP 500 while the write took effect. An operation treating non-zero as refusal would refuse a write that worked, leaving the caller to retry or abort against a controller whose state had already moved. The confirmation here is the configurations listing naming the image, read separately from the start call. AN EMPTY LISTING IS ITS OWN OUTCOME because it has its own remedy. MegaRacImageListingEmpty means the controller's share cache went stale — measured on this unit, which stopped enumerating even a file it had listed before, until mount_cd was re-asserted. Reading it as "the share has no images" sends an operator to inspect an export that is fine. The listing is a cache, not the share, and the two are different states. MegaRacAttachUnconfirmed is kept distinct from MegaRacStartRefused for the same reason: the mount daemon is asynchronous, so start-accepted-but-not-yet-listed is PENDING rather than failed. It is reported rather than slept through, because a caller that polls until something looks successful has no deadline. THE CREDENTIAL ASYMMETRY IS NAMED RATHER THAN HIDDEN. OpenSession is form-encoded and curl offers no netrc equivalent for a form body, so the password reaches argv for exactly one call; every subsequent operation carries no credential because the cookie is the bearer. The ipmi operations beside this take a password FILE for the same threat and can, because -f exists there. Wrapping the difference away would make one call's exposure look like none. Corpus: 27 blocking, unchanged from the previous head, none in the new files. WHAT IS STILL NOT MODELED: the ISO fetch itself. The image was downloaded to the share by hand-run curl, and nothing in this repository yet places a verified artifact on a controller-reachable share. The digest was checked against extdeps.provisioning.ubuntu_install_media noble_numbat_2404_3_live_server_arm64 .content_sha256 before anything booted it, but that check was also performed by hand. Attach is now modeled; procurement is half. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01D7PLjw5FMYdZ2GWn3FeMpR --- .../machine_intake/megarac_media_attach.dag | 238 ++++++++++++++++++ .../mtcollins1_media_attach.dag | 75 ++++++ 2 files changed, 313 insertions(+) create mode 100644 dag/gunbc/machine_intake/megarac_media_attach.dag create mode 100644 dag/gunbc/machine_intake/mtcollins1_media_attach.dag diff --git a/dag/gunbc/machine_intake/megarac_media_attach.dag b/dag/gunbc/machine_intake/megarac_media_attach.dag new file mode 100644 index 00000000000..8670b659257 --- /dev/null +++ b/dag/gunbc/machine_intake/megarac_media_attach.dag @@ -0,0 +1,238 @@ +module gunbc.machine_intake_megarac_media_attach + +import std.types { Bool, Int, NonEmptyStr, String } +import extdeps.bmc.megarac +import extdeps.languages.json.parse { + JsonArray, + JsonDocumentParse, + JsonDocumentParsed, + JsonDocumentUnreadable, + JsonMemberFound, + JsonNumber, + JsonString, + JsonValue, + json_object_unique_member, + parse_json_document, +} + +// ATTACHING A REMOTE IMAGE, AS A MODELED OPERATION RATHER THAN A SHELL SCRIPT. extdeps.bmc.megarac +// described this API in prose for months and carried no transport, so the first attach against a +// Mt. Collins unit was driven by hand-written curl — the unmodeled realization DESIGN §6 names, and +// the reason the netboot lane could not claim its image was procured through .dag. The service rows +// made the surface reachable; this module is the caller that makes it reached. +// +// EVERY STEP READS BACK, AND THAT IS FORCED BY THE CONTROLLER'S OWN BEHAVIOUR rather than chosen for +// rigour. gunbc.machine_intake_mtcollins1_diskless_boot_receipt records two measured cases of this +// firmware answering HTTP 500 WHILE THE WRITE TOOK EFFECT: PUT media/general (code 16425) with the +// share re-mounting anyway, and stop-media (code 1411) with the following attach succeeding. An +// operation that treated a non-zero exit as refusal would refuse a write that worked, and the caller +// would then retry or abort against a controller whose state had already moved. So success here is +// never an exit status; it is the configurations listing naming the image. +// +// THE IMAGE INDEX IS RESOLVED, NEVER ASSUMED. start-media takes an image_index, and the controller +// assigns indices by its own enumeration of the share — the same file was index 0 on one scan and +// index 1 after another image appeared. A baked index attaches whatever now occupies that slot, +// which on a share holding a memtest ISO and an installer ISO is a silent wrong-boot rather than a +// failure. The index is read from the images listing by matching the image NAME. +// +// AND AN EMPTY LISTING IS NOT AN EMPTY SHARE. After a new file appeared the controller returned an +// empty image list including for a file it had previously enumerated, until mount_cd was +// re-asserted. The listing is a cache. This module refuses on an empty listing rather than +// concluding the image is absent, because those are different states with different remedies and +// only one of them is the share's fault. +type MegaRacAttachOutcome + = MegaRacAttached { image_name: NonEmptyStr, image_index: Int } + | MegaRacSessionRefused { detail: String } + | MegaRacSessionTokenUnreadable { detail: String } + | MegaRacImagesUnreadable { detail: String } + | MegaRacImageListingEmpty + | MegaRacImageNotEnumerated { image_name: NonEmptyStr } + | MegaRacStartRefused { detail: String } + | MegaRacAttachUnconfirmed { image_name: NonEmptyStr, configurations: String } + +// THE CSRF TOKEN IS READ FROM THE PARSED DOCUMENT, not scraped. The shell version matched a regex +// over the response body, which is a second reading of JSON beside the one this repository already +// owns; extdeps.languages.json.parse is the single recognition authority and a document it cannot +// parse is unreadable rather than partially believed. +fn json_string_member(doc: JsonValue, key: String) -> String? { + match json_object_unique_member(v: doc, key: key) { + JsonMemberFound { value: v } => + match v { + JsonString { value: s } => Present { value: s } + _ => none + } + _ => none + } +} + +fn json_number_member(doc: JsonValue, key: String) -> String? { + match json_object_unique_member(v: doc, key: key) { + JsonMemberFound { value: v } => + match v { + JsonNumber { lexeme: n } => Present { value: n } + _ => none + } + _ => none + } +} + +// THE INDEX OF A NAMED IMAGE, folded over the parsed array. Returns the FIRST match and the fold +// keeps it, because a controller enumerating one name twice is a state this module has never +// observed and would rather report as the index it acted on than silently prefer the last. +fn image_index_named(elements: List, image_name: String) -> String? { + fold(elements, init: none, f: (acc, element) => + match acc { + Present { value: _ } => acc + Absent => + match json_string_member(doc: element, key: "image_name") { + Absent => none + Present { value: found } => + if found == image_name { + json_number_member(doc: element, key: "image_index") + } else { + none + } + } + } + ) +} + +fn images_array_elements(doc: JsonValue) -> List { + match doc { + JsonArray { elements: es } => es + _ => [] + } +} + +// THE START-MEDIA BODY, rendered here rather than by the caller so image_redirection cannot be +// omitted by a caller who does not know it is required. extdeps.bmc.megarac records that omitting +// it makes the firmware refuse with opaque codes 13410/13460, recovered from the firmware's own UI +// bundle rather than from documentation — exactly the kind of fact that is lost when a body is +// assembled at a call site. +fn start_media_body(image_name: NonEmptyStr, image_index: String) -> NonEmptyStr { + join([ + "{\"image_name\":\"", image_name as String, + "\",\"image_type\":1,\"image_index\":", image_index, + ",\"image_redirection\":1}" + ], "") as NonEmptyStr +} + +// THE EFFECT SEQUENCE. Session, list, resolve, start, confirm — and the confirmation is a separate +// read rather than the start call's own answer, for the reason given at the top of this module. +// +// THE CREDENTIAL IS A VALUE HERE AND A FILE PATH EVERYWHERE ELSE IN THIS LANE, and that asymmetry is +// forced rather than chosen: OpenSession is form-encoded, and curl offers no netrc equivalent for a +// form body, so the password reaches argv for exactly one call. Every subsequent operation carries +// no credential at all because the cookie is the bearer. The ipmi operations beside this take a +// password FILE for the same threat and can, because -f exists there. Naming the asymmetry is the +// point; hiding it behind a wrapper would make one call's exposure look like none. +func megarac_attach_remote_image( + bmc_host: NonEmptyStr, + username: NonEmptyStr, + password: NonEmptyStr, + cookie_jar: NonEmptyStr, + image_name: NonEmptyStr, +) -> MegaRacAttachOutcome + uses net: std.resources.Network +{ + session = megarac.Media.OpenSession( + bmc_host: bmc_host, + username: username, + password: password, + cookie_jar: cookie_jar, + ) + if session.success == false { + return MegaRacSessionRefused { detail: session.body } + } + + let session_doc = match parse_json_document(s: session.body) { + JsonDocumentParsed { value: v } => some_json_value(v) + JsonDocumentUnreadable { gap: _ } => none + } + let csrf = match session_doc { + Absent => none + Present { value: doc } => json_string_member(doc: doc, key: "CSRFToken") + } + + match csrf { + Absent => MegaRacSessionTokenUnreadable { detail: session.body } + Present { value: token } => + megarac_attach_with_token( + bmc_host: bmc_host, + cookie_jar: cookie_jar, + csrf_token: token as NonEmptyStr, + image_name: image_name, + ) + } +} + +func megarac_attach_with_token( + bmc_host: NonEmptyStr, + cookie_jar: NonEmptyStr, + csrf_token: NonEmptyStr, + image_name: NonEmptyStr, +) -> MegaRacAttachOutcome + uses net: std.resources.Network +{ + images = megarac.Media.GetRemoteImages( + bmc_host: bmc_host, + cookie_jar: cookie_jar, + csrf_token: csrf_token, + ) + if images.success == false { + return MegaRacImagesUnreadable { detail: images.body } + } + + let elements = match parse_json_document(s: images.body) { + JsonDocumentParsed { value: v } => images_array_elements(doc: v) + JsonDocumentUnreadable { gap: _ } => [] + } + if (elements |> count) == 0 { + return MegaRacImageListingEmpty + } + + match image_index_named(elements: elements, image_name: image_name as String) { + Absent => MegaRacImageNotEnumerated { image_name: image_name } + Present { value: idx } => + megarac_start_and_confirm( + bmc_host: bmc_host, + cookie_jar: cookie_jar, + csrf_token: csrf_token, + image_name: image_name, + image_index: idx, + ) + } +} + +func megarac_start_and_confirm( + bmc_host: NonEmptyStr, + cookie_jar: NonEmptyStr, + csrf_token: NonEmptyStr, + image_name: NonEmptyStr, + image_index: String, +) -> MegaRacAttachOutcome + uses net: std.resources.Network +{ + start = megarac.Media.StartMedia( + bmc_host: bmc_host, + cookie_jar: cookie_jar, + csrf_token: csrf_token, + json_body: start_media_body(image_name: image_name, image_index: image_index), + ) + + configurations = megarac.Media.GetRemoteConfigurations( + bmc_host: bmc_host, + cookie_jar: cookie_jar, + csrf_token: csrf_token, + ) + + return if configurations.body |> contains(image_name as String) { + MegaRacAttached { image_name: image_name, image_index: 0 } + } else { + if start.success == false { + MegaRacStartRefused { detail: start.body } + } else { + MegaRacAttachUnconfirmed { image_name: image_name, configurations: configurations.body } + } + } +} diff --git a/dag/gunbc/machine_intake/mtcollins1_media_attach.dag b/dag/gunbc/machine_intake/mtcollins1_media_attach.dag new file mode 100644 index 00000000000..b4519312e56 --- /dev/null +++ b/dag/gunbc/machine_intake/mtcollins1_media_attach.dag @@ -0,0 +1,75 @@ +module gunbc.machine_intake_mtcollins1_media_attach + +import std.types { NonEmptyStr, String } +import std.process { ExitSuccess, ProcessExit, exit_failure } +import gunbc.machine_intake_mtcollins1_access_observation { mtcollins1_endpoint } +import gunbc.machine_intake_megarac_media_attach { + MegaRacAttachOutcome, + MegaRacAttachUnconfirmed, + MegaRacAttached, + MegaRacImageListingEmpty, + MegaRacImageNotEnumerated, + MegaRacImagesUnreadable, + MegaRacSessionRefused, + MegaRacSessionTokenUnreadable, + MegaRacStartRefused, + megarac_attach_remote_image, +} + +// THE POLICY LAYER FOR ONE HOST'S MEDIA ATTACH, separate from the operation for the same reason the +// boot-handoff wrappers are: which image this unit should carry is a workflow decision, and the +// operation stays reusable by the next controller without carrying anyone's choices. +data mtcollins1_diskless_image_name: NonEmptyStr = "ubuntu-24.04.3-live-server-arm64.iso" + +data mtcollins1_media_cookie_jar: NonEmptyStr = "/tmp/gunbc-mtcollins1-megarac-jar" + +data mtcollins1_media_username: NonEmptyStr = "admin" + +// EACH OUTCOME RENDERS ITS OWN REASON because they have different remedies, and two of them are +// easy to confuse in a way that sends an operator to the wrong place. +// +// MegaRacImageListingEmpty means the CONTROLLER'S CACHE went stale — measured on this unit after a +// new file appeared in the export, when it stopped enumerating even a file it had listed before. +// The remedy is to re-assert mount_cd, not to look for the missing file. Reading it as "the share +// is empty" sends someone to inspect an export that is fine. +// +// MegaRacAttachUnconfirmed is the one state where waiting is right: start-media was accepted and +// the configurations listing does not name the image yet. The controller's mount daemon is +// asynchronous, so this is pending rather than failed — but it is reported rather than slept +// through, because a caller that polls until something looks successful has no deadline. +fn mtcollins1_render_attach(outcome: MegaRacAttachOutcome) -> ProcessExit { + match outcome { + MegaRacAttached { image_name: _, image_index: _ } => ExitSuccess + MegaRacSessionRefused { detail: d } => + exit_failure(reason: concat("mtcollins1 media attach: session refused: ", d)) + MegaRacSessionTokenUnreadable { detail: d } => + exit_failure(reason: concat("mtcollins1 media attach: session succeeded but no CSRFToken could be read from its document: ", d)) + MegaRacImagesUnreadable { detail: d } => + exit_failure(reason: concat("mtcollins1 media attach: images listing unreadable: ", d)) + MegaRacImageListingEmpty => + exit_failure(reason: "mtcollins1 media attach: the controller enumerated NO images; that listing is a cache and has gone stale, so re-assert mount_cd rather than concluding the share is empty") + MegaRacImageNotEnumerated { image_name: n } => + exit_failure(reason: concat("mtcollins1 media attach: the controller enumerated images but not '", n as String, "'")) + MegaRacStartRefused { detail: d } => + exit_failure(reason: concat("mtcollins1 media attach: start-media refused AND the configurations listing does not name the image: ", d)) + MegaRacAttachUnconfirmed { image_name: n, configurations: c } => + exit_failure(reason: concat("mtcollins1 media attach: start-media was accepted but configurations does not yet name '", n as String, "'; the mount daemon is asynchronous so this may be pending: ", c)) + } +} + +func mtcollins1_attach_diskless_image(password_raw: String) -> ProcessExit + uses net: std.resources.Network +{ + if password_raw == "" { + return exit_failure(reason: "mtcollins1 media attach: no controller password supplied; refusing before any request") + } + return mtcollins1_render_attach( + outcome: megarac_attach_remote_image( + bmc_host: mtcollins1_endpoint.host, + username: mtcollins1_media_username, + password: password_raw as NonEmptyStr, + cookie_jar: mtcollins1_media_cookie_jar, + image_name: mtcollins1_diskless_image_name, + ) + ) +} From 571a42e10ff6d12f2d0f2e67efc165855c935708 Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Sun, 6 Sep 2026 02:22:22 +0000 Subject: [PATCH 03/11] Fetch the boot image through the model, and refuse to publish an unverified one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Attach and boot were modeled; procurement was not. The ISO the diskless boot ran from was downloaded by hand-run curl and checked by hand-run sha256sum, which left the one step that decides WHAT the machine boots outside the model. THE ORDER IS THE GUARANTEE, and it is why this is not simply "curl then check". The fetch lands in a STAGING path outside the export, the digest is measured there, and only an agreeing artifact is moved into the published path. Fetching straight into the export and checking afterwards leaves an interval in which a truncated or substituted image is mountable — and the controller re-scans that share on its OWN schedule, not this program's, so the interval is not bounded by anything here. Measurement and publication are reachable only through the arm where the fetch succeeded, and publication only through the arm where the digest agreed; both are separate funcs rather than statements after an `if`, the same shape that keeps the boot handoff's power cycle behind a confirmed read-back. A MISMATCH LEAVES THE STAGED FILE IN PLACE, deliberately. The bytes that disagreed are the evidence for the refusal, and deleting them turns a diagnosable result into a bare assertion. What matters is that they never reach the published path, not that they cease to exist. EXECUTED, WITH THE DISCRIMINATING RED FIRST: wrong pin 3.0 GB fetched, digest measured 2ee2163c..., refused against a deliberately impossible pin, and the published path was NOT CREATED — verified from the filesystem, not from the program's own report. The control publishes to a path the controller does not read, so a defect in the wall could not have put an unverified artifact where the unit would boot it. correct pin fetch succeeded, digest measured, digest AGREED, and publication failed: /srv/bmc is root-owned and the run is unprivileged. The existing published ISO is untouched — same size, same timestamp — so a failed publish did no collateral damage. THAT SECOND RESULT IS WHY BootImagePublishFailed IS ITS OWN ARM. "The artifact disagreed with its pin" and "the artifact was fine and I could not write it where it belongs" have different owners and different remedies; collapsing them would send someone to re-examine an image that was never the problem. The refusal names both paths so the remedy is readable from the message. THE PIN IS CONSUMED, NOT MINTED. extdeps.provisioning.ubuntu_install_media already carries content_sha256 for this artifact, cited to cdimage's own SHA256SUMS, and the value here is that figure. When the artifact row grows a locator field the policy should read both from it rather than carrying either; a second copy of a digest is a second authority for one fact, and the copy is always the one that rots. WHAT REMAINS: the export is root-owned, so the publish step needs an ownership or privilege decision that is an operational fact rather than a modeling one. Until it is made, this operation fetches and verifies through the model and refuses at the last step, which is a better state than the previous one — where the same work happened with no refusal available at all. Corpus: 27 blocking, unchanged, none in any file this commit touches. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01D7PLjw5FMYdZ2GWn3FeMpR --- dag/gunbc/machine_intake/boot_image_fetch.dag | 142 ++++++++++++++++++ .../mtcollins1_boot_image_fetch.dag | 82 ++++++++++ 2 files changed, 224 insertions(+) create mode 100644 dag/gunbc/machine_intake/boot_image_fetch.dag create mode 100644 dag/gunbc/machine_intake/mtcollins1_boot_image_fetch.dag diff --git a/dag/gunbc/machine_intake/boot_image_fetch.dag b/dag/gunbc/machine_intake/boot_image_fetch.dag new file mode 100644 index 00000000000..17e8fb3d80c --- /dev/null +++ b/dag/gunbc/machine_intake/boot_image_fetch.dag @@ -0,0 +1,142 @@ +module gunbc.machine_intake_boot_image_fetch + +import std.types { Bool, FilePath, NonEmptyStr, String } +import extdeps.shell +import extdeps.tools.curl { curl_download_command } +import gunbc.command_runner { run_shell_command } +import extdeps.exec.command { CommandTransport, LocalExec } +import std.process { ExitSuccess, ProcessExit } +import extdeps.tools.sha256sum { + Sha256FileDigest, + Sha256FileDigestUnavailable, + sha256sum_file_digest_via_shell, +} + +// PLACING A VERIFIED BOOT IMAGE WHERE A CONTROLLER CAN FETCH IT, as a modeled operation. The +// diskless boot this lane already achieved used an ISO downloaded to the share by hand-run curl and +// checked by a hand-run sha256sum, so procurement was the one step of the chain still outside the +// model while attach and boot were inside it. +// +// THE DIGEST DECIDES ADMISSION, AND IT IS MEASURED BEFORE THE FILE IS PUBLISHED. The order matters +// and is the whole point: an artifact whose bytes disagree with the pin must never occupy the path +// the controller reads. Fetching straight into the export and checking afterwards leaves a window +// where a truncated or substituted image is mountable, and on a share the controller re-scans on its +// own schedule that window is not bounded by this program's lifetime. So the fetch lands in a +// staging path, the digest is taken there, and only an agreeing artifact is moved into place. +// +// THE PIN IS NOT MINTED HERE. extdeps.provisioning.ubuntu_install_media already carries +// content_sha256 for each artifact row, cited to upstream's own SHA256SUMS, and this module consumes +// that value rather than restating it. A second copy of a digest is a second authority for one fact, +// and the one that rots is always the copy. +type BootImageFetchOutcome + = BootImageAdmitted { path: NonEmptyStr, digest: NonEmptyStr } + | BootImageFetchFailed { url: NonEmptyStr } + | BootImageDigestUnreadable { path: NonEmptyStr, reason: String } + | BootImageDigestMismatchRefused { expected: NonEmptyStr, observed: NonEmptyStr } + | BootImagePublishFailed { staged: NonEmptyStr, destination: NonEmptyStr } + +// WHETHER A MEASURED DIGEST AGREES WITH THE PIN. Separated from the effect sequence so the judgment +// is nameable, and returning the OBSERVED value on disagreement rather than a Bool: a refusal that +// does not say what it saw sends the reader back to the machine to find out. +fn boot_image_digest_agrees(observed: Sha256FileDigest, expected: NonEmptyStr) -> Bool { + match observed { + Sha256FileDigestUnavailable { path: _, reason: _ } => false + Sha256FileDigest { digest: d } => (d.hex as String) == (expected as String) + } +} + +// THE OBSERVED HEX, OR A SELF-NAMING PLACEHOLDER WHEN THERE ISN'T ONE. A refusal carries what was +// seen so the reader does not have to go back to the machine, and an unreadable digest is reported +// as unreadable rather than as a mismatch against an empty string -- those are different failures +// with different owners: one is the artifact, the other is the tool or the path. +fn boot_image_observed_hex(observed: Sha256FileDigest) -> NonEmptyStr { + match observed { + Sha256FileDigestUnavailable { path: _, reason: _ } => "digest-unreadable" + Sha256FileDigest { digest: d } => d.hex + } +} + +// THE SEQUENCE, AND ITS ORDER IS THE GUARANTEE. Fetch to a staging path, measure there, and move +// into the published path ONLY on agreement. Fetching straight into the export and checking +// afterwards would leave an interval in which a truncated or substituted image is mountable, and the +// controller re-scans that share on its own schedule rather than this program's, so the interval is +// not bounded by anything here. +// +// A MISMATCH LEAVES THE STAGED FILE WHERE IT IS. It is deliberately not deleted: the bytes that +// disagreed are the evidence for why the fetch was refused, and removing them turns a diagnosable +// refusal into a bare assertion. What matters for safety is that they never reach the published +// path, not that they cease to exist. +func fetch_verified_boot_image( + url: NonEmptyStr, + staging_path: NonEmptyStr, + published_path: NonEmptyStr, + expected_sha256: NonEmptyStr, +) -> BootImageFetchOutcome + uses net: std.resources.Network +{ + let fetched = run_shell_command( + command: curl_download_command(url: url as String, destination: staging_path as String), + transport: LocalExec, + ) + return match fetched { + ExitSuccess => + verify_and_publish_boot_image( + staging_path: staging_path, + published_path: published_path, + expected_sha256: expected_sha256, + ) + _ => BootImageFetchFailed { url: url } + } +} + +// MEASUREMENT AND PUBLICATION ARE REACHABLE ONLY THROUGH A SUCCESSFUL FETCH, which is why this is a +// separate func rather than statements following an `if`. The same shape keeps the boot handoff's +// power cycle behind a confirmed read-back: a refusal cannot be reordered past it and nothing can +// fall through to it. +func verify_and_publish_boot_image( + staging_path: NonEmptyStr, + published_path: NonEmptyStr, + expected_sha256: NonEmptyStr, +) -> BootImageFetchOutcome + uses net: std.resources.Network +{ + let measured = sha256sum_file_digest_via_shell(path: staging_path) + return match measured { + Sha256FileDigestUnavailable { path: p, reason: r } => + BootImageDigestUnreadable { path: p, reason: r } + Sha256FileDigest { digest: _ } => + if boot_image_digest_agrees(observed: measured, expected: expected_sha256) { + publish_verified_boot_image( + staging_path: staging_path, + published_path: published_path, + digest: boot_image_observed_hex(observed: measured), + ) + } else { + BootImageDigestMismatchRefused { + expected: expected_sha256, + observed: boot_image_observed_hex(observed: measured), + } + } + } +} + +// THE PUBLISH IS ITS OWN FUNC so the move is reachable only through the arm where the digest already +// agreed, in the same shape the boot handoff uses to keep its power cycle behind a confirmed +// read-back. A future edit cannot fall through to it. +func publish_verified_boot_image( + staging_path: NonEmptyStr, + published_path: NonEmptyStr, + digest: NonEmptyStr, +) -> BootImageFetchOutcome + uses net: std.resources.Network +{ + moved = shell.Move.File( + source: staging_path as FilePath, + destination: published_path as FilePath, + ) + return if moved.success { + BootImageAdmitted { path: published_path, digest: digest } + } else { + BootImagePublishFailed { staged: staging_path, destination: published_path } + } +} diff --git a/dag/gunbc/machine_intake/mtcollins1_boot_image_fetch.dag b/dag/gunbc/machine_intake/mtcollins1_boot_image_fetch.dag new file mode 100644 index 00000000000..ab7bdf988ce --- /dev/null +++ b/dag/gunbc/machine_intake/mtcollins1_boot_image_fetch.dag @@ -0,0 +1,82 @@ +module gunbc.machine_intake_mtcollins1_boot_image_fetch + +import std.types { NonEmptyStr, String } +import std.process { ExitSuccess, ProcessExit, exit_failure } +import gunbc.machine_intake_boot_image_fetch { + BootImageAdmitted, + BootImageDigestMismatchRefused, + BootImageDigestUnreadable, + BootImageFetchFailed, + BootImageFetchOutcome, + BootImagePublishFailed, + fetch_verified_boot_image, +} + +// THE POLICY LAYER FOR ONE HOST'S BOOT IMAGE. Which artifact this unit should carry, and where the +// controller reads it from, are workflow decisions; the operation stays reusable. +// +// THE URL AND THE PIN ARE THE SAME ROW UPSTREAM. extdeps.provisioning.ubuntu_install_media +// noble_numbat_2404_3_live_server_arm64 carries content_sha256 cited to cdimage's own SHA256SUMS, +// and the artifact this lane already booted measured exactly that value. The digest below is that +// same figure; when the artifact row grows a locator field this policy should read both from it +// rather than carrying either. +data mtcollins1_boot_image_url: NonEmptyStr = "https://cdimage.ubuntu.com/releases/24.04/release/ubuntu-24.04.3-live-server-arm64.iso" + +data mtcollins1_boot_image_sha256: NonEmptyStr = "2ee2163c9b901ff5926400e80759088ff3b879982a3956c02100495b489fd555" + +// STAGING IS OUTSIDE THE EXPORT, PUBLICATION IS INSIDE IT. That is the whole safety property: the +// controller enumerates /srv/bmc on its own schedule, so an artifact placed there is mountable +// before anything has measured it. Staging elsewhere means a file only ever appears in the export +// after its digest agreed. +data mtcollins1_boot_image_staging_path: NonEmptyStr = "/var/tmp/gunbc-mtcollins1-boot-image.iso" + +data mtcollins1_boot_image_published_path: NonEmptyStr = "/srv/bmc/ubuntu-24.04.3-live-server-arm64.iso" + +fn mtcollins1_render_fetch(outcome: BootImageFetchOutcome) -> ProcessExit { + match outcome { + BootImageAdmitted { path: _, digest: _ } => ExitSuccess + BootImageFetchFailed { url: u } => + exit_failure(reason: concat("mtcollins1 boot image: fetch failed for ", u as String)) + BootImageDigestUnreadable { path: p, reason: r } => + exit_failure(reason: concat("mtcollins1 boot image: staged at ", p as String, " but its digest could not be read: ", r)) + BootImageDigestMismatchRefused { expected: e, observed: o } => + exit_failure(reason: concat("mtcollins1 boot image: REFUSED, staged bytes digest ", o as String, " does not match the pinned ", e as String, "; the staged file is left in place as the evidence and was NOT published")) + BootImagePublishFailed { staged: s, destination: d } => + exit_failure(reason: concat("mtcollins1 boot image: digest agreed but publishing ", s as String, " to ", d as String, " failed")) + } +} + +func mtcollins1_fetch_boot_image() -> ProcessExit + uses net: std.resources.Network +{ + return mtcollins1_render_fetch( + outcome: fetch_verified_boot_image( + url: mtcollins1_boot_image_url, + staging_path: mtcollins1_boot_image_staging_path, + published_path: mtcollins1_boot_image_published_path, + expected_sha256: mtcollins1_boot_image_sha256, + ) + ) +} + +// THE DISCRIMINATING RED. Same artifact, same paths, a pin that cannot match. If this returns +// success the verification is decorative, and the wrong-pin case is the one that matters: a +// mismatched image reaching the export is exactly the failure the staging split exists to prevent. +// It publishes to a path the controller does NOT read, so a defect in the wall cannot put an +// unverified artifact where the unit would boot it. +data mtcollins1_wrong_pin: NonEmptyStr = "0000000000000000000000000000000000000000000000000000000000000000" + +data mtcollins1_wrong_pin_published_path: NonEmptyStr = "/var/tmp/gunbc-mtcollins1-wrongpin-published.iso" + +func mtcollins1_fetch_boot_image_wrong_pin_control() -> ProcessExit + uses net: std.resources.Network +{ + return mtcollins1_render_fetch( + outcome: fetch_verified_boot_image( + url: mtcollins1_boot_image_url, + staging_path: mtcollins1_boot_image_staging_path, + published_path: mtcollins1_wrong_pin_published_path, + expected_sha256: mtcollins1_wrong_pin, + ) + ) +} From 6e8e70902e6e64950d55b8ce29510b7cf8f68138 Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Sun, 6 Sep 2026 03:04:00 +0000 Subject: [PATCH 04/11] Name who may change what a machine boots MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The modeled fetch verifies an artifact correctly and then cannot publish it: /srv/bmc on srv2 was drwxr-xr-x root root and gunbc runs unprivileged, so the operation ended at BootImagePublishFailed. That refusal was accurate, and the question behind it is not a permissions detail. THE EXPORT IS THE PAYLOAD HALF OF THE BOOT PATH. Write access to the directory a controller reads its virtual media from is equivalent to choosing what the host runs -- the same authority the out-of-band handoff exercises from the control half. This lane spent its effort making the control half admit, verify and refuse; leaving the payload half as "whatever can write the directory" secures one end of a chain. BOTH EASY ANSWERS ARE WRONG IN THE SAME WAY. Escalating the publish step means converge acquires privilege for a file write, which is not proportionate to putting a verified ISO in a directory. Widening the directory to the interactive login is the mirror failure: every process running as that user could then replace the boot payload, and on a workstation-shaped account that population is unbounded and unnamed. THE PRINCIPAL ALREADY EXISTED AND A FIRST DRAFT MINTED A SECOND ONE. That draft carried a free PosixLoginName "gunbc-media"; gunbc.fleet_posix_accounts already owns FleetPosixAccountKey, and FleetAccountAutomation is declared there with login_policy KeyOnlyAutomation -- a non-interactive identity that exists for exactly this. A media-specific account would have been the nicknaming DESIGN §3 forbids. The requirement now names the key. NO DEDICATED BOOT-PAYLOAD ACCOUNT IS SPLIT OFF, though a narrower principal is available in principle. Nothing has established that converge automation should be unable to change what a host boots -- converge is the thing that decides what a host boots -- so the split would be speculative separation. If a fact ever demands it, the key vocabulary is where the variant goes. GROUP AND OTHER ARE BOTH DENIED WRITE, and that is the load-bearing half. An export owned by a named principal but group-writable has MOVED the authority rather than narrowed it: the population able to replace the boot payload becomes the group's membership, which is the unnamed category this row exists to avoid. FOUR STANDINGS, ALL EXERCISED. root is its own arm rather than one case of a wrong owner, because the remedies differ -- a wrong-but-unprivileged owner is a chgrp, while root means the publish step would have to escalate, which is the design decision being refused rather than a misconfiguration to correct. An account outside the fleet roster is its own arm too: reporting it as "wrong principal" would name a required principal while saying nothing about what is actually there. root-owned ExportOwnershipIsRoot the automation principal satisfied an ordinary interactive account wrong-principal an account outside the roster unrecognized The interactive case is the one worth asserting: it is the outcome someone reaches by chgrp-ing the directory to their own login to make the publish step work, which succeeds and silently widens exactly what this row exists to name. THE FIXTURE WAS WRONG AND THE HOST CORRECTED IT. It carried name "gunbc" at uid 997, an account that exists nowhere. srv2's fleet automation principal is gunbc-fleet at uid 1001, already the principal the host reach and identity probes compare against. A fixture whose account does not exist tests the fold against a fiction and passes just as happily; it was found by reading the host rather than by trusting the row. THE HOST CHANGE IS MADE: /srv/bmc is now gunbc-fleet:gunbc-fleet mode 755. No account creation was required. The controller still reads the share and the attached image remains bound (redirection_status 1); the images LISTING went stale again across the change, which is the cache behaviour already recorded in the diskless boot receipt rather than a regression. WHAT THIS DOES NOT CLOSE, and it is a moved gap rather than a solved one: the publish step still refuses from an operator session, because gunbc runs as uid 1000 while the export belongs to 1001. That is the requirement working, not failing -- it says the AUTOMATION principal owns the export, so converge should run as that principal. Special-casing a grant to make an operator session write there would reintroduce precisely the widening refused above. Corpus: 27 blocking, unchanged, none in any file this commit touches. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01D7PLjw5FMYdZ2GWn3FeMpR --- .../boot_image_export_ownership.dag | 216 ++++++++++++++++++ 1 file changed, 216 insertions(+) create mode 100644 dag/gunbc/machine_intake/boot_image_export_ownership.dag diff --git a/dag/gunbc/machine_intake/boot_image_export_ownership.dag b/dag/gunbc/machine_intake/boot_image_export_ownership.dag new file mode 100644 index 00000000000..ea1a3546143 --- /dev/null +++ b/dag/gunbc/machine_intake/boot_image_export_ownership.dag @@ -0,0 +1,216 @@ +module gunbc.machine_intake_boot_image_export_ownership + +import std.types { Bool, Int, NonEmptyStr, String } +import std.process { ExitSuccess, ProcessExit, exit_failure } +import extdeps.access.posix { + FileOwnership, + LoginShellObserved, + LoginShellUnobserved, + PosixUser, + posix_user_is_root, +} +import gunbc.fleet_posix_accounts { + FleetAccountAutomation, + FleetAccountCiRunner, + FleetAccountHostAdministrator, + FleetAccountInferenceService, + FleetAccountOperator, + FleetAccountPublicationHelper, + FleetAccountRoot, + FleetPosixAccountKey, +} + +// WHO MAY CHANGE WHAT A MACHINE BOOTS. The directory a controller reads its virtual media from is +// the payload half of the boot path: write access to it is equivalent to choosing what the host +// runs, which is the same authority the out-of-band boot handoff exercises from the control half. +// This lane spent its effort making the control half admit, verify and refuse; leaving the payload +// half as "whatever can write the directory" would secure one end of a chain. +// +// THE MEASURED STATE IS ROOT-OWNED AND UNWRITABLE BY THE CONVERGE IDENTITY. /srv/bmc on srv2 is +// drwxr-xr-x root root, gunbc runs as uid 1000, and the modeled fetch consequently verifies an +// artifact correctly and then cannot publish it — BootImagePublishFailed, measured 2026-09-06. +// That refusal is accurate rather than a defect, and it is what this row exists to resolve. +// +// ROOT IS THE WRONG ANSWER EVEN THOUGH IT WOULD WORK. Escalating the publish step means converge +// acquires privilege for a file write, and a grant that broad is not proportionate to "put a +// verified ISO in a directory". Widening the directory to the interactive login account is the +// mirror failure: it makes every process running as that user able to replace the boot payload, +// and on a workstation-shaped account that population is unbounded and unnamed. +// +// SO THE OWNER IS A NAMED FLEET PRINCIPAL, which is the only option where the question "who can +// change what this machine boots" has an answer that is a NAME rather than a category. +// +// AND IT IS AN EXISTING PRINCIPAL RATHER THAN A NEW ONE. A first draft of this row carried a free +// PosixLoginName "gunbc-media", which is a second account vocabulary beside the one the fleet +// already has: gunbc.fleet_posix_accounts owns FleetPosixAccountKey, and FleetAccountAutomation is +// already declared there with login_policy KeyOnlyAutomation — a non-interactive identity that +// exists precisely for work like this. Minting a media-specific account would have been the +// nicknaming DESIGN §3 forbids, and the requirement now names the key. +// +// WHY NOT A DEDICATED BOOT-PAYLOAD ACCOUNT, since a narrower principal is available in principle: +// splitting "the identity that may publish a boot image" from "the automation identity" would be +// speculative separation. Nothing has established that converge automation should be unable to +// change what a host boots — converge is the thing that decides what a host boots. If that +// separation is ever wanted it will be because some fact demanded it, and the key vocabulary is +// where the new variant would go. +type BootImageExportOwnershipRequirement { + export_path: NonEmptyStr + required_owner: FleetPosixAccountKey + writable_by_owner: Bool + writable_by_group: Bool + writable_by_other: Bool +} + +// THE REQUIREMENT, not an observation. Nothing here reports what /srv/bmc currently is; it states +// what an export feeding a controller must satisfy before the publish step can be admitted. The +// observation that today's directory does NOT satisfy it is the fetch operation's refusal, which +// already names both paths. +// +// GROUP AND OTHER ARE BOTH DENIED WRITE, and that is the load-bearing half. An export whose owner +// is a service identity but whose group is writable has moved the authority rather than narrowed +// it: the population that can replace the boot payload becomes the group's membership, which is +// exactly the unnamed category this row exists to avoid. +data boot_image_export_ownership_requirement: BootImageExportOwnershipRequirement = BootImageExportOwnershipRequirement { + export_path: "/srv/bmc", + required_owner: FleetAccountAutomation, + writable_by_owner: true, + writable_by_group: false, + writable_by_other: false, +} + +// WHETHER AN OBSERVED OWNERSHIP SATISFIES THE REQUIREMENT. Takes the observed owner rather than +// reading the filesystem, so the judgment is testable without a host and so the observation stays +// the caller's obligation. +type ExportOwnershipStanding + = ExportOwnershipSatisfied + | ExportOwnershipIsRoot + | ExportOwnershipWrongPrincipal { observed: FleetPosixAccountKey, required: FleetPosixAccountKey } + | ExportOwnershipUnrecognizedAccount { observed: NonEmptyStr } + +// ROOT IS ITS OWN ARM rather than one case of a wrong owner, because the remedy differs. A +// wrong-but-unprivileged owner is a chgrp away; root means the publish step would have to escalate, +// which is the design decision this row exists to refuse rather than a misconfiguration to correct. +// COMPARING TWO CLOSED KEYS THROUGH A LABEL, because the enum carries no equality of its own and a +// nested match over the pair would be forty-nine arms to answer one question. This is the same +// shape v1.compiler.compile var_binding_kind_name uses for VarBindingKind: a total serialization of +// a closed vocabulary, exhaustive so that a new account variant is a compile error here rather than +// a silent non-match. +// +// IT LIVES HERE RATHER THAN UPSTREAM ON PURPOSE, and should move if a second consumer appears: +// gunbc.fleet_posix_accounts is the vocabulary's home, and a label fn belongs beside the type once +// more than one caller wants it. One caller does not justify editing the authority. +fn fleet_account_key_label(key: FleetPosixAccountKey) -> String { + match key { + FleetAccountRoot => "root" + FleetAccountOperator => "operator" + FleetAccountCiRunner => "ci-runner" + FleetAccountHostAdministrator => "host-administrator" + FleetAccountAutomation => "automation" + FleetAccountInferenceService => "inference-service" + FleetAccountPublicationHelper => "publication-helper" + } +} + +fn fleet_account_key_matches(a: FleetPosixAccountKey, b: FleetPosixAccountKey) -> Bool { + fleet_account_key_label(key: a) == fleet_account_key_label(key: b) +} + +// AN OWNER THIS FLEET DOES NOT RECOGNISE IS ITS OWN ARM, not a wrong principal. A directory owned +// by an account outside the fleet's declared roster is a different situation from one owned by the +// wrong declared principal: the second is a chgrp, the first means someone provisioned an account +// this model has never heard of, and reporting it as "wrong principal" would name a required +// principal while saying nothing about what is actually there. +fn export_ownership_standing( + observed: PosixUser, + observed_key: FleetPosixAccountKey?, + requirement: BootImageExportOwnershipRequirement, +) -> ExportOwnershipStanding { + if posix_user_is_root(user: observed) { + ExportOwnershipIsRoot + } else { + match observed_key { + Absent => ExportOwnershipUnrecognizedAccount { observed: observed.name } + Present { value: k } => + if fleet_account_key_matches(a: k, b: requirement.required_owner) { + ExportOwnershipSatisfied + } else { + ExportOwnershipWrongPrincipal { observed: k, required: requirement.required_owner } + } + } + } +} + +// THE ARMS, EXERCISED. Each fixture is the shape it stands for: the export's measured state today, +// the intended automation principal, an ordinary interactive account, and an account outside the +// fleet roster entirely. +// +// THE INTERACTIVE CASE IS THE ONE WORTH ASSERTING. It is the outcome someone reaches by chgrp-ing +// the directory to their own login to make the publish step work — which succeeds, and silently +// widens "who can change what this machine boots" to every process running as that user. +data posix_root_fixture: PosixUser = PosixUser { + name: "root", + uid: 0, + gid: 0, + shell: LoginShellObserved { path: "/bin/bash" }, +} + +// THE AUTOMATION FIXTURE IS THE REAL ACCOUNT, read from srv2 rather than invented. A first draft +// carried name "gunbc" at uid 997, which exists nowhere: the fleet's automation principal on this +// host is gunbc-fleet at uid 1001, and it is already the principal the host reach and identity +// probes compare against as observed_principal. A fixture whose account does not exist tests the +// fold against a fiction, and would have passed just as happily. +data posix_automation_fixture: PosixUser = PosixUser { + name: "gunbc-fleet", + uid: 1001, + gid: 1001, + shell: LoginShellObserved { path: "/bin/bash" }, +} + +data posix_interactive_fixture: PosixUser = PosixUser { + name: "briansrls", + uid: 1000, + gid: 1000, + shell: LoginShellObserved { path: "/bin/bash" }, +} + +fn export_ownership_label(standing: ExportOwnershipStanding) -> String { + match standing { + ExportOwnershipSatisfied => "satisfied" + ExportOwnershipIsRoot => "root" + ExportOwnershipWrongPrincipal { observed: _, required: _ } => "wrong-principal" + ExportOwnershipUnrecognizedAccount { observed: _ } => "unrecognized" + } +} + +func boot_image_export_ownership_probe() -> ProcessExit { + let on_root = export_ownership_label(standing: export_ownership_standing( + observed: posix_root_fixture, + observed_key: Present { value: FleetAccountRoot }, + requirement: boot_image_export_ownership_requirement)) + let on_automation = export_ownership_label(standing: export_ownership_standing( + observed: posix_automation_fixture, + observed_key: Present { value: FleetAccountAutomation }, + requirement: boot_image_export_ownership_requirement)) + let on_interactive = export_ownership_label(standing: export_ownership_standing( + observed: posix_interactive_fixture, + observed_key: Present { value: FleetAccountOperator }, + requirement: boot_image_export_ownership_requirement)) + let on_unknown = export_ownership_label(standing: export_ownership_standing( + observed: posix_interactive_fixture, + observed_key: none, + requirement: boot_image_export_ownership_requirement)) + + if on_root != "root" { + return exit_failure(reason: concat("export ownership: a root-owned export answered ", on_root)) + } + if on_automation != "satisfied" { + return exit_failure(reason: concat("export ownership: the automation principal did not satisfy the requirement, it answered ", on_automation)) + } + if on_interactive != "wrong-principal" { + return exit_failure(reason: concat("export ownership: an ordinary interactive account was ADMITTED as the export owner, answering ", on_interactive, "; that is the widening this requirement exists to refuse")) + } + if on_unknown != "unrecognized" { + return exit_failure(reason: concat("export ownership: an account outside the fleet roster answered ", on_unknown, " rather than being reported as unrecognized")) + } + return ExitSuccess +} From c1203c9419829360ac928bcb651e14a02ac9b152 Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Sun, 6 Sep 2026 03:27:36 +0000 Subject: [PATCH 05/11] Record the platform facts this lane tripped over MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three things surfaced while driving the controller for netboot, none of them the object of a measurement, each with an owner outside this lane. They existed only in a session transcript until now. THE UNIT IS RUNNING NON-REDUNDANT, AND IT IS CURRENT RATHER THAN HISTORICAL. PSU2_Status (0xeb) asserts BOTH [Presence detected] AND [Power Supply AC lost] right now — not merely in the event log — so the supply is installed and has no AC input. The corroborating sensors agree: PSU1_POUT 160 W PSU2_POUT 0 W PSU1_FAN_SPEED 5000 RPM PSU2_FAN_SPEED 0 RPM One supply is carrying the entire load while the other is inert. The SEL carries the assertion on 09/04 and again on 09/05 controller time, so this predates the netboot work rather than being caused by it. IT IS RANKED ABOVE THE CLOCK SKEW THIS LANE ALSO RECORDED. A wrong clock makes receipts unjoinable and is fixed by one write. A single-corded machine has no redundancy at all, cannot be fixed by any write, and this lane is actively making that machine something the fleet depends on for boot. The remedy is physical — a cord, an outlet, or a PDU port — and belongs to whoever is in the room. PRESENCE AND AC INPUT ARE SEPARATE FIELDS rather than one health flag, because the present-but-unpowered case is the interesting one: an absent supply is a visible gap in a chassis, while a present one with no cord looks correct from the front and reports itself as installed. Redundancy is a property of the POWERED supplies, not the installed ones. THE CONTROLLER HAS TWO INTERFACES AND ONLY ONE IS THE ONE WE USE. 192.168.1.228 is the dedicated management port; 192.168.1.231 is a second interface on the SAME controller, established rather than assumed — mc info at both addresses reports Device ID 32 and firmware 0.32. It is recorded because it was once MISREAD AS THE HOST. A DHCP capture from that address was initially taken for the host PXE-booting; it was not. Its vendor class is "udhcp 1.21.1", the BusyBox client the controller's own Linux runs, and the host NIC is a third address entirely. The discriminator is the vendor class rather than the MAC's proximity: a PXE client identifies itself as PXEClient:Arch:NNNNN and asks for a boot file, and a Linux DHCP client does neither. A second management address answering the same credentials is also a second reachable surface, which an access policy should know about the unit. THE PROBE EXERCISES BOTH DIRECTIONS. Asserting only that the measured rows are not redundant would pass just as happily against a check that answered "not redundant" unconditionally — and this unit's live case is the unhappy one, so that is exactly the mistake available here. A corrected pair, identical but with AC present, must be reported redundant for the live verdict to carry any information. Corpus: 27 blocking, unchanged, none in any file this commit touches. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01D7PLjw5FMYdZ2GWn3FeMpR --- .../mtcollins1_platform_observation.dag | 110 ++++++++++++++++++ 1 file changed, 110 insertions(+) create mode 100644 dag/gunbc/machine_intake/mtcollins1_platform_observation.dag diff --git a/dag/gunbc/machine_intake/mtcollins1_platform_observation.dag b/dag/gunbc/machine_intake/mtcollins1_platform_observation.dag new file mode 100644 index 00000000000..355237701d5 --- /dev/null +++ b/dag/gunbc/machine_intake/mtcollins1_platform_observation.dag @@ -0,0 +1,110 @@ +module gunbc.machine_intake_mtcollins1_platform_observation + +import std.types { Bool, Int, List, NonEmptyStr, String } +import std.process { ExitSuccess, ProcessExit, exit_failure } + +// PLATFORM FACTS OBSERVED WHILE DOING SOMETHING ELSE, which is why they are recorded here rather +// than left in a session transcript. None of these was the object of a measurement; each surfaced +// while driving the controller for the netboot lane, and each has an owner outside that lane. +// +// THE REDUNDANCY LOSS IS THE ONE THAT MATTERS AND IT IS CURRENT, not historical. PSU2_Status (0xeb) +// asserts BOTH [Presence detected] AND [Power Supply AC lost] right now, not merely in the event +// log: the supply is installed and has no AC input. The corroborating sensors agree — +// +// PSU1_POUT 160 Watts PSU2_POUT 0 Watts +// PSU1_FAN_SPEED 5000 RPM PSU2_FAN_SPEED 0 RPM +// +// so one supply is carrying the entire load while the other is inert. The SEL carries the assertion +// twice, on 09/04 and 09/05 controller time, meaning this predates the netboot work rather than +// being caused by it. +// +// WHY IT IS RANKED ABOVE THE CLOCK SKEW ALSO RECORDED IN THIS LANE: a wrong clock makes receipts +// unjoinable and is fixed by one write. A single-corded machine has no redundancy at all, cannot be +// fixed by any write, and this lane is actively making that machine something the fleet depends on +// for boot. The remedy is physical — a cord, an outlet, or a PDU port — and belongs to whoever is +// in the room. +type PowerSupplyStanding { + sensor_name: NonEmptyStr + present: Bool + ac_input_present: Bool + output_watts: Int + fan_rpm: Int +} + +data mtcollins1_psu1_standing: PowerSupplyStanding = PowerSupplyStanding { + sensor_name: "PSU1_Status", + present: true, + ac_input_present: true, + output_watts: 160, + fan_rpm: 5000, +} + +data mtcollins1_psu2_standing: PowerSupplyStanding = PowerSupplyStanding { + sensor_name: "PSU2_Status", + present: true, + ac_input_present: false, + output_watts: 0, + fan_rpm: 0, +} + +data mtcollins1_power_redundancy_observed_at: NonEmptyStr = "2026-09-06T02:30Z" + +// A SUPPLY THAT IS PRESENT BUT UNPOWERED IS THE INTERESTING CASE, and it is why presence and AC +// input are separate fields rather than one health flag. An absent supply is a visible gap in a +// chassis; a present one with no cord looks correct from the front and reports itself as installed. +// Redundancy is a property of the POWERED supplies, not the installed ones. +fn power_supplies_redundant(a: PowerSupplyStanding, b: PowerSupplyStanding) -> Bool { + a.present && a.ac_input_present && b.present && b.ac_input_present +} + +// THE CONTROLLER HAS TWO NETWORK INTERFACES AND ONLY ONE IS THE ONE WE USE. 192.168.1.228 is the +// dedicated management port; 192.168.1.231 is a second interface on the SAME controller, which is +// established rather than assumed — `mc info` at both addresses reports Device ID 32 and firmware +// 0.32, and the second answers DHCP with vendor class "udhcp 1.21.1", the BusyBox client the +// controller's own Linux runs. +// +// IT IS RECORDED BECAUSE IT WAS ONCE MISREAD AS THE HOST. A DHCP capture from that address was +// initially taken for the host PXE-booting; it was not. The host NIC is a third address entirely +// (28:c1:3c:8a:d6:4a), and the discriminator is the vendor class rather than the MAC's proximity — +// a PXE client identifies itself as PXEClient:Arch:NNNNN and asks for a boot file, while a Linux +// DHCP client does neither. A second management address that answers to the same credentials is +// also a second reachable surface, which is a fact an access policy should know about the unit. +type ControllerInterface { + address: NonEmptyStr + mac: NonEmptyStr + is_dedicated_management_port: Bool +} + +data mtcollins1_controller_interfaces: List = [ + ControllerInterface { + address: "192.168.1.228", + mac: "28:c1:3c:8a:d6:4c", + is_dedicated_management_port: true, + }, + ControllerInterface { + address: "192.168.1.231", + mac: "28:c1:3c:8a:d6:4d", + is_dedicated_management_port: false, + }, +] + +// THE FOLD IS EXERCISED AGAINST THE MEASURED ROWS AND AGAINST A CORRECTED PAIR. The second is what +// makes the first mean something: a redundancy check that answered "not redundant" unconditionally +// would pass the live case for the wrong reason, and this unit's live case is the unhappy one. +data mtcollins1_psu2_if_corded: PowerSupplyStanding = PowerSupplyStanding { + sensor_name: "PSU2_Status", + present: true, + ac_input_present: true, + output_watts: 80, + fan_rpm: 4800, +} + +func mtcollins1_power_redundancy_probe() -> ProcessExit { + if power_supplies_redundant(a: mtcollins1_psu1_standing, b: mtcollins1_psu2_standing) { + return exit_failure(reason: "power redundancy: the measured rows were reported REDUNDANT, but PSU2 asserts [Power Supply AC lost] with 0 W output and 0 RPM") + } + if power_supplies_redundant(a: mtcollins1_psu1_standing, b: mtcollins1_psu2_if_corded) == false { + return exit_failure(reason: "power redundancy: two present, powered supplies were NOT reported redundant, so the check cannot distinguish the corded case and its verdict on the live rows carries no information") + } + return ExitSuccess +} From 8c910c1520a7b73575d76100eb81fe002fe7cd4b Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Sun, 6 Sep 2026 09:48:19 +0000 Subject: [PATCH 06/11] Make the media attach idempotent, and retract a dangerous remedy Attaching an already-attached image refused. For a converge step that is the failure mode that matters most, since converge is run repeatedly by construction. The cause was a wrong reading of the controller. /remote/images enumerates what is AVAILABLE to attach, not what the share holds: measured on Mt. Collins with two ISOs exported, the listing named both; after start-media attached one, it returned only the other, and the survivor's image_index moved 1 -> 0. The attached image had moved to /remote/configurations. The earlier reading was that the listing is a cache that goes stale, with the operator-facing remedy "re-assert mount_cd". That remedy appeared to work for the worst possible reason: toggling mount_cd detaches everything, returning the image to the available pool. On a host booted from that medium -- this lane's whole purpose -- the advice removes the running root. It is retracted rather than reworded. So configurations is now read first and the converged case performs no write at all. That is not an optimisation: start-media against an already-presented image is a write whose effect on a booted host is unmodelled, and reading first is how we avoid making it. Also drops the index comparison from the confirmation. It was added reading image_index, which is absent from a configurations row, so every genuine attachment was reported unconfirmed -- caught because the check failed closed. Re-reading it as media_index passed, and was still wrong: media_index is the virtual CD device slot, beside session_index, not a position in the image enumeration. It agreed only because this unit presents one CD device, so both are 0. Joining them is nicknaming, and it would have begun refusing on any controller with a second slot while blaming the image. Verified against the live controller, one input differing: the real image returns success in three calls with no write; an absent name refuses after the two reads, before start. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01D7PLjw5FMYdZ2GWn3FeMpR --- dag/extdeps/bmc/megarac.dag | 27 ++- .../boot_image_export_ownership.dag | 22 ++- .../machine_intake/megarac_media_attach.dag | 169 ++++++++++++++++-- .../mtcollins1_diskless_boot_receipt.dag | 23 ++- .../mtcollins1_media_attach.dag | 106 +++++++++-- .../mtcollins1_platform_observation.dag | 36 +++- 6 files changed, 338 insertions(+), 45 deletions(-) diff --git a/dag/extdeps/bmc/megarac.dag b/dag/extdeps/bmc/megarac.dag index e57f800ea11..8f6aad3d536 100644 --- a/dag/extdeps/bmc/megarac.dag +++ b/dag/extdeps/bmc/megarac.dag @@ -165,9 +165,18 @@ fn megarac_image_redirection_wire(r: MegaRacImageRedirectionRequired) -> NonEmpt // // SESSION AUTH IS COOKIE-AND-HEADER, NOT netrc, which is why these cannot reuse the redfish.Http // shape beside them. POST /api/session returns a CSRFToken in its body and sets a session cookie; -// every subsequent request must present BOTH the cookie jar and an X-CSRFTOKEN header. The jar path -// is a parameter rather than a fixed location so an attempt owns its own jar and two concurrent -// attempts cannot share one session. +// every subsequent request must present BOTH the cookie jar and an X-CSRFTOKEN header. +// +// THE JAR PATH BEING A PARAMETER DOES NOT MAKE AN ATTEMPT OWN ITS JAR, and an earlier revision of +// this note claimed it did. A parameter permits a per-attempt path; it does not require one, and +// the first production caller passed a fixed literal — so two concurrent attempts shared one bearer +// file, which is the opposite of what the note asserted. Ownership is a property of the CALLER's +// allocation, and the caller is where it has to be established. +// +// A SESSION THAT IS NEVER CLOSED IS A LEAKED REMOTE RESOURCE. CloseSession exists because the +// controller's session table is finite and this lane exhausted a service's sessions by opening one +// per probe and closing none. Every operation sequence that opens a session owes a close on every +// exit, including refusals — an operation that failed still allocated the session that carried it. // // THE CREDENTIAL IS FORM-ENCODED IN THE SESSION CALL AND NOWHERE ELSE. It reaches argv there, which // is a real weakness and is stated rather than hidden: curl offers no netrc equivalent for a @@ -260,4 +269,16 @@ service megarac.Media { nonzero => String "megarac stop-media POST failed" } } + + operation CloseSession { + input { bmc_host: NonEmptyStr, cookie_jar: NonEmptyStr, csrf_token: NonEmptyStr } + output { body: String from "stdout", success: Bool from "exit_success" } + transport shell { + argv: ["curl", "--fail-with-body", "-sS", "-k", "-b", "{cookie_jar}", "-H", "X-CSRFTOKEN: {csrf_token}", "-X", "DELETE", "https://{bmc_host}/api/session"] + } + exit { + 0 => Unit + nonzero => String "megarac session DELETE failed" + } + } } diff --git a/dag/gunbc/machine_intake/boot_image_export_ownership.dag b/dag/gunbc/machine_intake/boot_image_export_ownership.dag index ea1a3546143..aae6648c89a 100644 --- a/dag/gunbc/machine_intake/boot_image_export_ownership.dag +++ b/dag/gunbc/machine_intake/boot_image_export_ownership.dag @@ -26,10 +26,24 @@ import gunbc.fleet_posix_accounts { // This lane spent its effort making the control half admit, verify and refuse; leaving the payload // half as "whatever can write the directory" would secure one end of a chain. // -// THE MEASURED STATE IS ROOT-OWNED AND UNWRITABLE BY THE CONVERGE IDENTITY. /srv/bmc on srv2 is -// drwxr-xr-x root root, gunbc runs as uid 1000, and the modeled fetch consequently verifies an -// artifact correctly and then cannot publish it — BootImagePublishFailed, measured 2026-09-06. -// That refusal is accurate rather than a defect, and it is what this row exists to resolve. +// THE STATE THIS ROW WAS WRITTEN AGAINST HAS SINCE CHANGED, and saying so is the point rather than +// an afterthought. When it was authored /srv/bmc on srv2 was drwxr-xr-x root root, so the modeled +// fetch verified an artifact correctly and then could not publish it. The directory was +// subsequently chowned to gunbc-fleet:gunbc-fleet mode 0755, which satisfies the ownership half of +// the requirement below. +// +// THE PUBLICATION STILL REFUSES FROM AN OPERATOR SESSION, and for a different reason than before: +// gunbc runs as uid 1000 while the export now belongs to 1001. That is the requirement working — it +// says the AUTOMATION principal owns the export, so converge should run as that principal — rather +// than the earlier state, which was simply that nobody but root could write there. +// +// WHAT THIS ROW DOES NOT YET DO, stated because the gap is easy to miss under a page of prose about +// authority: NOTHING CONSUMES IT AT THE WRITE BOUNDARY. publish_verified_boot_image calls mv +// directly and never asks export_ownership_standing anything, so the policy below has no causal +// position — the program publishes whenever the operating system's permissions happen to allow it, +// including under root or a world-writable export. The requirement is currently a description of +// what ought to hold, checked by a probe against fixtures, not a wall. Closing that needs a real +// stat plus effective-principal observation bound to the path, admitted before the move. // // ROOT IS THE WRONG ANSWER EVEN THOUGH IT WOULD WORK. Escalating the publish step means converge // acquires privilege for a file write, and a grant that broad is not proportionate to "put a diff --git a/dag/gunbc/machine_intake/megarac_media_attach.dag b/dag/gunbc/machine_intake/megarac_media_attach.dag index 8670b659257..fd9692859bd 100644 --- a/dag/gunbc/machine_intake/megarac_media_attach.dag +++ b/dag/gunbc/machine_intake/megarac_media_attach.dag @@ -35,17 +35,54 @@ import extdeps.languages.json.parse { // which on a share holding a memtest ISO and an installer ISO is a silent wrong-boot rather than a // failure. The index is read from the images listing by matching the image NAME. // -// AND AN EMPTY LISTING IS NOT AN EMPTY SHARE. After a new file appeared the controller returned an -// empty image list including for a file it had previously enumerated, until mount_cd was -// re-asserted. The listing is a cache. This module refuses on an empty listing rather than -// concluding the image is absent, because those are different states with different remedies and -// only one of them is the share's fault. +// THE IMAGES LISTING ENUMERATES WHAT IS AVAILABLE TO ATTACH, NOT WHAT THE SHARE HOLDS, and an +// earlier revision of this module got that exactly backwards in a way worth recording because the +// wrong diagnosis had a dangerous remedy attached to it. Measured on this unit 2026-09-06: with two +// ISOs in the export, /remote/images listed both; after start-media attached one, the listing +// returned ONLY THE OTHER, and the survivor's image_index moved from 1 to 0. The attached image had +// not vanished — it had moved to /remote/configurations, which is where a presented image lives. +// +// THE EARLIER READING WAS THAT THE LISTING IS A CACHE THAT GOES STALE, with the remedy "re-assert +// mount_cd". That remedy appeared to work, and the reason it appeared to work is the problem: +// toggling mount_cd DETACHES EVERYTHING, which returns the image to the available pool. On a host +// that is booted from that medium — which is this lane's entire purpose — the advice amounts to +// pulling the running root out from under it. A remedy that works by undoing the desired state is +// worse than no remedy, and it was written into an operator-facing message. +// +// SO THE ORDER OF READS IS LOAD-BEARING AND CONFIGURATIONS COMES FIRST. Asking the available list +// whether an already-attached image is present is asking the wrong authority a question it answers +// "no" for the one reason that means SUCCESS. That is what made the operation non-idempotent: a +// second converge against a correctly-attached host refused, which for a converge step is the +// failure mode that matters most, since converge is run repeatedly by construction. +// THE CLEANUP OUTCOME IS CARRIED BESIDE THE ATTACH OUTCOME, never folded into it. Whether the +// image attached and whether the session was released are different facts with different owners: a +// successful attach that leaked a session is still a successful attach, and reporting it as a +// failure would hide the thing that worked while a caller retried it and leaked another. The +// controller's session table is finite — this lane exhausted one service's sessions by opening one +// per probe and closing none — so the release has to be observable rather than assumed. +// +// SessionReleaseUncertain EXISTS BECAUSE A LOST RESPONSE IS NOT A KNOWN STATE. A DELETE whose reply +// never arrived may have released the session or may not; treating that as released would +// under-count leaks exactly when the transport is already misbehaving, which is when leaks matter. +type SessionRelease + = SessionReleased + | SessionReleaseRefused { detail: String } + | SessionReleaseNotAttempted + +type MegaRacAttachResult { + outcome: MegaRacAttachOutcome + session_release: SessionRelease +} + type MegaRacAttachOutcome - = MegaRacAttached { image_name: NonEmptyStr, image_index: Int } + = MegaRacAttached { image_name: NonEmptyStr, image_index: String } + | MegaRacAlreadyAttached { image_name: NonEmptyStr } + | MegaRacConfigurationsUnreadable { detail: String } | MegaRacSessionRefused { detail: String } | MegaRacSessionTokenUnreadable { detail: String } | MegaRacImagesUnreadable { detail: String } | MegaRacImageListingEmpty + | MegaRacImageListingUnparseable { detail: String } | MegaRacImageNotEnumerated { image_name: NonEmptyStr } | MegaRacStartRefused { detail: String } | MegaRacAttachUnconfirmed { image_name: NonEmptyStr, configurations: String } @@ -132,7 +169,7 @@ func megarac_attach_remote_image( password: NonEmptyStr, cookie_jar: NonEmptyStr, image_name: NonEmptyStr, -) -> MegaRacAttachOutcome +) -> MegaRacAttachResult uses net: std.resources.Network { session = megarac.Media.OpenSession( @@ -142,7 +179,10 @@ func megarac_attach_remote_image( cookie_jar: cookie_jar, ) if session.success == false { - return MegaRacSessionRefused { detail: session.body } + return MegaRacAttachResult { + outcome: MegaRacSessionRefused { detail: session.body }, + session_release: SessionReleaseNotAttempted, + } } let session_doc = match parse_json_document(s: session.body) { @@ -154,10 +194,14 @@ func megarac_attach_remote_image( Present { value: doc } => json_string_member(doc: doc, key: "CSRFToken") } - match csrf { - Absent => MegaRacSessionTokenUnreadable { detail: session.body } + return match csrf { + Absent => + MegaRacAttachResult { + outcome: MegaRacSessionTokenUnreadable { detail: session.body }, + session_release: SessionReleaseNotAttempted, + } Present { value: token } => - megarac_attach_with_token( + megarac_attach_then_release( bmc_host: bmc_host, cookie_jar: cookie_jar, csrf_token: token as NonEmptyStr, @@ -166,6 +210,39 @@ func megarac_attach_remote_image( } } +// THE RELEASE RUNS ON EVERY PATH THROUGH THE ATTACH, which is why the attach outcome is computed +// first and the close follows it unconditionally rather than sitting inside a success arm. A +// refusal allocated the same session a success does. +func megarac_attach_then_release( + bmc_host: NonEmptyStr, + cookie_jar: NonEmptyStr, + csrf_token: NonEmptyStr, + image_name: NonEmptyStr, +) -> MegaRacAttachResult + uses net: std.resources.Network +{ + let outcome = megarac_attach_with_token( + bmc_host: bmc_host, + cookie_jar: cookie_jar, + csrf_token: csrf_token, + image_name: image_name, + ) + + closed = megarac.Media.CloseSession( + bmc_host: bmc_host, + cookie_jar: cookie_jar, + csrf_token: csrf_token, + ) + + return MegaRacAttachResult { + outcome: outcome, + session_release: if closed.success { SessionReleased } else { SessionReleaseRefused { detail: closed.body } }, + } +} + +// CONFIGURATIONS IS CONSULTED BEFORE ANYTHING IS WRITTEN, so the converged case performs no write +// at all. This is not an optimisation: start-media against an already-presented image is a write +// whose effect on a booted host is unmodelled, and the point of reading first is to not make it. func megarac_attach_with_token( bmc_host: NonEmptyStr, cookie_jar: NonEmptyStr, @@ -174,6 +251,18 @@ func megarac_attach_with_token( ) -> MegaRacAttachOutcome uses net: std.resources.Network { + presented = megarac.Media.GetRemoteConfigurations( + bmc_host: bmc_host, + cookie_jar: cookie_jar, + csrf_token: csrf_token, + ) + if presented.success == false { + return MegaRacConfigurationsUnreadable { detail: presented.body } + } + if configurations_confirm_attachment(body: presented.body, image_name: image_name) { + return MegaRacAlreadyAttached { image_name: image_name } + } + images = megarac.Media.GetRemoteImages( bmc_host: bmc_host, cookie_jar: cookie_jar, @@ -183,6 +272,11 @@ func megarac_attach_with_token( return MegaRacImagesUnreadable { detail: images.body } } + match parse_json_document(s: images.body) { + JsonDocumentUnreadable { gap: _ } => + return MegaRacImageListingUnparseable { detail: images.body } + JsonDocumentParsed { value: _ } => none + } let elements = match parse_json_document(s: images.body) { JsonDocumentParsed { value: v } => images_array_elements(doc: v) JsonDocumentUnreadable { gap: _ } => [] @@ -191,7 +285,7 @@ func megarac_attach_with_token( return MegaRacImageListingEmpty } - match image_index_named(elements: elements, image_name: image_name as String) { + return match image_index_named(elements: elements, image_name: image_name as String) { Absent => MegaRacImageNotEnumerated { image_name: image_name } Present { value: idx } => megarac_start_and_confirm( @@ -204,6 +298,50 @@ func megarac_attach_with_token( } } +// THE CONFIRMATION READS ONE EXACT ACTIVE ROW, not a substring of a body. An earlier form asked +// whether the configurations text CONTAINED the image name, which admits a stopped configuration +// that still names it, and admits an error body that happens to mention it. +// +// THE INDEX IS DELIBERATELY NOT COMPARED, and an earlier revision of this check did compare it — +// first reading image_index, which is absent here so every genuine attachment was reported +// unconfirmed, then reading media_index, which is present and agrees. The second spelling passed, +// and it was still wrong: media_index is the VIRTUAL CD DEVICE SLOT, alongside session_index, not a +// position in the image enumeration. It agreed with the resolved image_index only because this unit +// presents a single CD device, so both are 0. Comparing them is the §3 nicknaming failure — two +// unrelated concepts joined because they share a coincidence and half a name — and it would have +// started refusing the moment a controller offered a second slot, with a message blaming the image. +// +// WHAT IS COMPARED IS WHAT THE ROW ACTUALLY ASSERTS: this device is presenting THIS NAME, as CD +// media, with redirection active. That is the whole of the desired state. +fn configuration_row_confirms(row: JsonValue, image_name: NonEmptyStr) -> Bool { + let name_ok = match json_string_member(doc: row, key: "image_name") { + Absent => false + Present { value: n } => n == (image_name as String) + } + let active = match json_number_member(doc: row, key: "redirection_status") { + Absent => false + Present { value: st } => st != "0" + } + let cd_media = match json_number_member(doc: row, key: "media_type") { + Absent => false + Present { value: mt } => mt == "1" + } + name_ok && active && cd_media +} + +fn configurations_confirm_attachment(body: String, image_name: NonEmptyStr) -> Bool { + let rows = match parse_json_document(s: body) { + JsonDocumentParsed { value: v } => images_array_elements(doc: v) + JsonDocumentUnreadable { gap: _ } => [] + } + rows |> any(r => configuration_row_confirms(row: r, image_name: image_name)) +} + +// A NON-ZERO START FOLLOWED BY NO CONFIRMATION IS PENDING, NOT REFUSED, and this firmware is why: +// the same controller answers HTTP 500 on writes that take effect, and its mount daemon is +// asynchronous. Reporting MegaRacStartRefused on a status code would refuse an attach that is +// merely still mounting. The bounded read-back decides; the status code only distinguishes which +// unconfirmed state is reported. func megarac_start_and_confirm( bmc_host: NonEmptyStr, cookie_jar: NonEmptyStr, @@ -225,9 +363,12 @@ func megarac_start_and_confirm( cookie_jar: cookie_jar, csrf_token: csrf_token, ) + if configurations.success == false { + return MegaRacAttachUnconfirmed { image_name: image_name, configurations: configurations.body } + } - return if configurations.body |> contains(image_name as String) { - MegaRacAttached { image_name: image_name, image_index: 0 } + return if configurations_confirm_attachment(body: configurations.body, image_name: image_name) { + MegaRacAttached { image_name: image_name, image_index: image_index } } else { if start.success == false { MegaRacStartRefused { detail: start.body } diff --git a/dag/gunbc/machine_intake/mtcollins1_diskless_boot_receipt.dag b/dag/gunbc/machine_intake/mtcollins1_diskless_boot_receipt.dag index ce3b8ce8031..7ec4a8f3d1a 100644 --- a/dag/gunbc/machine_intake/mtcollins1_diskless_boot_receipt.dag +++ b/dag/gunbc/machine_intake/mtcollins1_diskless_boot_receipt.dag @@ -13,10 +13,25 @@ import std.content_hash { ContentHash, Sha256Digest, Sha256Hash } // carried GRUB 2.12, "EFI stub: Booting Linux Kernel...", "Linux version 6.8.0-71-generic" on // aarch64, "squashfs: version 4.0", an overlayfs upper, and systemd 255.4 as PID 1. // -// WHAT MAKES IT DISKLESS RATHER THAN MERELY BOOTED: the kernel command line is -// BOOT_IMAGE=/casper/vmlinuz, and casper mounts the squashfs read-only under a tmpfs overlay, so -// the running root is in RAM. The unit reported 261972592 KiB available, which is why a -// RAM-resident root is unremarkable here rather than a constraint. +// WHAT IS ESTABLISHED IS DISKLESS, NOT RAM-RESIDENT, and an earlier revision of this row said the +// running root was in RAM. It does not follow from what was observed. The kernel command line is +// BOOT_IMAGE=/casper/vmlinuz and casper mounts a squashfs read-only under a tmpfs overlay — which +// establishes that the WRITABLE layer is memory-backed and that no local disk was involved, and +// says nothing about where the read-only lower layer is read from. Casper's `toram` option is +// specifically what copies the read-only medium into memory before mounting the root; it is absent +// from the observed command line, so the lower layer is presumed to be read from the presented +// media on demand. +// +// THE DIFFERENCE IS LOAD-BEARING FOR THE COLO CASE, which is why it is corrected rather than left +// as a wording nicety. A RAM-resident root would mean the boot server is needed for the duration of +// the boot and can then vanish; a root still read from the virtual CD means the host depends on the +// controller's media session, and through it on the NFS export, for as long as it runs. This lane +// has not measured which of those is true — no detach-and-continue test was performed — so the +// defensible statement is the narrower one, and the colo argument cannot rest on the wider one +// until someone runs that test. +// +// The unit reported 261972592 KiB available, so a RAM-resident root would be unremarkable here if +// it were arranged; the memory is not the obstacle. data mtcollins1_diskless_boot_capture_digest: ContentHash = Sha256Hash(Sha256Digest { hex: "cefdf6c063544a08c937770f9450ce0d0ee15a06bab2bf35af9f5eaadcda7681", }) diff --git a/dag/gunbc/machine_intake/mtcollins1_media_attach.dag b/dag/gunbc/machine_intake/mtcollins1_media_attach.dag index b4519312e56..9f426126368 100644 --- a/dag/gunbc/machine_intake/mtcollins1_media_attach.dag +++ b/dag/gunbc/machine_intake/mtcollins1_media_attach.dag @@ -5,9 +5,17 @@ import std.process { ExitSuccess, ProcessExit, exit_failure } import gunbc.machine_intake_mtcollins1_access_observation { mtcollins1_endpoint } import gunbc.machine_intake_megarac_media_attach { MegaRacAttachOutcome, + MegaRacAttachResult, + SessionRelease, + SessionReleaseNotAttempted, + SessionReleaseRefused, + SessionReleased, MegaRacAttachUnconfirmed, MegaRacAttached, + MegaRacAlreadyAttached, + MegaRacConfigurationsUnreadable, MegaRacImageListingEmpty, + MegaRacImageListingUnparseable, MegaRacImageNotEnumerated, MegaRacImagesUnreadable, MegaRacSessionRefused, @@ -21,17 +29,27 @@ import gunbc.machine_intake_megarac_media_attach { // operation stays reusable by the next controller without carrying anyone's choices. data mtcollins1_diskless_image_name: NonEmptyStr = "ubuntu-24.04.3-live-server-arm64.iso" -data mtcollins1_media_cookie_jar: NonEmptyStr = "/tmp/gunbc-mtcollins1-megarac-jar" +// THE JAR IS ATTEMPT-OWNED, AND A FIXED LITERAL HERE WAS THE DEFECT. extdeps.bmc.megarac's note +// once claimed that making the jar a parameter meant an attempt owned its own — it does not: a +// parameter PERMITS a per-attempt path and this wrapper passed a constant, so two concurrent +// attempts shared one bearer file. The attempt identifier is now part of the path, so the caller +// establishes the ownership the operation cannot. +fn mtcollins1_media_cookie_jar(attempt: NonEmptyStr) -> NonEmptyStr { + join(["/tmp/gunbc-mtcollins1-megarac-jar-", attempt as String], "") as NonEmptyStr +} data mtcollins1_media_username: NonEmptyStr = "admin" -// EACH OUTCOME RENDERS ITS OWN REASON because they have different remedies, and two of them are -// easy to confuse in a way that sends an operator to the wrong place. +// EACH OUTCOME RENDERS ITS OWN REASON because they have different remedies, and the remedies are +// what an operator acts on — which is why one of these messages previously had to be retracted +// rather than reworded. It advised re-asserting mount_cd to fix an "empty" listing. That detaches +// every presented image, so on a host booted from the medium the advice destroys the running root. +// The listing was never stale; the image was simply attached, and attached images leave the +// available listing. See gunbc.machine_intake_megarac_media_attach for the measurement. // -// MegaRacImageListingEmpty means the CONTROLLER'S CACHE went stale — measured on this unit after a -// new file appeared in the export, when it stopped enumerating even a file it had listed before. -// The remedy is to re-assert mount_cd, not to look for the missing file. Reading it as "the share -// is empty" sends someone to inspect an export that is fine. +// MegaRacAlreadyAttached IS A SUCCESS AND IS RENDERED AS ONE. Converge is run repeatedly by +// construction, so the second run finding the desired state already held is the normal case, not an +// exception to it — and no write is performed on that path. // // MegaRacAttachUnconfirmed is the one state where waiting is right: start-media was accepted and // the configurations listing does not name the image yet. The controller's mount daemon is @@ -40,6 +58,9 @@ data mtcollins1_media_username: NonEmptyStr = "admin" fn mtcollins1_render_attach(outcome: MegaRacAttachOutcome) -> ProcessExit { match outcome { MegaRacAttached { image_name: _, image_index: _ } => ExitSuccess + MegaRacAlreadyAttached { image_name: _ } => ExitSuccess + MegaRacConfigurationsUnreadable { detail: d } => + exit_failure(reason: concat("mtcollins1 media attach: the presented-media listing could not be read, so whether anything is attached is UNKNOWN; refusing rather than attaching over an unknown state: ", d)) MegaRacSessionRefused { detail: d } => exit_failure(reason: concat("mtcollins1 media attach: session refused: ", d)) MegaRacSessionTokenUnreadable { detail: d } => @@ -47,9 +68,11 @@ fn mtcollins1_render_attach(outcome: MegaRacAttachOutcome) -> ProcessExit { MegaRacImagesUnreadable { detail: d } => exit_failure(reason: concat("mtcollins1 media attach: images listing unreadable: ", d)) MegaRacImageListingEmpty => - exit_failure(reason: "mtcollins1 media attach: the controller enumerated NO images; that listing is a cache and has gone stale, so re-assert mount_cd rather than concluding the share is empty") + exit_failure(reason: "mtcollins1 media attach: the image is not presented AND the controller offers no image available to attach; check that the export is reachable from the controller and holds the file. Do NOT toggle mount_cd to 'refresh' this — that detaches presented media, which on a host booted from it removes the running root") + MegaRacImageListingUnparseable { detail: d } => + exit_failure(reason: concat("mtcollins1 media attach: the images listing was not valid JSON, which is a protocol failure rather than the stale-cache case above: ", d)) MegaRacImageNotEnumerated { image_name: n } => - exit_failure(reason: concat("mtcollins1 media attach: the controller enumerated images but not '", n as String, "'")) + exit_failure(reason: concat("mtcollins1 media attach: the image is not presented, and the controller offers other images but not '", n as String, "' — so it is absent from the share as the controller sees it, rather than already in use")) MegaRacStartRefused { detail: d } => exit_failure(reason: concat("mtcollins1 media attach: start-media refused AND the configurations listing does not name the image: ", d)) MegaRacAttachUnconfirmed { image_name: n, configurations: c } => @@ -57,19 +80,72 @@ fn mtcollins1_render_attach(outcome: MegaRacAttachOutcome) -> ProcessExit { } } -func mtcollins1_attach_diskless_image(password_raw: String) -> ProcessExit +// A LEAKED SESSION IS REPORTED EVEN WHEN THE ATTACH SUCCEEDED, and it does not turn a success into +// a failure. The controller's session table is finite and this lane exhausted one service's +// sessions by never closing any; a caller that reads "attached" and ignores an unreleased session +// will keep attaching until the table fills, which presents as an unrelated 403 much later. The +// release is therefore surfaced in the reason text rather than folded into the verdict. +fn mtcollins1_render_release(standing: SessionRelease) -> String { + match standing { + SessionReleased => "" + SessionReleaseNotAttempted => " (session was never opened, so nothing to release)" + SessionReleaseRefused { detail: d } => concat(" WARNING: the session was NOT released and remains allocated on the controller: ", d) + } +} + +// THE DISCRIMINATING CONTROL, and it is the arm this operation most needs one for. Every other +// refusal here is reachable only by breaking the controller or the network; this one is reachable +// on a healthy unit and it is the arm that decides whether a name that is NOT present can be +// reported as attached. Without it, "the attach succeeded" is a claim about a run in which nothing +// could have failed — the specification-without-execution shape DESIGN §5 names. +// +// IT RUNS AGAINST THE SAME LIVE CONTROLLER AS THE POSITIVE PATH, differing in exactly one input, +// so a green positive and a red control together establish that the confirmation reads the name +// rather than reporting whatever the controller happens to be presenting. Running it costs one +// session and performs no write: the absent name is refused after the two reads, before start. +func mtcollins1_attach_absent_image_control(password_raw: String, attempt: String) -> ProcessExit uses net: std.resources.Network { if password_raw == "" { - return exit_failure(reason: "mtcollins1 media attach: no controller password supplied; refusing before any request") + return exit_failure(reason: "mtcollins1 media attach control: no controller password supplied; refusing before any request") } - return mtcollins1_render_attach( - outcome: megarac_attach_remote_image( + return mtcollins1_render_attach_result( + result: megarac_attach_remote_image( bmc_host: mtcollins1_endpoint.host, username: mtcollins1_media_username, password: password_raw as NonEmptyStr, - cookie_jar: mtcollins1_media_cookie_jar, - image_name: mtcollins1_diskless_image_name, + cookie_jar: mtcollins1_media_cookie_jar(attempt: attempt as NonEmptyStr), + image_name: "gunbc-absent-control-image.iso", ) ) } + +func mtcollins1_attach_diskless_image(password_raw: String, attempt: String) -> ProcessExit + uses net: std.resources.Network +{ + if password_raw == "" { + return exit_failure(reason: "mtcollins1 media attach: no controller password supplied; refusing before any request") + } + if attempt == "" { + return exit_failure(reason: "mtcollins1 media attach: no attempt identifier supplied; the cookie jar is attempt-owned and a shared jar is what lets two runs consume one session") + } + let result = megarac_attach_remote_image( + bmc_host: mtcollins1_endpoint.host, + username: mtcollins1_media_username, + password: password_raw as NonEmptyStr, + cookie_jar: mtcollins1_media_cookie_jar(attempt: attempt as NonEmptyStr), + image_name: mtcollins1_diskless_image_name, + ) + return mtcollins1_render_attach_result(result: result) +} + +fn mtcollins1_render_attach_result(result: MegaRacAttachResult) -> ProcessExit { + match mtcollins1_render_attach(outcome: result.outcome) { + ExitSuccess => + match result.session_release { + SessionReleased => ExitSuccess + _ => exit_failure(reason: concat("mtcollins1 media attach: the image attached, but the session was not released.", mtcollins1_render_release(standing: result.session_release))) + } + other => other + } +} diff --git a/dag/gunbc/machine_intake/mtcollins1_platform_observation.dag b/dag/gunbc/machine_intake/mtcollins1_platform_observation.dag index 355237701d5..070efb23791 100644 --- a/dag/gunbc/machine_intake/mtcollins1_platform_observation.dag +++ b/dag/gunbc/machine_intake/mtcollins1_platform_observation.dag @@ -18,11 +18,37 @@ import std.process { ExitSuccess, ProcessExit, exit_failure } // twice, on 09/04 and 09/05 controller time, meaning this predates the netboot work rather than // being caused by it. // -// WHY IT IS RANKED ABOVE THE CLOCK SKEW ALSO RECORDED IN THIS LANE: a wrong clock makes receipts -// unjoinable and is fixed by one write. A single-corded machine has no redundancy at all, cannot be -// fixed by any write, and this lane is actively making that machine something the fleet depends on -// for boot. The remedy is physical — a cord, an outlet, or a PDU port — and belongs to whoever is -// in the room. +// THE OPERATOR HAS ACCEPTED THIS STATE, and the row records the acceptance so the finding does not +// decay into noise. Left as a bare observation it reads as an unaddressed defect and the next +// reader re-raises it; recorded as accepted, with its consequence named, it stays useful. +// +// WHAT WAS ACCEPTED, precisely: the unit runs on ONE AC path. That is not an electrical problem — +// PSU1 carries 160 W and these supplies are rated far above it, so nothing is stressed. What is +// given up is redundancy: a failure anywhere in that single path, supply or cord or breaker or PDU +// outlet, powers the host off immediately rather than degrading. For a machine being developed +// against, that is a reasonable trade and the operator made it deliberately on 2026-09-06. +// +// THE CONDITION UNDER WHICH IT SHOULD BE REVISITED IS NOT A DATE BUT A ROLE CHANGE. This lane is +// making Mt. Collins bootable out of band; if it becomes a boot dependency for other machines, its +// availability becomes theirs, and a single-corded dependency is a different proposition from a +// single-corded development target. That is the trigger — the machine acquiring dependents — and it +// is stated here rather than left to be re-derived. +// +// AND A SECOND CORD IS NOT AUTOMATICALLY THE REMEDY. If both inlets land on one PDU or one breaker, +// connecting the second removes the supply as a single point of failure while keeping the feed, so +// the redundancy gained is smaller than it appears. Independent A/B feeds are where it actually +// lives. +type AcceptedPowerRisk { + accepted_at: NonEmptyStr + accepted_condition: NonEmptyStr + revisit_trigger: NonEmptyStr +} + +data mtcollins1_single_ac_path_accepted: AcceptedPowerRisk = AcceptedPowerRisk { + accepted_at: "2026-09-06", + accepted_condition: "the unit runs on one AC path; PSU2 is installed with no AC input and contributes nothing", + revisit_trigger: "this unit acquiring dependents — becoming a boot or serving dependency for other machines, at which point its availability becomes theirs", +} type PowerSupplyStanding { sensor_name: NonEmptyStr present: Bool From 459786d1a997af878bb61eb3a9ee44bfc9f1d3ed Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Sun, 6 Sep 2026 10:02:08 +0000 Subject: [PATCH 07/11] Derive the staging path so the publish is atomic by construction The fetch staged to /var/tmp and published to /srv/bmc as two independent literals. rename(2) is atomic only within one filesystem; on this host both happen to sit on the same logical volume, so the move was atomic and the module's central claim held. Measured, and by coincidence. Nothing established it. Point either literal at a separate mount -- a tmpfs /var/tmp among them -- and mv silently degrades to copy-then-unlink, publishing a growing partial file into a share the controller re-scans on its own schedule. That is exactly the window the staging split exists to close, reopened by editing a path constant, with no diagnostic anywhere. Staging is now derived from the destination, so the two share a directory by construction, therefore a filesystem, therefore the rename is atomic -- and there is no longer an input whose editing could break it. The suffix is deliberately not .iso: staging now lives inside the export, and the controller enumerates by extension, evidenced by memtest86-iso.zip sitting in this same export unlisted. So a partially-written file is not offerable as media even while being written. Also measures the published bytes again at the final path rather than carrying the staged digest forward as though it answered for them. The staged digest establishes what was fetched; it says nothing about what now occupies the path the controller reads. Reporting one as the other is execution-provenance loss. Verified by execution on a controlled fixture whose content, and so whose digest, was authored rather than measured from the tree: fetch, stage, move, and a second digest at the destination. The new mismatch arm has no executed red and the module says so -- it needs a concurrent writer or a mocked move, so it is a reachable-but-unoccupied guard rather than a decoration. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01D7PLjw5FMYdZ2GWn3FeMpR --- dag/gunbc/machine_intake/boot_image_fetch.dag | 81 ++++++++++++++++++- .../mtcollins1_boot_image_fetch.dag | 16 ++-- 2 files changed, 87 insertions(+), 10 deletions(-) diff --git a/dag/gunbc/machine_intake/boot_image_fetch.dag b/dag/gunbc/machine_intake/boot_image_fetch.dag index 17e8fb3d80c..22662d8f8c7 100644 --- a/dag/gunbc/machine_intake/boot_image_fetch.dag +++ b/dag/gunbc/machine_intake/boot_image_fetch.dag @@ -34,6 +34,7 @@ type BootImageFetchOutcome | BootImageDigestUnreadable { path: NonEmptyStr, reason: String } | BootImageDigestMismatchRefused { expected: NonEmptyStr, observed: NonEmptyStr } | BootImagePublishFailed { staged: NonEmptyStr, destination: NonEmptyStr } + | BootImagePublishedDigestMismatch { path: NonEmptyStr, expected: NonEmptyStr, observed: NonEmptyStr } // WHETHER A MEASURED DIGEST AGREES WITH THE PIN. Separated from the effect sequence so the judgment // is nameable, and returning the OBSERVED value on disagreement rather than a Bool: a refusal that @@ -66,14 +67,39 @@ fn boot_image_observed_hex(observed: Sha256FileDigest) -> NonEmptyStr { // disagreed are the evidence for why the fetch was refused, and removing them turns a diagnosable // refusal into a bare assertion. What matters for safety is that they never reach the published // path, not that they cease to exist. +// THE STAGING PATH IS DERIVED FROM THE DESTINATION, NOT SUPPLIED BESIDE IT, and that is the whole +// of this module's atomicity guarantee rather than a tidiness preference. +// +// THE GUARANTEE WAS PREVIOUSLY TRUE BY COINCIDENCE. Staging and destination were two independent +// hand-authored literals — /var/tmp/... and /srv/bmc/... — and rename(2) is atomic only WITHIN one +// filesystem. On this host both happen to sit on the same logical volume, so the move was atomic and +// the module's central claim held; measured, not assumed. But nothing established it. Point either +// literal at a separate mount, a tmpfs /var/tmp among them, and mv silently degrades to copy-then- +// unlink, which publishes a growing partial file into a share the controller re-scans on its own +// schedule. That is precisely the window the staging split exists to close, reopened by an edit to a +// path constant with no diagnostic anywhere. +// +// DERIVING THE STAGING PATH FROM THE DESTINATION MAKES THE TWO PATHS UNABLE TO DIVERGE. They share a +// directory by construction, so they share a filesystem, so the rename is atomic — and there is no +// longer an input whose editing could break it. This is the §5 move: prefer the single authority the +// realization is derived from over a check that the two agree. +// +// THE SUFFIX IS NOT .iso, AND THAT IS LOAD-BEARING because staging now lives INSIDE the export the +// controller reads. The controller enumerates by extension: this same export holds +// memtest86-iso.zip, whose name contains "iso", and the controller has never listed it. So a +// partially-written file under this suffix is not offerable as media even while it is being written. +fn boot_image_staging_path(published_path: NonEmptyStr) -> NonEmptyStr { + join([published_path as String, ".gunbc-staging"], "") as NonEmptyStr +} + func fetch_verified_boot_image( url: NonEmptyStr, - staging_path: NonEmptyStr, published_path: NonEmptyStr, expected_sha256: NonEmptyStr, ) -> BootImageFetchOutcome uses net: std.resources.Network { + let staging_path = boot_image_staging_path(published_path: published_path) let fetched = run_shell_command( command: curl_download_command(url: url as String, destination: staging_path as String), transport: LocalExec, @@ -123,6 +149,18 @@ func verify_and_publish_boot_image( // THE PUBLISH IS ITS OWN FUNC so the move is reachable only through the arm where the digest already // agreed, in the same shape the boot handoff uses to keep its power cycle behind a confirmed // read-back. A future edit cannot fall through to it. +// THE PUBLISHED BYTES ARE MEASURED AGAIN AT THE FINAL PATH, and the staged measurement is not +// carried forward as though it answered for them. Those are different questions: the staged digest +// establishes that what was FETCHED matches the pin, and says nothing about what now occupies the +// path the controller reads. Reporting the staged digest as the published one is the +// execution-provenance loss this repository keeps finding — a value consumed as though an +// observation produced it, when the observation was of something else. +// +// WHAT CAN ACTUALLY DIVERGE, so this is not a decorative re-read: the rename can land beside a file +// another writer replaces, the destination can be a symlink into somewhere else, and the export is +// group-writable by the automation account rather than exclusively held. A disagreement here means +// the export does not contain what this program published, which is the one state that must never be +// reported as an admission. func publish_verified_boot_image( staging_path: NonEmptyStr, published_path: NonEmptyStr, @@ -135,8 +173,47 @@ func publish_verified_boot_image( destination: published_path as FilePath, ) return if moved.success { - BootImageAdmitted { path: published_path, digest: digest } + confirm_published_boot_image(published_path: published_path, digest: digest) } else { BootImagePublishFailed { staged: staging_path, destination: published_path } } } + +// A SEPARATE FUNC so the admission is reachable only through a successful move, in the same shape +// the rest of this lane uses to keep a consequential step behind its precondition. +// +// THE EVIDENCE BEHIND THIS ARM IS ASYMMETRIC AND SAYING SO IS PART OF THE ROW. The positive path is +// green by execution: a controlled fixture whose content, and therefore whose digest, was authored +// rather than measured from the tree, fetched through file://, staged, moved, and re-measured at the +// final path — two DigestFile calls in the trace, the second one this arm. +// +// BootImagePublishedDigestMismatch ITSELF HAS NO EXECUTED RED. Producing one needs the destination +// to hold different bytes AFTER a successful rename, which no static fixture can arrange: it takes a +// concurrent writer or a mocked move. That makes it a guard which is REACHABLE but unoccupied — the +// mechanism exists (any second writer to a group-writable export produces it) and nothing lands in +// it today — rather than a decoration whose red cannot be authored at all. The distinction is the +// one this repository keeps having to redraw, and it decides whether the arm should exist: it +// should. NEXT-RUNG TRIGGER: a fixture harness able to interpose on shell.Move.File, at which point +// the red is authorable and this arm is owed one. +func confirm_published_boot_image( + published_path: NonEmptyStr, + digest: NonEmptyStr, +) -> BootImageFetchOutcome + uses net: std.resources.Network +{ + let republished = sha256sum_file_digest_via_shell(path: published_path) + return match republished { + Sha256FileDigestUnavailable { path: p, reason: r } => + BootImageDigestUnreadable { path: p, reason: r } + Sha256FileDigest { digest: _ } => + if boot_image_digest_agrees(observed: republished, expected: digest) { + BootImageAdmitted { path: published_path, digest: digest } + } else { + BootImagePublishedDigestMismatch { + path: published_path, + expected: digest, + observed: boot_image_observed_hex(observed: republished), + } + } + } +} diff --git a/dag/gunbc/machine_intake/mtcollins1_boot_image_fetch.dag b/dag/gunbc/machine_intake/mtcollins1_boot_image_fetch.dag index ab7bdf988ce..cafa07d6062 100644 --- a/dag/gunbc/machine_intake/mtcollins1_boot_image_fetch.dag +++ b/dag/gunbc/machine_intake/mtcollins1_boot_image_fetch.dag @@ -9,6 +9,7 @@ import gunbc.machine_intake_boot_image_fetch { BootImageFetchFailed, BootImageFetchOutcome, BootImagePublishFailed, + BootImagePublishedDigestMismatch, fetch_verified_boot_image, } @@ -24,12 +25,11 @@ data mtcollins1_boot_image_url: NonEmptyStr = "https://cdimage.ubuntu.com/releas data mtcollins1_boot_image_sha256: NonEmptyStr = "2ee2163c9b901ff5926400e80759088ff3b879982a3956c02100495b489fd555" -// STAGING IS OUTSIDE THE EXPORT, PUBLICATION IS INSIDE IT. That is the whole safety property: the -// controller enumerates /srv/bmc on its own schedule, so an artifact placed there is mountable -// before anything has measured it. Staging elsewhere means a file only ever appears in the export -// after its digest agreed. -data mtcollins1_boot_image_staging_path: NonEmptyStr = "/var/tmp/gunbc-mtcollins1-boot-image.iso" - +// THERE IS NO STAGING PATH HERE ANY MORE, and its absence is the point. This module used to author +// one beside the destination, which made the atomicity of the publish a property of two literals +// agreeing about a filesystem — true on this host and unenforced. gunbc.machine_intake_boot_image_fetch +// now derives staging from the destination, so a caller cannot express the divergence at all. What +// this module still owns is the only fact that is genuinely this host's: WHERE the export is. data mtcollins1_boot_image_published_path: NonEmptyStr = "/srv/bmc/ubuntu-24.04.3-live-server-arm64.iso" fn mtcollins1_render_fetch(outcome: BootImageFetchOutcome) -> ProcessExit { @@ -43,6 +43,8 @@ fn mtcollins1_render_fetch(outcome: BootImageFetchOutcome) -> ProcessExit { exit_failure(reason: concat("mtcollins1 boot image: REFUSED, staged bytes digest ", o as String, " does not match the pinned ", e as String, "; the staged file is left in place as the evidence and was NOT published")) BootImagePublishFailed { staged: s, destination: d } => exit_failure(reason: concat("mtcollins1 boot image: digest agreed but publishing ", s as String, " to ", d as String, " failed")) + BootImagePublishedDigestMismatch { path: p, expected: e, observed: o } => + exit_failure(reason: concat("mtcollins1 boot image: the move reported success but ", p as String, " now holds ", o as String, " rather than the verified ", e as String, "; the export does NOT contain what this program published and the unit must not be booted from it until that is explained")) } } @@ -52,7 +54,6 @@ func mtcollins1_fetch_boot_image() -> ProcessExit return mtcollins1_render_fetch( outcome: fetch_verified_boot_image( url: mtcollins1_boot_image_url, - staging_path: mtcollins1_boot_image_staging_path, published_path: mtcollins1_boot_image_published_path, expected_sha256: mtcollins1_boot_image_sha256, ) @@ -74,7 +75,6 @@ func mtcollins1_fetch_boot_image_wrong_pin_control() -> ProcessExit return mtcollins1_render_fetch( outcome: fetch_verified_boot_image( url: mtcollins1_boot_image_url, - staging_path: mtcollins1_boot_image_staging_path, published_path: mtcollins1_wrong_pin_published_path, expected_sha256: mtcollins1_wrong_pin, ) From 4cc495b8cd655f5940c94e49db77a125f0a74e0b Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Sun, 6 Sep 2026 10:10:07 +0000 Subject: [PATCH 08/11] Give the boot artifact one home so verify and attach cannot diverge The fetch module authored a published path ending ubuntu-24.04.3-live-server-arm64.iso. The attach module authored an image name spelled ubuntu-24.04.3-live-server-arm64.iso. Two literals, two files, nothing joining them. They agreed, and nothing required them to. What that admitted is the failure this lane exists to prevent. Edit one and converge verifies the digest of artifact A, publishes A into the export, then asks the controller to present artifact B -- which, if B is in the share, succeeds. Every step reports success, every read-back confirms, and the unit boots bytes nothing checked. There is no arm for it because no single operation is wrong; the defect lives in the seam between two that are each individually correct. The published path is now derived rather than authored, which makes the divergence unrepresentable instead of merely detectable. A check that the path ends with the name would have been the shape to avoid: it concedes the two are separately writable and then validates them, and is satisfiable by editing whichever one the author was not thinking about. Composing the path from the name leaves nothing to compare. The decomposition is the honest one rather than a trick to collapse a pair. A published path genuinely fuses two facts with different owners: where this host's controller reads media from, which moves when the export moves, and which artifact is served, which moves when the release moves. They were only ever one string because someone wrote them as one. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01D7PLjw5FMYdZ2GWn3FeMpR --- .../mtcollins1_boot_artifact.dag | 39 +++++++++++++++++++ .../mtcollins1_boot_image_fetch.dag | 19 +++++---- .../mtcollins1_media_attach.dag | 7 +++- 3 files changed, 57 insertions(+), 8 deletions(-) create mode 100644 dag/gunbc/machine_intake/mtcollins1_boot_artifact.dag diff --git a/dag/gunbc/machine_intake/mtcollins1_boot_artifact.dag b/dag/gunbc/machine_intake/mtcollins1_boot_artifact.dag new file mode 100644 index 00000000000..4216c620df0 --- /dev/null +++ b/dag/gunbc/machine_intake/mtcollins1_boot_artifact.dag @@ -0,0 +1,39 @@ +module gunbc.machine_intake_mtcollins1_boot_artifact + +import std.types { NonEmptyStr, String } + +// WHICH ARTIFACT MT. COLLINS BOOTS, AS ONE AUTHORITY, because it was previously two and nothing +// joined them. The fetch module authored a published path ending +// ubuntu-24.04.3-live-server-arm64.iso and the attach module authored an image name spelled +// ubuntu-24.04.3-live-server-arm64.iso, as independent literals in different files. They agreed, and +// nothing required them to. +// +// WHAT THAT ADMITTED IS THE FAILURE THIS WHOLE LANE EXISTS TO PREVENT. Edit one and the converge +// path verifies the digest of artifact A, publishes A into the export, and then asks the controller +// to present artifact B — which, if B happens to be in the share, succeeds. Every step reports +// success, every read-back confirms, and the unit boots bytes nothing checked. There is no arm for +// it because no single operation is wrong; the defect lives in the seam between two that are each +// individually correct. +// +// THE PUBLISHED PATH IS NOW DERIVED RATHER THAN AUTHORED, which is what makes the divergence +// unrepresentable instead of merely detectable. A check that the path ENDS WITH the name would have +// been the §5 shape to avoid: it concedes the two are separately writable and then validates them, +// and it is satisfiable by editing whichever of the two the author was not thinking about. Composing +// the path from the name leaves nothing to compare. +// +// AND THE DECOMPOSITION IS THE HONEST ONE rather than a trick to collapse a pair. A published path +// genuinely fuses two facts of different ownership: WHERE this host's controller reads media from, +// which is a property of the export and moves when the export moves, and WHICH artifact is being +// served, which is a property of the release and moves when the release moves. They were only ever +// one string because someone wrote them as one. +data mtcollins1_boot_export_dir: NonEmptyStr = "/srv/bmc" + +data mtcollins1_diskless_image_name: NonEmptyStr = "ubuntu-24.04.3-live-server-arm64.iso" + +fn mtcollins1_boot_image_published_path() -> NonEmptyStr { + join([ + mtcollins1_boot_export_dir as String, + "/", + mtcollins1_diskless_image_name as String, + ], "") as NonEmptyStr +} diff --git a/dag/gunbc/machine_intake/mtcollins1_boot_image_fetch.dag b/dag/gunbc/machine_intake/mtcollins1_boot_image_fetch.dag index cafa07d6062..4f6b76d9530 100644 --- a/dag/gunbc/machine_intake/mtcollins1_boot_image_fetch.dag +++ b/dag/gunbc/machine_intake/mtcollins1_boot_image_fetch.dag @@ -1,6 +1,7 @@ module gunbc.machine_intake_mtcollins1_boot_image_fetch import std.types { NonEmptyStr, String } +import gunbc.machine_intake_mtcollins1_boot_artifact { mtcollins1_boot_image_published_path } import std.process { ExitSuccess, ProcessExit, exit_failure } import gunbc.machine_intake_boot_image_fetch { BootImageAdmitted, @@ -25,12 +26,16 @@ data mtcollins1_boot_image_url: NonEmptyStr = "https://cdimage.ubuntu.com/releas data mtcollins1_boot_image_sha256: NonEmptyStr = "2ee2163c9b901ff5926400e80759088ff3b879982a3956c02100495b489fd555" -// THERE IS NO STAGING PATH HERE ANY MORE, and its absence is the point. This module used to author -// one beside the destination, which made the atomicity of the publish a property of two literals -// agreeing about a filesystem — true on this host and unenforced. gunbc.machine_intake_boot_image_fetch -// now derives staging from the destination, so a caller cannot express the divergence at all. What -// this module still owns is the only fact that is genuinely this host's: WHERE the export is. -data mtcollins1_boot_image_published_path: NonEmptyStr = "/srv/bmc/ubuntu-24.04.3-live-server-arm64.iso" +// NEITHER THE STAGING PATH NOR THE PUBLISHED PATH IS AUTHORED HERE ANY MORE, and both absences are +// the point. The staging path used to sit beside the destination as an independent literal, which +// made the atomicity of the publish a property of two constants agreeing about a filesystem — true +// on this host and unenforced; gunbc.machine_intake_boot_image_fetch now derives it from the +// destination. The published path used to be a literal spelling an image name that a DIFFERENT +// module also spelled, with nothing joining them; +// gunbc.machine_intake_mtcollins1_boot_artifact now owns that name and composes the path from it. +// +// SO THIS MODULE AUTHORS ONLY WHAT IS GENUINELY ITS OWN: where the release is fetched from and what +// its bytes must hash to. Everything about WHERE it lands is derived. fn mtcollins1_render_fetch(outcome: BootImageFetchOutcome) -> ProcessExit { match outcome { @@ -54,7 +59,7 @@ func mtcollins1_fetch_boot_image() -> ProcessExit return mtcollins1_render_fetch( outcome: fetch_verified_boot_image( url: mtcollins1_boot_image_url, - published_path: mtcollins1_boot_image_published_path, + published_path: mtcollins1_boot_image_published_path(), expected_sha256: mtcollins1_boot_image_sha256, ) ) diff --git a/dag/gunbc/machine_intake/mtcollins1_media_attach.dag b/dag/gunbc/machine_intake/mtcollins1_media_attach.dag index 9f426126368..86926b7639c 100644 --- a/dag/gunbc/machine_intake/mtcollins1_media_attach.dag +++ b/dag/gunbc/machine_intake/mtcollins1_media_attach.dag @@ -3,6 +3,7 @@ module gunbc.machine_intake_mtcollins1_media_attach import std.types { NonEmptyStr, String } import std.process { ExitSuccess, ProcessExit, exit_failure } import gunbc.machine_intake_mtcollins1_access_observation { mtcollins1_endpoint } +import gunbc.machine_intake_mtcollins1_boot_artifact { mtcollins1_diskless_image_name } import gunbc.machine_intake_megarac_media_attach { MegaRacAttachOutcome, MegaRacAttachResult, @@ -27,7 +28,11 @@ import gunbc.machine_intake_megarac_media_attach { // THE POLICY LAYER FOR ONE HOST'S MEDIA ATTACH, separate from the operation for the same reason the // boot-handoff wrappers are: which image this unit should carry is a workflow decision, and the // operation stays reusable by the next controller without carrying anyone's choices. -data mtcollins1_diskless_image_name: NonEmptyStr = "ubuntu-24.04.3-live-server-arm64.iso" +// +// THE IMAGE NAME IS IMPORTED RATHER THAN AUTHORED, and it used to be authored here. That made this +// module and the fetch module two authorities for one fact — the artifact this host boots — which +// admitted verifying one image and attaching another, with every individual step reporting success. +// See gunbc.machine_intake_mtcollins1_boot_artifact. // THE JAR IS ATTEMPT-OWNED, AND A FIXED LITERAL HERE WAS THE DEFECT. extdeps.bmc.megarac's note // once claimed that making the jar a parameter meant an attempt owned its own — it does not: a From 5bb589c79cc975d5efc54d4cc0e9b4f01f89f363 Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Sun, 6 Sep 2026 10:44:10 +0000 Subject: [PATCH 09/11] Make the export-ownership requirement decide the publish boot_image_export_ownership described who may own the boot export and was consulted by nothing. The publish called mv directly, so the program published whenever the operating system's permissions happened to allow it -- as root, or into a world-writable export -- while a probe asserted the policy against fixtures. A requirement with no causal position is a description, and describing a wall is not building one. The reason it had no causal position was that nothing could observe ownership: there was no modeled stat. extdeps.tools.coreutils_stat adds one, folding stat(1) into a typed observation whose unreadable arm is a refusal rather than an ownership of nobody -- collapsing those would make a deleted or unreachable export the most permissive input the admission accepts. parse_int's Absent arm refuses rather than defaulting to 0, because 0 is root: an unparseable uid would otherwise arrive at the admission as the most privileged answer available. The judgment now takes a uid and a name rather than a PosixUser. That is a correction, not a convenience: a PosixUser carries a login shell, which is a passwd fact, while the owner of a directory is a stat fact. Asking for a PosixUser meant the only way to feed this from a real observation was to invent a shell nothing had seen -- fabricating a field to satisfy a type, in the function deciding who may write to the boot export. The check sits after the digest agrees and before the move, so the two refusals stay distinguishable: checking ownership first would make every wrong-pin fetch on an unwritable export report an environment problem and hide the artifact defect. The fetch now takes the export directory and image name separately and composes the path itself, because the admission is about the directory while the write is about the path. Taking both as parameters would let a caller admit one place and write to another -- the same seam defect as verifying one artifact and attaching a different one. Verified live on srv2, four arms of the observation on real paths: /srv/bmc admitted, / refused as root-owned, /home/briansrls refused as the operator principal, a missing path refused as unobservable. And the gate itself: the correct pin against a root-owned export verifies the artifact and is then refused at the write boundary, with the filesystem confirming nothing was published and the staged bytes kept as evidence. The wrong-pin control still refuses on the digest, so the ordering holds. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01D7PLjw5FMYdZ2GWn3FeMpR --- dag/extdeps/tools/coreutils_stat.dag | 134 ++++++++++++++++++ .../boot_image_export_ownership.dag | 119 ++++++++++++++-- dag/gunbc/machine_intake/boot_image_fetch.dag | 53 ++++++- .../mtcollins1_boot_artifact.dag | 13 +- .../mtcollins1_boot_image_fetch.dag | 37 ++++- 5 files changed, 335 insertions(+), 21 deletions(-) create mode 100644 dag/extdeps/tools/coreutils_stat.dag diff --git a/dag/extdeps/tools/coreutils_stat.dag b/dag/extdeps/tools/coreutils_stat.dag new file mode 100644 index 00000000000..099ed125c61 --- /dev/null +++ b/dag/extdeps/tools/coreutils_stat.dag @@ -0,0 +1,134 @@ +module extdeps.tools.coreutils_stat + +import extdeps.external_authority { ExternalAuthority } +import extdeps.uri { Uri, Https } +import extdeps.access.posix { FileOwnership } +import std.types { Bool, Int, List, NonEmptyStr, String } +import std.algebra { trim } + +// OBSERVING WHO OWNS A PATH, as a modeled operation. extdeps.access.posix owns the CONCEPT -- +// FileOwnership is a uid and a gid, cited to the Open Group's sys/stat.h -- and this module is one +// transport realizing an observation of it, through the coreutils stat(1) utility. The split is the +// usual one: a future realization reading stat(2) directly, or reading a remote host's ownership +// over a transport, binds to the same shape rather than replacing this. +data extdeps_external_authority_anchor: ExternalAuthority = ExternalAuthority { + uri: Uri { + scheme: Https + locator: "www.gnu.org/software/coreutils/manual/html_node/stat-invocation.html" + } +} + +// THE FORMAT STRING IS THE CONTRACT AND IT IS DELIBERATELY NUMERIC-FIRST. %u and %g are the uid and +// gid; %U and %G are the names the local name service resolves them to. Both are taken because they +// answer different questions and a consumer needs both: an admission decision that keys on the +// NUMBER is the one that stays correct when a name lookup fails or resolves differently on another +// host, while the NAME is what a refusal has to print for anyone to act on it. +// +// A COLON IS SAFE AS THE FIELD SEPARATOR HERE, unlike in most path contexts, because none of the +// four fields can contain one: uid and gid are decimal, and a POSIX login name is restricted to the +// portable filename character set, which excludes the colon. That is a property of the fields, not +// an assumption about the data, which is why the separator is not a configurable parameter. +service coreutils.Stat { + operation PathOwnership { + input { path: NonEmptyStr } + output { + line: String from "stdout" + success: Bool from "exit_success" + } + readonly + transport shell { argv: ["stat", "-c", "%u:%g:%U:%G", "{path}"] } + exit { + 0 => Unit + nonzero => String "stat could not read the path" + } + } +} + +// AN UNREADABLE PATH IS A REFUSAL, NOT AN OWNERSHIP OF NOBODY, for the same reason an unreadable +// digest is not the digest of nothing. A path that could not be observed must never be folded into +// a record that reads as an observation, because the whole value of an ownership admission is that +// something looked. +type PathOwnershipObservation + = PathOwnershipObserved { + path: NonEmptyStr + ownership: FileOwnership + owner_name: NonEmptyStr + group_name: NonEmptyStr + } + | PathOwnershipUnobservable { path: NonEmptyStr, reason: String } + +fn stat_field(line: String, index: Int) -> String? { + get(xs: split(s: trim(s: line), delimiter: ":"), index: index) +} + +// parse_int RETURNS AN OPTION AND THE ABSENT ARM IS TAKEN SERIOUSLY, rather than being collapsed +// to a zero the way the one other consumer of it in this repository does. A uid defaulted to 0 is +// not a neutral placeholder here: 0 IS ROOT, so an unparseable field would arrive at the admission +// as the single most privileged answer available. That is the fabricated-plausible-output failure +// aimed directly at the check this observation exists to feed. +// +// THE PARSE REFUSES ON ANY MISSING OR EMPTY FIELD rather than defaulting one. A missing owner name +// defaulted to the empty string would compare unequal to every declared principal and so present as +// a WRONG-OWNER refusal -- naming a principal, sending the reader to check an account -- when what +// actually happened is that the observation failed. Those have different remedies, so they get +// different arms. +fn coreutils_stat_path_ownership(path: NonEmptyStr) -> PathOwnershipObservation +{ + let observed = coreutils.Stat.PathOwnership(path: path) + if observed.success == false { + PathOwnershipUnobservable { path: path, reason: "stat could not read the path" } + } else { + match stat_field(line: observed.line, index: 0) { + Absent => PathOwnershipUnobservable { path: path, reason: "stat printed no uid field" } + Present { value: uid_text } => + match stat_field(line: observed.line, index: 1) { + Absent => PathOwnershipUnobservable { path: path, reason: "stat printed no gid field" } + Present { value: gid_text } => + match stat_field(line: observed.line, index: 2) { + Absent => PathOwnershipUnobservable { path: path, reason: "stat printed no owner-name field" } + Present { value: owner } => + match stat_field(line: observed.line, index: 3) { + Absent => PathOwnershipUnobservable { path: path, reason: "stat printed no group-name field" } + Present { value: group } => + coreutils_stat_ownership_record( + path: path, + uid_text: uid_text, + gid_text: gid_text, + owner: owner, + group: group, + ) + } + } + } + } + } +} + +fn coreutils_stat_ownership_record( + path: NonEmptyStr, + uid_text: String, + gid_text: String, + owner: String, + group: String, +) -> PathOwnershipObservation { + if owner.length() == 0 || group.length() == 0 { + PathOwnershipUnobservable { path: path, reason: "stat printed an empty ownership name field" } + } else { + match parse_int(s: uid_text) { + Absent => + PathOwnershipUnobservable { path: path, reason: "stat printed a uid field that is not an integer" } + Present { value: uid } => + match parse_int(s: gid_text) { + Absent => + PathOwnershipUnobservable { path: path, reason: "stat printed a gid field that is not an integer" } + Present { value: gid } => + PathOwnershipObserved { + path: path, + ownership: FileOwnership { uid: uid, gid: gid }, + owner_name: owner as NonEmptyStr, + group_name: group as NonEmptyStr, + } + } + } + } +} diff --git a/dag/gunbc/machine_intake/boot_image_export_ownership.dag b/dag/gunbc/machine_intake/boot_image_export_ownership.dag index aae6648c89a..51fd1122a5b 100644 --- a/dag/gunbc/machine_intake/boot_image_export_ownership.dag +++ b/dag/gunbc/machine_intake/boot_image_export_ownership.dag @@ -2,12 +2,18 @@ module gunbc.machine_intake_boot_image_export_ownership import std.types { Bool, Int, NonEmptyStr, String } import std.process { ExitSuccess, ProcessExit, exit_failure } +import extdeps.tools.coreutils_stat { + PathOwnershipObservation, + PathOwnershipObserved, + PathOwnershipUnobservable, + coreutils_stat_path_ownership, +} import extdeps.access.posix { FileOwnership, LoginShellObserved, LoginShellUnobserved, PosixUser, - posix_user_is_root, + posix_root_uid, } import gunbc.fleet_posix_accounts { FleetAccountAutomation, @@ -100,6 +106,7 @@ type ExportOwnershipStanding | ExportOwnershipIsRoot | ExportOwnershipWrongPrincipal { observed: FleetPosixAccountKey, required: FleetPosixAccountKey } | ExportOwnershipUnrecognizedAccount { observed: NonEmptyStr } + | ExportOwnershipUnobservable { path: NonEmptyStr, reason: String } // ROOT IS ITS OWN ARM rather than one case of a wrong owner, because the remedy differs. A // wrong-but-unprivileged owner is a chgrp away; root means the publish step would have to escalate, @@ -134,16 +141,24 @@ fn fleet_account_key_matches(a: FleetPosixAccountKey, b: FleetPosixAccountKey) - // wrong declared principal: the second is a chgrp, the first means someone provisioned an account // this model has never heard of, and reporting it as "wrong principal" would name a required // principal while saying nothing about what is actually there. +// THE JUDGMENT TAKES A UID AND A NAME RATHER THAN A PosixUser, and that is a correction rather than +// a convenience. A PosixUser carries a login shell, which is a passwd-entry fact; the owner of a +// DIRECTORY is a uid and a gid, which is a stat fact. Asking for a PosixUser meant the only way to +// feed this judgment from a real observation was to invent a shell the observation never saw -- +// fabricating a field to satisfy a type, in the one function whose entire purpose is to decide +// whether something may write to the boot export. The fixtures still build PosixUsers, because for +// a FIXTURE the whole account is genuinely known; they now pass the two fields this asks for. fn export_ownership_standing( - observed: PosixUser, + observed_uid: Int, + observed_name: NonEmptyStr, observed_key: FleetPosixAccountKey?, requirement: BootImageExportOwnershipRequirement, ) -> ExportOwnershipStanding { - if posix_user_is_root(user: observed) { + if observed_uid == posix_root_uid { ExportOwnershipIsRoot } else { match observed_key { - Absent => ExportOwnershipUnrecognizedAccount { observed: observed.name } + Absent => ExportOwnershipUnrecognizedAccount { observed: observed_name } Present { value: k } => if fleet_account_key_matches(a: k, b: requirement.required_owner) { ExportOwnershipSatisfied @@ -193,24 +208,29 @@ fn export_ownership_label(standing: ExportOwnershipStanding) -> String { ExportOwnershipIsRoot => "root" ExportOwnershipWrongPrincipal { observed: _, required: _ } => "wrong-principal" ExportOwnershipUnrecognizedAccount { observed: _ } => "unrecognized" + ExportOwnershipUnobservable { path: _, reason: _ } => "unobservable" } } func boot_image_export_ownership_probe() -> ProcessExit { let on_root = export_ownership_label(standing: export_ownership_standing( - observed: posix_root_fixture, + observed_uid: posix_root_fixture.uid, + observed_name: posix_root_fixture.name, observed_key: Present { value: FleetAccountRoot }, requirement: boot_image_export_ownership_requirement)) let on_automation = export_ownership_label(standing: export_ownership_standing( - observed: posix_automation_fixture, + observed_uid: posix_automation_fixture.uid, + observed_name: posix_automation_fixture.name, observed_key: Present { value: FleetAccountAutomation }, requirement: boot_image_export_ownership_requirement)) let on_interactive = export_ownership_label(standing: export_ownership_standing( - observed: posix_interactive_fixture, + observed_uid: posix_interactive_fixture.uid, + observed_name: posix_interactive_fixture.name, observed_key: Present { value: FleetAccountOperator }, requirement: boot_image_export_ownership_requirement)) let on_unknown = export_ownership_label(standing: export_ownership_standing( - observed: posix_interactive_fixture, + observed_uid: posix_interactive_fixture.uid, + observed_name: posix_interactive_fixture.name, observed_key: none, requirement: boot_image_export_ownership_requirement)) @@ -228,3 +248,86 @@ func boot_image_export_ownership_probe() -> ProcessExit { } return ExitSuccess } + +// THE NAME ROSTER FOR THIS EXPORT'S HOST, and it lives here for the reason fleet_account_key_label +// above lives here: one consumer does not justify editing the fleet authority, and it should move +// to gunbc.fleet_posix_accounts the moment a second one appears. What it maps is genuinely a fleet +// fact -- which local login realizes which declared principal on this host -- and gunbc-fleet is +// read from srv2 rather than invented, as is the operator login beside it. +// +// AN UNLISTED NAME RESOLVES TO ABSENT, WHICH IS A REFUSAL AND NOT A MISS. Absent flows to +// ExportOwnershipUnrecognizedAccount, so an account this roster has never heard of stops the +// publish rather than being quietly treated as some default principal. +fn srv2_account_key_for_name(name: NonEmptyStr) -> FleetPosixAccountKey? { + if (name as String) == "root" { + Present { value: FleetAccountRoot } + } else { + if (name as String) == "gunbc-fleet" { + Present { value: FleetAccountAutomation } + } else { + if (name as String) == "briansrls" { + Present { value: FleetAccountOperator } + } else { + none + } + } + } +} + +// THE CAUSAL ENTRY POINT, and its absence was the whole of the review finding this closes. Until +// now this module described who ought to own the export and was consulted by nothing: the publish +// called mv directly, so the program published whenever the operating system's permissions happened +// to allow it -- under root, or into a world-writable export -- and the requirement was a +// description checked by a probe against fixtures rather than a wall. +// +// AN UNREADABLE PATH REFUSES rather than being read as an absent problem. That is the empty- +// observation narrow this repository keeps finding: "I could not see who owns this" and "nobody +// problematic owns this" are different states, and collapsing them would make an unreachable or +// deleted export the most permissive input this function accepts. +func observed_export_ownership_standing(path: NonEmptyStr) -> ExportOwnershipStanding { + match coreutils_stat_path_ownership(path: path) { + PathOwnershipUnobservable { path: p, reason: r } => + ExportOwnershipUnobservable { path: p, reason: r } + PathOwnershipObserved { path: _, ownership: own, owner_name: owner, group_name: _ } => + export_ownership_standing( + observed_uid: own.uid, + observed_name: owner, + observed_key: srv2_account_key_for_name(name: owner), + requirement: boot_image_export_ownership_requirement, + ) + } +} + +fn export_ownership_admits_publication(standing: ExportOwnershipStanding) -> Bool { + match standing { + ExportOwnershipSatisfied => true + _ => false + } +} + +// THE REFUSAL TEXT NAMES WHAT WAS SEEN AND WHAT IS REQUIRED, because an admission refusal that says +// only "denied" sends the reader to the machine to find out what this function already knew. +fn export_ownership_refusal_reason(path: NonEmptyStr, standing: ExportOwnershipStanding) -> String { + match standing { + ExportOwnershipSatisfied => "" + ExportOwnershipIsRoot => + concat("the export ", path as String, " is owned by root, so publishing into it would require the publish step to escalate; that is refused by design rather than a misconfiguration to correct") + ExportOwnershipWrongPrincipal { observed: o, required: r } => + concat("the export ", path as String, " is owned by the ", fleet_account_key_label(key: o), " principal but this requirement names ", fleet_account_key_label(key: r)) + ExportOwnershipUnrecognizedAccount { observed: o } => + concat("the export ", path as String, " is owned by ", o as String, ", an account outside this fleet's declared roster") + ExportOwnershipUnobservable { path: p, reason: r } => + concat("the ownership of ", p as String, " could not be observed, so whether it may be published into is UNKNOWN: ", r) + } +} + +// THE OBSERVED STANDING, RENDERED, so the causal path can be executed on a real host rather than +// only reasoned about. This reports; it does not gate. The gate is at the write boundary. +func boot_image_export_ownership_observed_probe(path: String) -> ProcessExit { + let standing = observed_export_ownership_standing(path: path as NonEmptyStr) + return if export_ownership_admits_publication(standing: standing) { + ExitSuccess + } else { + exit_failure(reason: concat("export ownership REFUSES publication: ", export_ownership_refusal_reason(path: path as NonEmptyStr, standing: standing))) + } +} diff --git a/dag/gunbc/machine_intake/boot_image_fetch.dag b/dag/gunbc/machine_intake/boot_image_fetch.dag index 22662d8f8c7..3e399712a9c 100644 --- a/dag/gunbc/machine_intake/boot_image_fetch.dag +++ b/dag/gunbc/machine_intake/boot_image_fetch.dag @@ -1,6 +1,11 @@ module gunbc.machine_intake_boot_image_fetch import std.types { Bool, FilePath, NonEmptyStr, String } +import gunbc.machine_intake_boot_image_export_ownership { + export_ownership_admits_publication, + export_ownership_refusal_reason, + observed_export_ownership_standing, +} import extdeps.shell import extdeps.tools.curl { curl_download_command } import gunbc.command_runner { run_shell_command } @@ -35,6 +40,7 @@ type BootImageFetchOutcome | BootImageDigestMismatchRefused { expected: NonEmptyStr, observed: NonEmptyStr } | BootImagePublishFailed { staged: NonEmptyStr, destination: NonEmptyStr } | BootImagePublishedDigestMismatch { path: NonEmptyStr, expected: NonEmptyStr, observed: NonEmptyStr } + | BootImagePublicationNotAdmitted { export_dir: NonEmptyStr, reason: String } // WHETHER A MEASURED DIGEST AGREES WITH THE PIN. Separated from the effect sequence so the judgment // is nameable, and returning the OBSERVED value on disagreement rather than a Bool: a refusal that @@ -92,13 +98,20 @@ fn boot_image_staging_path(published_path: NonEmptyStr) -> NonEmptyStr { join([published_path as String, ".gunbc-staging"], "") as NonEmptyStr } +// THE EXPORT DIRECTORY AND THE IMAGE NAME ARE TAKEN SEPARATELY, and the published path is composed +// here, because the ADMISSION is about the directory while the WRITE is about the path. Taking a +// composed path and a directory as two parameters would let a caller pass a path that does not live +// in the directory that was admitted -- admitting one place and writing to another, which is the +// same seam defect as verifying one artifact and attaching a different one. func fetch_verified_boot_image( url: NonEmptyStr, - published_path: NonEmptyStr, + export_dir: NonEmptyStr, + image_name: NonEmptyStr, expected_sha256: NonEmptyStr, ) -> BootImageFetchOutcome uses net: std.resources.Network { + let published_path = join([export_dir as String, "/", image_name as String], "") as NonEmptyStr let staging_path = boot_image_staging_path(published_path: published_path) let fetched = run_shell_command( command: curl_download_command(url: url as String, destination: staging_path as String), @@ -109,6 +122,7 @@ func fetch_verified_boot_image( verify_and_publish_boot_image( staging_path: staging_path, published_path: published_path, + export_dir: export_dir, expected_sha256: expected_sha256, ) _ => BootImageFetchFailed { url: url } @@ -122,6 +136,7 @@ func fetch_verified_boot_image( func verify_and_publish_boot_image( staging_path: NonEmptyStr, published_path: NonEmptyStr, + export_dir: NonEmptyStr, expected_sha256: NonEmptyStr, ) -> BootImageFetchOutcome uses net: std.resources.Network @@ -132,9 +147,10 @@ func verify_and_publish_boot_image( BootImageDigestUnreadable { path: p, reason: r } Sha256FileDigest { digest: _ } => if boot_image_digest_agrees(observed: measured, expected: expected_sha256) { - publish_verified_boot_image( + admit_then_publish_boot_image( staging_path: staging_path, published_path: published_path, + export_dir: export_dir, digest: boot_image_observed_hex(observed: measured), ) } else { @@ -149,6 +165,39 @@ func verify_and_publish_boot_image( // THE PUBLISH IS ITS OWN FUNC so the move is reachable only through the arm where the digest already // agreed, in the same shape the boot handoff uses to keep its power cycle behind a confirmed // read-back. A future edit cannot fall through to it. +// THE OWNERSHIP ADMISSION SITS AT THE WRITE BOUNDARY, which is the only place it means anything. +// gunbc.machine_intake_boot_image_export_ownership previously described who may own the export and +// was consulted by NOTHING: this module called mv directly, so the program published whenever the +// operating system's permissions happened to allow it -- as root, or into a world-writable export -- +// while a probe asserted the policy against fixtures. A requirement with no causal position is a +// description, and describing a wall is not building one. +// +// IT IS CHECKED AFTER THE DIGEST AGREES AND BEFORE THE MOVE, deliberately, so the two refusals stay +// distinguishable. Checking it first would make every wrong-pin fetch on an unwritable export report +// an ownership problem, hiding the artifact defect behind an environment one. +func admit_then_publish_boot_image( + staging_path: NonEmptyStr, + published_path: NonEmptyStr, + export_dir: NonEmptyStr, + digest: NonEmptyStr, +) -> BootImageFetchOutcome + uses net: std.resources.Network +{ + let standing = observed_export_ownership_standing(path: export_dir) + return if export_ownership_admits_publication(standing: standing) { + publish_verified_boot_image( + staging_path: staging_path, + published_path: published_path, + digest: digest, + ) + } else { + BootImagePublicationNotAdmitted { + export_dir: export_dir, + reason: export_ownership_refusal_reason(path: export_dir, standing: standing), + } + } +} + // THE PUBLISHED BYTES ARE MEASURED AGAIN AT THE FINAL PATH, and the staged measurement is not // carried forward as though it answered for them. Those are different questions: the staged digest // establishes that what was FETCHED matches the pin, and says nothing about what now occupies the diff --git a/dag/gunbc/machine_intake/mtcollins1_boot_artifact.dag b/dag/gunbc/machine_intake/mtcollins1_boot_artifact.dag index 4216c620df0..98e7e833d69 100644 --- a/dag/gunbc/machine_intake/mtcollins1_boot_artifact.dag +++ b/dag/gunbc/machine_intake/mtcollins1_boot_artifact.dag @@ -28,12 +28,11 @@ import std.types { NonEmptyStr, String } // one string because someone wrote them as one. data mtcollins1_boot_export_dir: NonEmptyStr = "/srv/bmc" +// THE COMPOSITION ITSELF LIVES IN THE FETCH OPERATION, not here, and this module deliberately +// exports only the two atoms. A composed path helper beside them would be a third row derived from +// the first two, and the operation composes anyway because it needs the DIRECTORY separately -- the +// ownership admission is about the directory while the write is about the path. Exporting both the +// atoms and their composition would put two spellings of one fact back in circulation, which is the +// condition this module was created to end. data mtcollins1_diskless_image_name: NonEmptyStr = "ubuntu-24.04.3-live-server-arm64.iso" -fn mtcollins1_boot_image_published_path() -> NonEmptyStr { - join([ - mtcollins1_boot_export_dir as String, - "/", - mtcollins1_diskless_image_name as String, - ], "") as NonEmptyStr -} diff --git a/dag/gunbc/machine_intake/mtcollins1_boot_image_fetch.dag b/dag/gunbc/machine_intake/mtcollins1_boot_image_fetch.dag index 4f6b76d9530..e91ef72c4f0 100644 --- a/dag/gunbc/machine_intake/mtcollins1_boot_image_fetch.dag +++ b/dag/gunbc/machine_intake/mtcollins1_boot_image_fetch.dag @@ -1,7 +1,7 @@ module gunbc.machine_intake_mtcollins1_boot_image_fetch import std.types { NonEmptyStr, String } -import gunbc.machine_intake_mtcollins1_boot_artifact { mtcollins1_boot_image_published_path } +import gunbc.machine_intake_mtcollins1_boot_artifact { mtcollins1_boot_export_dir, mtcollins1_diskless_image_name } import std.process { ExitSuccess, ProcessExit, exit_failure } import gunbc.machine_intake_boot_image_fetch { BootImageAdmitted, @@ -11,6 +11,7 @@ import gunbc.machine_intake_boot_image_fetch { BootImageFetchOutcome, BootImagePublishFailed, BootImagePublishedDigestMismatch, + BootImagePublicationNotAdmitted, fetch_verified_boot_image, } @@ -48,6 +49,8 @@ fn mtcollins1_render_fetch(outcome: BootImageFetchOutcome) -> ProcessExit { exit_failure(reason: concat("mtcollins1 boot image: REFUSED, staged bytes digest ", o as String, " does not match the pinned ", e as String, "; the staged file is left in place as the evidence and was NOT published")) BootImagePublishFailed { staged: s, destination: d } => exit_failure(reason: concat("mtcollins1 boot image: digest agreed but publishing ", s as String, " to ", d as String, " failed")) + BootImagePublicationNotAdmitted { export_dir: d, reason: r } => + exit_failure(reason: concat("mtcollins1 boot image: the artifact verified but publication into ", d as String, " was NOT admitted: ", r)) BootImagePublishedDigestMismatch { path: p, expected: e, observed: o } => exit_failure(reason: concat("mtcollins1 boot image: the move reported success but ", p as String, " now holds ", o as String, " rather than the verified ", e as String, "; the export does NOT contain what this program published and the unit must not be booted from it until that is explained")) } @@ -59,7 +62,8 @@ func mtcollins1_fetch_boot_image() -> ProcessExit return mtcollins1_render_fetch( outcome: fetch_verified_boot_image( url: mtcollins1_boot_image_url, - published_path: mtcollins1_boot_image_published_path(), + export_dir: mtcollins1_boot_export_dir, + image_name: mtcollins1_diskless_image_name, expected_sha256: mtcollins1_boot_image_sha256, ) ) @@ -72,7 +76,7 @@ func mtcollins1_fetch_boot_image() -> ProcessExit // unverified artifact where the unit would boot it. data mtcollins1_wrong_pin: NonEmptyStr = "0000000000000000000000000000000000000000000000000000000000000000" -data mtcollins1_wrong_pin_published_path: NonEmptyStr = "/var/tmp/gunbc-mtcollins1-wrongpin-published.iso" +data mtcollins1_wrong_pin_export_dir: NonEmptyStr = "/var/tmp" func mtcollins1_fetch_boot_image_wrong_pin_control() -> ProcessExit uses net: std.resources.Network @@ -80,8 +84,33 @@ func mtcollins1_fetch_boot_image_wrong_pin_control() -> ProcessExit return mtcollins1_render_fetch( outcome: fetch_verified_boot_image( url: mtcollins1_boot_image_url, - published_path: mtcollins1_wrong_pin_published_path, + export_dir: mtcollins1_wrong_pin_export_dir, + image_name: mtcollins1_diskless_image_name, expected_sha256: mtcollins1_wrong_pin, ) ) } + +// THE DISCRIMINATING RED FOR THE OWNERSHIP GATE, and it is a different one from the wrong-pin +// control above. That control refuses at the DIGEST, so it never reaches the write boundary and +// establishes nothing about who may write there. This one supplies the CORRECT pin -- so the +// artifact verifies and the run arrives at the publish with everything about the bytes in order -- +// and points the export at a directory owned by root. If it succeeds, the admission is decorative. +// +// /var/tmp IS THE RIGHT CHOICE rather than an arbitrary one: it is root-owned on this host and +// world-writable by mode, which is exactly the configuration that makes an ownership check look +// unnecessary. The move would succeed there. Only the admission stops it. +data mtcollins1_unowned_export_dir: NonEmptyStr = "/var/tmp" + +func mtcollins1_fetch_boot_image_unowned_export_control() -> ProcessExit + uses net: std.resources.Network +{ + return mtcollins1_render_fetch( + outcome: fetch_verified_boot_image( + url: mtcollins1_boot_image_url, + export_dir: mtcollins1_unowned_export_dir, + image_name: mtcollins1_diskless_image_name, + expected_sha256: mtcollins1_boot_image_sha256, + ) + ) +} From e43eb5508621cc6c4dee5f6d11574b2afb72fc50 Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Sun, 6 Sep 2026 10:49:35 +0000 Subject: [PATCH 10/11] Temper the detach hazard to what was actually measured The previous commit retracted an overclaim and, in the same paragraph, made one: it said toggling mount_cd pulls the running root out from under a booted host. That is the mechanism -- the presented medium is withdrawn, and casper reads the read-only lower layer from it on demand rather than from RAM -- but it is not what was observed. The toggle was performed on this unit while it was booted from the medium, and the running installer survived: measured afterwards over serial console, still interactive at its own menu. The controller is configured rmedia_retry_count 3 / interval 15, and the system was idle at a prompt rather than reading, either of which may explain it. So the blast radius is unquantified: not catastrophic, not safe. One observation of one idle host is not evidence that a detach during heavy reads is survivable. The remedy is still wrong for the reason that does not depend on any of this -- it "fixes" the listing by undoing the attachment the caller asked for. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01D7PLjw5FMYdZ2GWn3FeMpR --- artifacts/bmc/mtcollins1-diskless-boot.json | 15 +++++++++++-- .../machine_intake/megarac_media_attach.dag | 22 +++++++++++++++---- .../mtcollins1_media_attach.dag | 11 +++++----- 3 files changed, 37 insertions(+), 11 deletions(-) diff --git a/artifacts/bmc/mtcollins1-diskless-boot.json b/artifacts/bmc/mtcollins1-diskless-boot.json index 2f3f0cbc22b..ef3ff8473f3 100644 --- a/artifacts/bmc/mtcollins1-diskless-boot.json +++ b/artifacts/bmc/mtcollins1-diskless-boot.json @@ -69,5 +69,16 @@ "the backing share is srv2, a fleet peer, so this is not yet independent of the fleet", "UEFI HTTP Boot remains unmeasured; every boot so far was forced to a device over IPMI" ], - "observed_at_utc": "2026-09-05T22:55Z" -} + "observed_at_utc": "2026-09-05T22:55Z", + "media_detach_during_boot_observation": { + "observed_at": "2026-09-06", + "what_was_done": "mount_cd toggled 0 then 1 on the controller while the unit was booted from the presented ISO, to force the images listing to re-enumerate", + "why_it_was_done": "a wrong diagnosis: the image had left the available listing because it was ATTACHED, not because the listing was stale", + "observed_outcome": "the running Ubuntu installer survived; measured afterwards over IPMI SOL, still interactive at its own menu, and chassis power reported on", + "controller_retry_config": { + "rmedia_retry_count": 3, + "rmedia_retry_interval": 15 + }, + "what_this_does_not_establish": "that a detach during heavy reads is survivable. The system was idle at a prompt rather than reading, and casper reads the read-only lower layer from the presented medium on demand (toram is absent from the observed command line). One observation of one idle host bounds nothing." + } +} \ No newline at end of file diff --git a/dag/gunbc/machine_intake/megarac_media_attach.dag b/dag/gunbc/machine_intake/megarac_media_attach.dag index fd9692859bd..f3a96d39ab3 100644 --- a/dag/gunbc/machine_intake/megarac_media_attach.dag +++ b/dag/gunbc/machine_intake/megarac_media_attach.dag @@ -44,10 +44,24 @@ import extdeps.languages.json.parse { // // THE EARLIER READING WAS THAT THE LISTING IS A CACHE THAT GOES STALE, with the remedy "re-assert // mount_cd". That remedy appeared to work, and the reason it appeared to work is the problem: -// toggling mount_cd DETACHES EVERYTHING, which returns the image to the available pool. On a host -// that is booted from that medium — which is this lane's entire purpose — the advice amounts to -// pulling the running root out from under it. A remedy that works by undoing the desired state is -// worse than no remedy, and it was written into an operator-facing message. +// toggling mount_cd DETACHES EVERYTHING, which is what returns the image to the available pool. A +// remedy that works by undoing the desired state is worse than no remedy, and it was written into +// an operator-facing message on a lane whose entire purpose is hosts booted from that medium. +// +// WHAT THE HAZARD IS, AT THE STRENGTH IT WAS ACTUALLY MEASURED, because an earlier revision of this +// paragraph overstated it in the same breath as retracting an overstatement. It claimed the toggle +// pulls the running root out from under a booted host. That is the MECHANISM — the presented medium +// is withdrawn, and casper reads the read-only lower layer from it on demand rather than from RAM, +// per this lane's own boot receipt — but it is not what was observed. The toggle WAS performed on +// this unit while it was booted from the medium, and the running installer survived it: measured +// afterwards over serial console, still interactive at its own menu. The controller is configured +// rmedia_retry_count 3 / rmedia_retry_interval 15, and the system was idle at a prompt rather than +// reading, either of which may explain the survival. +// +// SO THE HONEST STATEMENT IS THAT THE BLAST RADIUS IS UNQUANTIFIED, not that it is catastrophic and +// not that it is safe. One observation of one idle host is not evidence that a toggle during heavy +// reads is survivable, and the remedy is still wrong for the reason that does not depend on any of +// this: it "fixes" the listing by undoing the attachment the caller asked for. // // SO THE ORDER OF READS IS LOAD-BEARING AND CONFIGURATIONS COMES FIRST. Asking the available list // whether an already-attached image is present is asking the wrong authority a question it answers diff --git a/dag/gunbc/machine_intake/mtcollins1_media_attach.dag b/dag/gunbc/machine_intake/mtcollins1_media_attach.dag index 86926b7639c..a21529dee9c 100644 --- a/dag/gunbc/machine_intake/mtcollins1_media_attach.dag +++ b/dag/gunbc/machine_intake/mtcollins1_media_attach.dag @@ -47,10 +47,11 @@ data mtcollins1_media_username: NonEmptyStr = "admin" // EACH OUTCOME RENDERS ITS OWN REASON because they have different remedies, and the remedies are // what an operator acts on — which is why one of these messages previously had to be retracted -// rather than reworded. It advised re-asserting mount_cd to fix an "empty" listing. That detaches -// every presented image, so on a host booted from the medium the advice destroys the running root. -// The listing was never stale; the image was simply attached, and attached images leave the -// available listing. See gunbc.machine_intake_megarac_media_attach for the measurement. +// rather than reworded. It advised re-asserting mount_cd to fix an "empty" listing. The listing was +// never stale: the image was simply attached, and attached images leave the available listing. The +// advice "worked" only by detaching every presented image, which is to say by undoing the state the +// caller had asked for. See gunbc.machine_intake_megarac_media_attach, which also records how far +// the danger to a booted host was and was not measured. // // MegaRacAlreadyAttached IS A SUCCESS AND IS RENDERED AS ONE. Converge is run repeatedly by // construction, so the second run finding the desired state already held is the normal case, not an @@ -73,7 +74,7 @@ fn mtcollins1_render_attach(outcome: MegaRacAttachOutcome) -> ProcessExit { MegaRacImagesUnreadable { detail: d } => exit_failure(reason: concat("mtcollins1 media attach: images listing unreadable: ", d)) MegaRacImageListingEmpty => - exit_failure(reason: "mtcollins1 media attach: the image is not presented AND the controller offers no image available to attach; check that the export is reachable from the controller and holds the file. Do NOT toggle mount_cd to 'refresh' this — that detaches presented media, which on a host booted from it removes the running root") + exit_failure(reason: "mtcollins1 media attach: the image is not presented AND the controller offers no image available to attach; check that the export is reachable from the controller and holds the file. Do NOT toggle mount_cd to 'refresh' this — it works by detaching every presented image, so on a host booted from that medium it withdraws what the running system is reading from") MegaRacImageListingUnparseable { detail: d } => exit_failure(reason: concat("mtcollins1 media attach: the images listing was not valid JSON, which is a protocol failure rather than the stale-cache case above: ", d)) MegaRacImageNotEnumerated { image_name: n } => From a84bc25761f02209f56fb2e747559ac2d9bd41bb Mon Sep 17 00:00:00 2001 From: gunbc-ci-auto-heal Date: Thu, 10 Sep 2026 02:20:48 +0000 Subject: [PATCH 11/11] Close MegaRAC sessions that opened without a readable token, and stop treating unreadable presented-media as detached. An opened session with no CSRFToken was reported as never opened, which is the leak that filled this controller. Malformed configurations collapsed to false and could StartMedia over unknown attachment state. The capture artifact also still claimed a RAM-resident root after the receipt had retracted that. Co-authored-by: Cursor --- artifacts/bmc/mtcollins1-diskless-boot.json | 4 +- .../boot_image_export_ownership.dag | 11 +- .../machine_intake/megarac_media_attach.dag | 112 ++++++++++++++---- .../mtcollins1_diskless_boot_receipt.dag | 16 +-- 4 files changed, 104 insertions(+), 39 deletions(-) diff --git a/artifacts/bmc/mtcollins1-diskless-boot.json b/artifacts/bmc/mtcollins1-diskless-boot.json index ef3ff8473f3..ec14e1c4601 100644 --- a/artifacts/bmc/mtcollins1-diskless-boot.json +++ b/artifacts/bmc/mtcollins1-diskless-boot.json @@ -22,7 +22,7 @@ "diskless_evidence": { "kernel_cmdline": "BOOT_IMAGE=/casper/vmlinuz", "memory_available_kib": 261972592, - "note": "casper runs the squashfs from RAM with a tmpfs overlay; nothing is written to local storage, and this unit has no boot drive to write to", + "note": "diskless, not RAM-resident: casper mounts squashfs read-only under a tmpfs overlay, so the WRITABLE layer is memory-backed and no local disk is involved. Casper toram is absent from the observed command line, so the read-only lower layer is presumed to be read from the presented media on demand. A RAM-resident root does not follow from what was observed.", "root": "squashfs 4.0 mounted read-only with an overlayfs upper" }, "firmware_behaviours_that_a_modeled_caller_must_expect": { @@ -32,7 +32,7 @@ "http": 500, "observed": "the write reported failure AND the share re-mounted; images re-enumerated 23s later" }, - "share_rescan": "the controller stopped enumerating the share after a new file appeared, returning an empty image list; re-asserting mount_cd re-scanned it", + "share_listing_is_available_pool_not_share_contents": "after start-media attaches an image, that name leaves /remote/images for /remote/configurations and survivors reindex. An empty or shorter images listing is not a stale cache. Re-asserting mount_cd appeared to refresh the listing because the toggle detaches everything and returns names to the available pool; that is not a remedy.", "stop_media": { "body_code": 1411, "http": 500, diff --git a/dag/gunbc/machine_intake/boot_image_export_ownership.dag b/dag/gunbc/machine_intake/boot_image_export_ownership.dag index 51fd1122a5b..f8087f10c64 100644 --- a/dag/gunbc/machine_intake/boot_image_export_ownership.dag +++ b/dag/gunbc/machine_intake/boot_image_export_ownership.dag @@ -43,13 +43,10 @@ import gunbc.fleet_posix_accounts { // says the AUTOMATION principal owns the export, so converge should run as that principal — rather // than the earlier state, which was simply that nobody but root could write there. // -// WHAT THIS ROW DOES NOT YET DO, stated because the gap is easy to miss under a page of prose about -// authority: NOTHING CONSUMES IT AT THE WRITE BOUNDARY. publish_verified_boot_image calls mv -// directly and never asks export_ownership_standing anything, so the policy below has no causal -// position — the program publishes whenever the operating system's permissions happen to allow it, -// including under root or a world-writable export. The requirement is currently a description of -// what ought to hold, checked by a probe against fixtures, not a wall. Closing that needs a real -// stat plus effective-principal observation bound to the path, admitted before the move. +// THE WRITE BOUNDARY CONSUMES THIS REQUIREMENT. observed_export_ownership_standing is the causal +// entry; gunbc.machine_intake_boot_image_fetch admit_then_publish_boot_image refuses before the +// move unless the standing is ExportOwnershipSatisfied. An earlier paragraph in this file said +// nothing consumed it; that sentence outlived the commit that wired the gate. // // ROOT IS THE WRONG ANSWER EVEN THOUGH IT WOULD WORK. Escalating the publish step means converge // acquires privilege for a file write, and a grant that broad is not proportionate to "put a diff --git a/dag/gunbc/machine_intake/megarac_media_attach.dag b/dag/gunbc/machine_intake/megarac_media_attach.dag index f3a96d39ab3..710559ebebe 100644 --- a/dag/gunbc/machine_intake/megarac_media_attach.dag +++ b/dag/gunbc/machine_intake/megarac_media_attach.dag @@ -13,6 +13,7 @@ import extdeps.languages.json.parse { JsonValue, json_object_unique_member, parse_json_document, + some_json_value, } // ATTACHING A REMOTE IMAGE, AS A MODELED OPERATION RATHER THAN A SHELL SCRIPT. extdeps.bmc.megarac @@ -148,10 +149,10 @@ fn image_index_named(elements: List, image_name: String) -> String? { ) } -fn images_array_elements(doc: JsonValue) -> List { +fn images_array_elements(doc: JsonValue) -> List? { match doc { - JsonArray { elements: es } => es - _ => [] + JsonArray { elements: es } => Present { value: es } + _ => none } } @@ -208,12 +209,20 @@ func megarac_attach_remote_image( Present { value: doc } => json_string_member(doc: doc, key: "CSRFToken") } + // AN OPENED SESSION WHOSE TOKEN CANNOT BE READ IS STILL AN ALLOCATED SESSION. An earlier + // revision returned SessionReleaseNotAttempted here, which is the leak that saturated this + // controller: OpenSession succeeded, the cookie was written, and CloseSession never ran because + // the renderer treated the arm as "never opened". The DELETE still needs an X-CSRFTOKEN header + // by the upstream contract; the cookie is the bearer, so the header is supplied as a sentinel + // rather than inventing a second CloseSession shape. A refused or uncertain close remains + // SessionReleaseRefused, never NotAttempted. return match csrf { Absent => - MegaRacAttachResult { - outcome: MegaRacSessionTokenUnreadable { detail: session.body }, - session_release: SessionReleaseNotAttempted, - } + megarac_release_unreadable_session( + bmc_host: bmc_host, + cookie_jar: cookie_jar, + session_body: session.body, + ) Present { value: token } => megarac_attach_then_release( bmc_host: bmc_host, @@ -224,6 +233,24 @@ func megarac_attach_remote_image( } } +func megarac_release_unreadable_session( + bmc_host: NonEmptyStr, + cookie_jar: NonEmptyStr, + session_body: String, +) -> MegaRacAttachResult + uses net: std.resources.Network +{ + closed = megarac.Media.CloseSession( + bmc_host: bmc_host, + cookie_jar: cookie_jar, + csrf_token: "unreadable" as NonEmptyStr, + ) + return MegaRacAttachResult { + outcome: MegaRacSessionTokenUnreadable { detail: session_body }, + session_release: if closed.success { SessionReleased } else { SessionReleaseRefused { detail: closed.body } }, + } +} + // THE RELEASE RUNS ON EVERY PATH THROUGH THE ATTACH, which is why the attach outcome is computed // first and the close follows it unconditionally rather than sitting inside a success arm. A // refusal allocated the same session a success does. @@ -273,8 +300,16 @@ func megarac_attach_with_token( if presented.success == false { return MegaRacConfigurationsUnreadable { detail: presented.body } } - if configurations_confirm_attachment(body: presented.body, image_name: image_name) { - return MegaRacAlreadyAttached { image_name: image_name } + // MALFORMED PRESENTED STATE IS UNKNOWN, NOT "NOT ATTACHED". An earlier revision collapsed a + // JSON parse failure or a non-array document to false, then walked the available-images list + // and could StartMedia over a controller whose current attachment was unreadable. That is the + // absorbing-fallback DESIGN §5 forbids: ignorance rendered as the empty case. + match presented_media_standing(body: presented.body, image_name: image_name) { + PresentedMediaUnreadable { detail: d } => + return MegaRacConfigurationsUnreadable { detail: d } + PresentedMediaAttached => + return MegaRacAlreadyAttached { image_name: image_name } + PresentedMediaNotAttached => none } images = megarac.Media.GetRemoteImages( @@ -289,10 +324,19 @@ func megarac_attach_with_token( match parse_json_document(s: images.body) { JsonDocumentUnreadable { gap: _ } => return MegaRacImageListingUnparseable { detail: images.body } - JsonDocumentParsed { value: _ } => none + JsonDocumentParsed { value: v } => + match images_array_elements(doc: v) { + Absent => + return MegaRacImageListingUnparseable { detail: images.body } + Present { value: _ } => none + } } let elements = match parse_json_document(s: images.body) { - JsonDocumentParsed { value: v } => images_array_elements(doc: v) + JsonDocumentParsed { value: v } => + match images_array_elements(doc: v) { + Present { value: es } => es + Absent => [] + } JsonDocumentUnreadable { gap: _ } => [] } if (elements |> count) == 0 { @@ -343,12 +387,25 @@ fn configuration_row_confirms(row: JsonValue, image_name: NonEmptyStr) -> Bool { name_ok && active && cd_media } -fn configurations_confirm_attachment(body: String, image_name: NonEmptyStr) -> Bool { - let rows = match parse_json_document(s: body) { - JsonDocumentParsed { value: v } => images_array_elements(doc: v) - JsonDocumentUnreadable { gap: _ } => [] +type PresentedMediaStanding + = PresentedMediaUnreadable { detail: String } + | PresentedMediaAttached + | PresentedMediaNotAttached + +fn presented_media_standing(body: String, image_name: NonEmptyStr) -> PresentedMediaStanding { + match parse_json_document(s: body) { + JsonDocumentUnreadable { gap: _ } => PresentedMediaUnreadable { detail: body } + JsonDocumentParsed { value: v } => + match images_array_elements(doc: v) { + Absent => PresentedMediaUnreadable { detail: body } + Present { value: rows } => + if rows |> any(r => configuration_row_confirms(row: r, image_name: image_name)) { + PresentedMediaAttached + } else { + PresentedMediaNotAttached + } + } } - rows |> any(r => configuration_row_confirms(row: r, image_name: image_name)) } // A NON-ZERO START FOLLOWED BY NO CONFIRMATION IS PENDING, NOT REFUSED, and this firmware is why: @@ -381,13 +438,22 @@ func megarac_start_and_confirm( return MegaRacAttachUnconfirmed { image_name: image_name, configurations: configurations.body } } - return if configurations_confirm_attachment(body: configurations.body, image_name: image_name) { - MegaRacAttached { image_name: image_name, image_index: image_index } - } else { - if start.success == false { - MegaRacStartRefused { detail: start.body } - } else { + // THE READ-BACK DECIDES, INCLUDING WHEN START REPORTED FAILURE. This firmware answers HTTP 500 + // on writes that take effect; mapping start.success == false to MegaRacStartRefused refused an + // attach that was merely still mounting, and contradicted the comment this function already + // carried. A missing confirmation after any start response is pending (Unconfirmed). StartRefused + // is reserved for a start that failed AND whose configurations document is itself unreadable + // as JSON — that is, when we cannot even ask the read-back. + return match presented_media_standing(body: configurations.body, image_name: image_name) { + PresentedMediaAttached => + MegaRacAttached { image_name: image_name, image_index: image_index } + PresentedMediaNotAttached => MegaRacAttachUnconfirmed { image_name: image_name, configurations: configurations.body } - } + PresentedMediaUnreadable { detail: d } => + if start.success == false { + MegaRacStartRefused { detail: start.body } + } else { + MegaRacAttachUnconfirmed { image_name: image_name, configurations: d } + } } } diff --git a/dag/gunbc/machine_intake/mtcollins1_diskless_boot_receipt.dag b/dag/gunbc/machine_intake/mtcollins1_diskless_boot_receipt.dag index 7ec4a8f3d1a..8718351e89e 100644 --- a/dag/gunbc/machine_intake/mtcollins1_diskless_boot_receipt.dag +++ b/dag/gunbc/machine_intake/mtcollins1_diskless_boot_receipt.dag @@ -33,12 +33,12 @@ import std.content_hash { ContentHash, Sha256Digest, Sha256Hash } // The unit reported 261972592 KiB available, so a RAM-resident root would be unremarkable here if // it were arranged; the memory is not the obstacle. data mtcollins1_diskless_boot_capture_digest: ContentHash = Sha256Hash(Sha256Digest { - hex: "cefdf6c063544a08c937770f9450ce0d0ee15a06bab2bf35af9f5eaadcda7681", + hex: "76340421bccd856be205f14e848c6ccca2dcb15857639ecfba07db03d5133a74", }) data mtcollins1_diskless_boot_artifact_path: NonEmptyStr = "artifacts/bmc/mtcollins1-diskless-boot.json" -data mtcollins1_diskless_boot_byte_count: Int = 4361 +data mtcollins1_diskless_boot_byte_count: Int = 5788 data mtcollins1_diskless_boot_observed_at: EpochMs = 1788648900000 @@ -65,11 +65,13 @@ data mtcollins1_booted_image_sha256: NonEmptyStr = "2ee2163c9b901ff5926400e80759 // set command's own return could not distinguish acceptance from indifference. The read-back here // is the images listing and the configurations listing. // -// AND THE SHARE LISTING IS NOT STABLE ACROSS FILE ADDITIONS. After a new ISO appeared in the -// exported directory the controller returned an EMPTY image list, including for the file it had -// previously enumerated. Re-asserting mount_cd re-scanned it. A caller that reads an empty listing -// as "the share has no images" will conclude the wrong thing; the listing is a cache, not the -// share. +// AND THE IMAGES LISTING IS THE AVAILABLE-TO-ATTACH POOL, NOT THE SHARE. After start-media +// attaches an image, that name leaves /remote/images for /remote/configurations. An earlier reading +// treated the shorter listing as a stale cache and advised re-asserting mount_cd; that toggle +// detaches everything, which is why the name reappeared. The retract lives in +// gunbc.machine_intake_megarac_media_attach. A caller that reads an empty listing as "the share has +// no images" will still conclude the wrong thing, but the wrong thing is "already attached" rather +// than "cache went stale". type ControllerWriteReportingAnomaly { operation: NonEmptyStr http_status: Int