Skip to content

mtcollins1 census: published-ISO boot-chain readback + module-identity instrument + decompression-refusal observation - #12487

Merged
gunbai-bot[bot] merged 8 commits into
mainfrom
session/zesty-newt-134
Sep 28, 2026
Merged

gunbai-bot[bot] merged 8 commits into
mainfrom
session/zesty-newt-134

Conversation

@gunbai-bot

@gunbai-bot gunbai-bot Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Work item: mtcollins1 census — name the module that fails ZSTD decompression + published-ISO initrd readback. The live-boot HOLD is untouched. This PR boots no hardware, edits no image by hand, force-loads nothing and disables no validation.

What the evidence already says (no RAM conclusion)

  • Run 36335369059 (failed attempt). Artifact mtcollins1-boot-receipts has digest sha256:16dcc1b34233cea3b338104a6cff5d88a46f8b6c2e815443653da5271062dd9b. The SOL capture inside it has sha256 761e79ca…fcee87. It shows two ZSTD-decompression failed with status 20 lines at 10.374590 and 10.380475.
  • Run 36253081549 (census-reaching attempt). Artifact digest sha256:ca38324ebcad6477911b0f56a86f91a0785a77dc31163ede6b9afc616fde3496; SOL capture sha256 83e818fb…8f5161. It has no refusal, and it has the renesas "fallback to ROM" warning.
  • Kernel v6.8, kernel/module/decompress.c. The printk carries only the decoder status and never names a module. Status 20 is zstd corruption_detected, which the kernel returns as -EINVAL.

1. Published-image boot-chain readback (built; executes after merge)

gunbc.machine_intake_mtcollins1_census_member_readback, fleet-converge mode mtcollins1_census_member_readback. It reaches srv2 over fleet SSH, like mtcollins1_census_medium_readback, because srv2 has had no runner slots since 2026-09-19. Each attempt is one transaction:

  1. Pin gate before anything is extracted. Both ISOs are measured against their pins: stock against its SHA256SUMS row, census against the boot medium row. A refusal ends the attempt.
  2. An owned workspace. The attempt runs mktemp -d /tmp/gunbc-mtcollins1-census-member.XXXXXXXX, which creates a new 0700 directory and never reuses a name. Concurrent dispatches share srv2 but never share a path; the mode's concurrency key is the runner host. If the workspace can't be created, nothing is extracted.
  3. Extraction. For each side and member (/boot/grub/grub.cfg, /casper/vmlinuz, /casper/initrd), xorriso extracts into the workspace, and only a successful extraction is hashed.
  4. Cleanup. The workspace is removed once, with rm -rf --one-file-system. The outcome is reported beside the verdict, never in place of it.

Workspace outcomes at the remote acknowledgement boundary. Each outcome states only what is established:

  • CreationNotRequested: a gate stopped the attempt before mktemp.
  • CreationNotDispatched: the transport refused the leg, so the invocation was never made.
  • CreationFailed: mktemp reported its own non-255 status, so nothing was created.
  • CreationUnconfirmed: ssh returned 255, which is its client status and does not show the command didn't run; or mktemp exited 0 with output that isn't exactly one path under the prefix. Residue may remain. The returned text is kept, and no unadmitted name is used to extract or delete.
  • After acquisition: Removed, RemovalFailed (rm's own failure), RemovalUnconfirmed (lost reply, not proof the directory remains), or RemovalNotDispatched.

The attempt holds only on an identical chain and Removed. Nothing under /srv/bmc is written.

The verdict names the first member in boot order that differs. grub.cfg is the member the remaster rewrites, so if it compares equal the run refuses with control_did_not_discriminate. That means the difference control failed; its cause is not chosen: the census file may not have been the one read, or the published artifact may lack the rewrite. Equal kernel and initrd bytes establish only the scoped comparison of stock against published bytes, not what reached host memory or why a decoder refused. If the initrds are equal, every module inside is equal by construction, so modules are not re-hashed one by one. A differing initrd is reported as the first differing boundary, and descending into its layers is the declared next step. For reference, the stock /casper/initrd sha256 measured offline is deb2c288c12e6f35c2b9242134116ea03f7d4eb91f0c354ecd993680ffabbbf7.

Scope of the acquisition controls. They run at the attempt/render consumer (census_member_attempt_of, the fold the producer runs), with legs built through the real transport adapter. The cases are:

  • no creation requested after a pin refusal;
  • an issued creation with 255, both with and without a partial path;
  • exit 0 with two paths, or with a path outside the prefix;
  • a reported mktemp failure, and an undispatched creation;
  • an acquired path that is removed;
  • cleanup reply loss beside a retained identical verdict;
  • rm's own failure, and an undispatched rm.

They do not execute overlapping attempts against a real host. Exclusivity of a successful acquisition rests on mktemp's create-new contract.

Not yet executed: the mode only exists once this PR is on main, and the fleet key is federated to main runs. After merge: dispatch fleet-converge with mode mtcollins1_census_member_readback, then attach the receipt here.

2. Module-identity instrument (modeled cmdline input)

The instrument is mtcollins1_census_module_identity_args = KMOD_LOG=info udev.log_level=debug SYSTEMD_LOG_TARGET=kmsg printk.devkmsg=on. It is appended to the census grub_kernel_cmdline build input, so it changes the build key to 5828991ca3dcace9 and therefore the digest. The boot's medium readback refuses until the image is republished, read back and repinned. No one edits GRUB by hand.

udev.log_level=debug alone does not reach SOL. Measured against the versions in this initrd (systemd-udevd 255.4-1ubuntu8.10, libkmod 31, kernel 6.8.0-71):

  • stderr goes to KVM, not SOL. init-top/udev runs udevd with SYSTEMD_LOG_LEVEL=info; udev.log_level=debug overrides it, because the cmdline is parsed after the environment. But daemonized udevd logs to stderr = /dev/console, and both captures' cmdlines end --- console=tty0. So /dev/console is the KVM, not SOL. Fix: SYSTEMD_LOG_TARGET=kmsg.
  • libkmod filters its own line. libkmod's path-bearing Failed to insert module '<path>': <strerror> is logged at INFO, and libkmod's default priority is ERR. Fix: KMOD_LOG=info. The kernel passes it into /init's environment, initramfs init does not scrub the environment, and udevd inherits it.
  • Rate limit. Userspace kmsg records are rate-limited by default. Fix: printk.devkmsg=on.
  • Console level (option A, eager-owl-205). INFO lines reach SOL at the default console loglevel. udev's per-device DEBUG lines stay in the kmsg ring, so the requesting device/modalias is recorded as "unread on SOL" and never inferred. ignore_loglevel was declined: it would flood the 115200 line and perturb the timing of a possible race.

Upstream facts are each homed in their own module: extdeps.linux.module_decompress, extdeps.kmod.libkmod, extdeps.systemd.udevd, and extdeps.linux.kernel (printk.devkmsg).

Offline control (condition 2 of the ruling). Setup:

  • qemu-system-aarch64 10.0.13, -M virt, one qemu-xhci device, TCG.
  • Stock /casper/vmlinuz, plus stock /casper/initrd with exactly one edit: 64 bytes inverted inside the zstd body of xhci-pci.ko.zst, frame header intact. Edited initrd sha256 83fd2a47…6f96.
  • Census cmdline, --- console=tty0 kept.

Results:

  • Without the args (log sha256 7e8db2f8…5552): ZSTD-decompression failed with status 20, and nothing names the module. This reproduces the hardware signature.
  • With the args (log sha256 b8b33393…6bcf), on ttyAMA0:
    [   27.190394] ZSTD-decompression failed with status 20
    [   27.209606] (udev-worker)[100]: Failed to insert module '/lib/modules/6.8.0-71-generic/kernel/drivers/usb/host/xhci-pci.ko.zst': Invalid argument
    
    This gives the module path, the worker pid, a kernel timestamp and the result. About 30 udevd manager warnings also reached SOL; no per-device debug lines did.

The control was run by hand in a session. It has no entry point yet; a modeled qemu instrument is the follow-up if this needs to be re-derivable.

Side effect to know about. The environment variables are inherited by the live system's PID 1, so the census boot's SOL will carry somewhat more INFO output after the initramfs as well.

3. The fact at its real strength

gunbc.machine_intake_kernel_module_decompression_observation keeps two independently observed populations:

  • the kernel's refusals: a timestamp and a ZstdErrorCode, with no module named;
  • the loader's named insert failures: path, worker, timestamp and errno, plus a later-driver reading.

Nothing in a capture links one invocation to the other. A zstd refusal returns -EINVAL, but EINVAL does not establish a decompression failure: the v6.8 loader also returns it for symbol-version and namespace checks and for invalid flags. So an Invalid argument insert failure is a candidate, and the identity reads RefusalIdentityCandidates (UNRESOLVED) or RefusalIdentityUnread. There is no "known" arm. A known identity needs an invocation-level link, which is the declared next rung.

Records that can't be parsed are kept. A recognized refusal line whose status can't be read, or a loader record truncated before its path or errno, is retained with its line and the field that failed. A refusal count is never stated as complete while an unparsed refusal record stands beside it, and uncatalogued numeric statuses stay ZstdErrorCodeOther.

The observation is carried by ConsoleRetained in mtcollins1_boot_diagnostic_bundle, the attempt-bound console route, and contributes one finding when a refusal record is present. It claims nothing about device availability, live-root acceptance, SOL health or hardware, and the renesas fallback to ROM line is not read.

4. Executing controls

kernel_module_decompression_observation_witness_test (13 claims):

  • Positive control. The census-reaching excerpt of run 36253081549, renesas warning included, reads as not refused with no findings.
  • Anonymous red. The failed excerpt of run 36335369059 is refused ×2 with corruption_detected and identity UNREAD.
  • Real instrumented capture. The qemu lines show the xhci-pci failure as an unresolved candidate, with every named failure kept. The reviewer's counterexample holds: an unrelated earlier EINVAL failure is only a candidate.
  • Unparsed records. Malformed-only, valid-plus-malformed (no complete count), an uncatalogued numeric status, incomplete loader records (path, errno), and unrelated text.
  • Inhabitance. It is carried by the real console reading.

mtcollins1_census_member_readback_witness_test (13 claims) covers the comparison verdicts, plus the pin gate that precedes extraction, workspace admission, a failed extraction never hashed, and cleanup retained beside the verdict.

Coordination: swift-wolf-904 (#12434, the SOL collector) and eager-koi-811 (KVM). This observation reads the retained capture on the existing console route and adds nothing to the collector. With this instrument, udevd writes to kmsg, which the kernel prints on every console, so the KVM (tty0) shows the same loader records as SOL.

🤖 Generated with Claude Code

Brian Searls and others added 3 commits September 28, 2026 01:52
… instrument, decompression-refusal observation

Deliverable 1: gunbc.machine_intake_mtcollins1_census_member_readback reads vmlinuz, initrd
and grub.cfg out of the pinned stock and census ISOs on srv2 over fleet SSH (fleet-converge mode
mtcollins1_census_member_readback), with grub.cfg as the in-run discriminating control.

Deliverable 2: mtcollins1_census_module_identity_args (KMOD_LOG=info udev.log_level=debug
SYSTEMD_LOG_TARGET=kmsg printk.devkmsg=on) as a census build input; option A per eager-owl-205.
Upstream facts homed in extdeps.linux.module_decompress, extdeps.kmod.libkmod, extdeps.systemd.udevd.

Deliverables 3-4: gunbc.machine_intake_kernel_module_decompression_observation rides ConsoleRetained;
controls from runs 36335369059 / 36253081549 captures and an offline qemu receipt.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gunbai-bot
gunbai-bot Bot marked this pull request as ready for review September 28, 2026 02:38
Brian Searls and others added 3 commits September 28, 2026 03:04
…hecker)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…t_command is the one -extract spelling (review 72051)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

Review 72051 (xorriso argv fork): agreed and fixed in f0350b0. xorriso.Iso ExtractPathBestEffort had no caller once the readback moved onto fleet SSH, so I deleted it instead of keeping two spellings. xorriso_extract_command is now the only -extract argv, and it takes NonEmptyStr inputs like the operation did. The allowlist entry in extdeps.exec.command now admits the one spelling, not a copy. The member-readback witness passes locally (7/7).

— sent from zesty-newt-134

@briansrls briansrls left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD — exact-head module-identity instrument and published-ISO readback

Reviewed f0350b0, reconfirmed identical to refs/pull/12487/head immediately before publication. The diagnostic direction, declared cmdline input, shared xorriso builder and separate read-only-on-the-ISOs mode are accepted. Three source defects remain in the observation/admission consumers. These can be repaired without another physical boot or completion of the general BMC simulator.

1. P1 — any EINVAL insertion in the capture becomes the identity of an anonymous ZSTD failure

kernel_module_decompression_observation first collects all kernel decompression messages. Once that set is nonempty, it scans the ENTIRE capture for loader insert failures whose strerror equals Invalid argument, places every one in FailedModuleIdentity, and labels the result ModuleIdentityKnown. There is no invocation, worker/request, time, or other evidence join between these populations. A loader failure before the first ZSTD message is admitted just as readily as one afterwards.

Source-derived counterexample: prepend an unrelated module insertion failure with errno EINVAL to failed_attempt_excerpt. The two originally anonymous ZSTD failures now acquire that unrelated module as their known identity. Add two such records and both become the alleged identities, irrespective of their order or the number of actual decompression attempts. Reusing the current uas control but changing its errno from No such device to Invalid argument also promotes it to the known set without any evidence it decompressed badly.

The implication is reversed: a ZSTD decoder error returns EINVAL, but EINVAL does not establish a ZSTD decoder error. Upstream Linux v6.8 kernel/module/main.c returns EINVAL for other loader conditions (for example symbol version/namespace checks and invalid flags); module initialization can also return an error. libkmod v31's path-bearing INFO record reports the insertion's returned errno, not the internal failing kernel stage. Sources: https://raw.githubusercontent.com/torvalds/linux/v6.8/kernel/module/main.c ; https://raw.githubusercontent.com/kmod-project/kmod/v31/libkmod/libkmod-module.c ; https://man7.org/linux/man-pages/man2/init_module.2.html .

Keep the named insertion records as useful independent observations, but keep their association with anonymous decompression events unestablished/candidate unless a discriminating invocation-level join exists. A nearest-line or arbitrary time-window heuristic is not that join. The controlled QEMU case can independently know which file was deliberately corrupted; that experimental ground truth does not make the same attribution rule valid on an unconstrained real boot.

The narrow repair need not add kernel tracing immediately: represent anonymous kernel refusals plus the actually named insertion failures, and report the unresolved relationship honestly. An automatic known decompression identity claim needs the stronger producer. Add unrelated-EINVAL before/after, interleaved workers, multiple anonymous failures with one named insertion, and the controlled positive, through the real console/bundle consumer. Preserve the existing different-errno discrimination, but do not treat it as sufficient.

2. P2 — a recognized but unreadable refusal is dropped into 'no refusal in this capture'

decompression_refusal returns the same none for an unrelated line and for a line containing the exact failure prefix whose code cannot be parsed. The enclosing fold drops both and returns ModuleDecompressionNotRefused / no findings when the resulting list is empty.

For example [ 10.374590] ZSTD-decompression failed with status , or the same complete prefix followed by unreadable, renders as no refusal in this capture. Partial/truncated evidence is particularly relevant on the observation channel being repaired here. When a valid record is beside a malformed one, only the valid record survives, silently undercounting the recognized failure population.

Separate nonmatching lines from recognized-unparsed/pending records, preserving the original line and the failed field. Do not invent a numeric status or certify an exact zero/count over unparsed members. Likewise keep a malformed loader record as unread evidence rather than claiming that no loader record reached the capture. Unknown valid numeric ZSTD codes should continue to use the existing Other arm; there is no request to enumerate all codes.

Add malformed/truncated and valid-plus-malformed controls at mtcollins1_boot_console_reading / bundle rendering, with unrelated text and the census-reaching excerpt as positives. A parsing uncertainty must not become a physical hardware assertion either.

3. P1 — concurrent readbacks share and delete the same remote scratch files

member_scratch_path is a constant per side/member, for example /tmp/gunbc-mtcollins1-census-member-census-initrd. Every readback runs separate SSH operations for rm -f, extract, hash, and rm -f on that pathname. There is no per-attempt owned directory or admission of existing ownership.

The new mode is ExecutorDomain: fleet_converge_mutation_group_for keys it by the selected runner host, while census_member_readback_target() ALWAYS addresses srv2. Dispatches on srv1 and srv3 can therefore operate concurrently on the same remote scratch namespace. One can remove another's extracted member before its hash (a false read failure). Across revisions/pins, an extraction can replace the object between the other attempt's extraction and hash; all four command statuses can be zero while the returned digest belongs to the other attempt. Cleanup can delete a sibling's in-progress observation. The GRUB-difference control does not protect the subsequent kernel/initrd scratch objects.

This is a source-derived interleaving, not a claimed live srv2 incident or an independently executed xorriso race. Calling the operation read-only on /srv/bmc does not make its remote scratch effects isolated.

Use an actually created, exclusively owned temporary directory/identity on srv2 and derive every member path beneath it. A run/attempt label without exclusive creation is not sufficient if it can be reused. Admit prerequisite operations before dependent extraction/hash, and clean only this attempt's resources; preserve the initiating failure and cleanup result separately. The current read_member_remote executes extraction even after the initial clear fails and gives a later cleanup failure priority over the initial problem. A failed setup must not authorize writing an occupied path. Reuse existing filesystem/typed-argv authorities; no new remote agent or host job is requested.

Add a local/controlled extraction-route test with two simultaneous invocations, different member contents, and delayed hash/cleanup, plus occupied/setup-failure and successful repeat cases. The current seven readback tests supply abstract digest values to the pure comparison; they do not exercise this acquisition/ownership boundary. The actual srv2 observation may still wait until after merge once the source is corrected.

Accepted instrument and wiring

  • The four arguments are declared input to mtcollins1_seeded_image_input: KMOD_LOG=info udev.log_level=debug SYSTEMD_LOG_TARGET=kmsg printk.devkmsg=on. Their authorities are separated; the new build-key expectation is 5828991ca3dcace9, and the old output-digest pin is not silently relabeled as the new image. Keep the republish/readback/repin requirement and the live hold. No hand GRUB modification is needed.
  • The reported QEMU A/B is the right kind of transport/log-retention evidence: same intended setup, one deliberately corrupted module, missing pathname without the arguments and a path-bearing loader failure on ttyAMA0 with them. Its receipt excerpts clearly distinguish designed uas/later-driver additions from measured lines. I did not download/rehash the full QEMU logs or execute that experiment, so it remains author-run evidence. A manual A/B receipt is not an enrolled reproducible QEMU regression lane; preserve that distinction without making a general QEMU workflow a prerequisite for these local repairs.
  • Option A's omission of per-device debug from SOL is explicit. The requesting device/modalias is not inferred. INFO logging can name failed insertion files but does not by itself guarantee correct attribution to a kernel event. Extra userspace logging and the inherited PID1 environment are diagnostic side effects, not changes to decompression validation. ignore_loglevel/forced loading remain absent.
  • The new observation is consumed by the real ConsoleRetained constructor, diagnostic rendering and findings. It is not an inert note. It adds diagnostic evidence, not a claim that a required device/live root failed or a new power action.
  • The readback mode is wired through the mode roster, wire/scope/key-demand/mutation folds, CI target roster, invocation, actual step and always-on-mode receipt upload. The target is the declared srv2 fleet endpoint/principal. It contacts no BMC and does not intentionally write /srv/bmc; remote scratch still needs the ownership repair above.
  • The duplicate uncalled ExtractPathBestEffort operation is gone. xorriso_extract_command is the consumed shared exact-member builder, reached through typed argv rather than an independently copied SSH shell command.
  • Equal verified initrd bytes are sufficient for the scoped stock-vs-published payload comparison without rehashing every member. They are not evidence of the bytes delivered to the physical host or a diagnosis of RAM/kernel faults.

Other evidence qualifications / adjacent class

The readback's pure join rejects off-pin/unmeasured ISOs, but the wet producer evaluates all member reads before invoking that join. Therefore its pin verdict precedes comparison, not the remote extraction effects. Do not describe the current pure control as executing 'no extraction after pin refusal'. Move admission before dependent reads if that is the intended protocol.

An equal grub.cfg correctly fails the declared difference control, but the census image was not read overstates the cause. The same observation could mean the actual census artifact omitted the intended rewrite. Report failure of the control without deciding which producer was wrong. Similarly, hashing each ISO before reopening its path for member extraction assumes that publication keeps that object unchanged; cite/use that existing immutability contract or retain a coherent read transaction rather than inferring path equality is object stability.

The adjacent class is joining independent observations into a stronger causal/identity claim, or measuring a shared mutable name while reporting an attempt-owned result. Preserve evidence of an unexpected result rather than turning it into either a hardware diagnosis or an artificial clean reading. The previous Casper specimen is still not newly bound to run 36335369059 by this PR, and the older run is census-reaching, not fully qualified merely because this excerpt has no ZSTD message.

Verification and disposition

I inspected the complete new observation/readback modules, their witness specimens, the image-input/build-key delta, shared command builder, workflow/console consumers and the primary Linux/libkmod error contracts. Exact-head run 36375121892 has all six reported checks successful: generated, compiler, clippy, floor, emit-build and the witnesses aggregate. That does not discriminate the counterexamples above. The 7/7 local readback result is author evidence; no .dag suite, full initrd audit, QEMU boot, xorriso extraction, remote command, image publication, repin, merge or hardware operation was executed in this review.

Source: HOLD at f0350b0… for these three boundaries. Diagnostic direction accepted. Integrated live-boot HOLD unchanged. The changes requested are in the new local folds and readback transaction, not a demand to finish the whole acceptance matrix before gaining useful module-load visibility.

Absent => []
Present { value: f } =>
if f.error == refused_decompression_strerror {
[FailedModuleIdentity {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] EINVAL is not an invocation join

This selects every EINVAL insertion in the entire capture whenever any anonymous ZSTD refusal exists, including insertions preceding the refusal or from unrelated workers. EINVAL is also returned by other module-load stages. Preserve named insertion failures as independent/candidate observations unless an invocation-level producer links them to the anonymous decompression event. Add an unrelated EINVAL record to failed_attempt_excerpt: it must not become the known decompression identity merely because its strerror matches.

let prefix = linux_module_zstd_decompress_failed_prefix as String
if !string_contains(s: l.rest, pattern: prefix) { none } else {
match parse_int(s: trim(join(split(s: l.rest, delimiter: prefix).skip(n: 1), prefix))) {
Absent => none

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve a recognized refusal whose code is unreadable

Returning none here makes the complete failure prefix plus a missing/malformed code indistinguishable from unrelated text. The parent then reports ModuleDecompressionNotRefused or silently undercounts alongside valid rows. Retain a recognized-unparsed/pending reading with the raw line/cause, and keep it visible through the console/bundle consumer; do not invent a code or a zero-refusal result.

remote_answer_of(outcome: typed_argv_exec_over_fleet_ssh(target: target, context: context, argv: argv_words(command: command)))
}

fn read_member_remote(target: SshTarget, context: FleetSshExecutionContext, iso_path: NonEmptyStr, side: IsoSide, member: CensusIsoMember) -> MemberReading {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] This scratch path is shared across independently admitted dispatches

The mode's concurrency is keyed by executor host, but every invocation targets srv2 and uses the same per-side/member /tmp name. Two executions can remove/overwrite one another's extracted members between SSH legs, so a successful hash need not refer to this attempt's extraction. Create an exclusive remote workspace, carry its identity through extract/hash/cleanup, refuse failed setup before dependent effects, and test overlapping attempts with different contents. No live srv2 mutation is needed for that control.

…ords kept; owned srv2 workspace

1. EINVAL insert failures are unresolved candidates; every named insert failure is kept as its own
   observation; no nearest-line rule promotes one to the refusal's identity.
2. A recognised refusal or loader record that cannot be parsed is retained with its line and the
   failed field; the refusal count is not stated complete beside an unparsed record.
3. The readback gates extraction on the ISO pins, acquires an exclusively created mktemp -d workspace
   on srv2, extracts under it, never hashes a failed extraction, and removes that workspace once,
   reporting cleanup beside the verdict.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@briansrls briansrls left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD — exact-head rebind; attribution/parser repairs accepted, one remaining workspace-outcome finding

Reviewed 6fee01d, the single successor of f0350b0. Reconfirmed identical to refs/pull/12487/head immediately before publication. The two diagnostic blockers and the fixed shared-scratch path are repaired. Keep those changes. One P2 remains at the new workspace acquisition/result boundary; this is not a request for a new transaction service, a physical boot, or the general BMC matrix.

P2 — an unacknowledged remote creation is reported as a workspace that was never created

member_workspace_of returns WorkspaceRefused for every nonzero RemoteAnswer and for an unreadable/rejected output path. census_member_readback then unconditionally assigns WorkspaceNeverCreated, whose renderer says workspace=never created (nothing extracted). That is justified when the pin/context gate prevented the creation call. It is not justified after the creation call was issued but its outcome was not established.

Source-derived counterexample:

  1. The admitted fleet-SSH command reaches srv2 and mktemp -d successfully creates its new directory.
  2. The SSH connection fails before the local caller receives a complete result/exit status. It can also fail after some or all of the pathname reached stdout.
  3. The local RemoteAnswer has exit 255. member_workspace_of refuses and discards stdout in this branch.
  4. The returned attempt says never created, and has no owned workspace cleanup or explicit unresolved-residue record, although a directory can remain on srv2.

This is not an allegation that mktemp's exclusive creation is broken, nor an independently reproduced live SSH incident. OpenSSH documents 255 as a client/transport error, not proof that the remote command did not run. The inspected transport preserves a ran leg's exit/stdout/stderr before remote_answer_of and member_workspace_of collapse it. remote_answer_of also represents a refused leg as exit 255, so the same integer cannot establish whether creation was attempted. A zero-exit but malformed/truncated reply likewise establishes no usable owned path, not non-creation.

The cleanup side has the dual wording problem: workspace_cleanup_of converts any nonzero result into WorkspaceNotRemoved and renders NOT REMOVED, although a lost SSH result can follow a successful remote removal. Keep removal unconfirmed distinct from a proved remaining path. The current fail-closed attempt verdict is appropriate and must remain; the finding is false residue/history, not that this branch currently returns ExitSuccess.

Narrow repair: preserve no-creation-attempt, acquired workspace, and attempted-but-unconfirmed creation as different outcomes. Retain the full command observation, the target/attempt context and any returned pathname as evidence; do not discard it or treat an unadmitted name as authority to delete. Reserve never created for a path whose creation was not issued or whose absence is actually established. An unconfirmed acquisition may continue to refuse with possible residue explicitly outstanding; this review does not require inventing ownership or automatic recovery after a lost reply. Similarly report attempted-but-unconfirmed cleanup without claiming the directory definitely remains. No extraction on an unadmitted workspace, and no cleanup by guessing or sweeping the shared prefix.

Add controls at the actual attempt/render consumer for: no creation requested after pin refusal; issued creation with exit 255 both with and without partial pathname; exit 0 with unusable/multiple path output; successful acquired path; and cleanup reply loss beside a retained comparison result. They must require the unknown/residue statement and non-holding verdict, not just a nonzero process result. Keep the existing legitimate mktemp failure and successful cleanup cases. Where execution was refused before dispatch, preserve that fact instead of making it indistinguishable from a lost response.

Primary contracts checked: https://man.openbsd.org/OpenBSD-7.6/ssh.1#EXIT_STATUS ; https://www.gnu.org/software/coreutils/manual/html_node/mktemp-invocation.html . This is about the remote acknowledgement boundary, not a claim that an ordinary local mktemp output error necessarily leaks a directory.

Accepted: the anonymous and named populations no longer manufacture an identity

RefusalIdentityCandidates is explicitly UNRESOLVED; the known-identity arm is gone. All named insertion failures remain their own population, including different errno values and loader-only captures. The earlier unrelated-EINVAL specimen is a candidate, not an attributed decompression failure. No nearest-line heuristic replaces the missing invocation link. The QEMU experiment's controlled corruption remains separate experimental ground truth, not a general inference about real host failures.

The existing console consumer still calls this observation and renders/finds it. It does not create a new device, live-root, RAM or SOL verdict. The request/modalias remains explicitly unread at the selected SOL logging level. The later-driver arm means a matching prefix printed later in the retained sequence; it is not successful driver-load or device-availability evidence.

Accepted: recognized unparsed records survive and the count is qualified

The new stem recognizers distinguish nonmatching text from recognized-but-unparsed status/path/errno records. A malformed-only refusal is not dropped into NoRefusalRecorded; a valid-plus-malformed population keeps both and does not claim a complete refusal count. Uncatalogued numeric status stays in the Other arm. The whole named-failure population is retained even without a kernel refusal.

Non-blocking evidence precision: UnparsedRecord.line currently contains l.rest, after the timestamp and CR/NUL normalization have been stripped, not the full original raw console line. The retained console remains the raw-evidence reference, but do not describe this field alone as a verbatim timestamped record. Carry the timestamp/source location or original line when using it for cross-event correlation. The new tests assert the message portion, not preservation of the original instant.

Accepted: successful acquisition no longer shares the fixed scratch namespace

The actual wet producer checks census_pin_gate before calling mktemp or xorriso. The admitted mktemp result supplies the workspace for all six side/member paths; the new builder is admitted at the existing typed-argv seal. read_member_remote issues sha256sum only inside successful extraction, and cleanup occurs once after the normal returned readback, with its result kept beside the comparison. The held verdict requires both identical boot-chain readings and successful workspace cleanup. The old per-member shared /tmp names and pre-extraction rm are gone.

GNU mktemp's create-new-directory contract is a valid basis for separation of successful concurrent acquisitions. The supplied-answer controls are honestly scoped as pure decision controls: they do not execute xorriso, overlapping fleet transactions, or an interruption between remote operations. I did not run those either. I am not retaining the old collision finding merely because an empirical collision test has not run; the remaining finding is the uncertainty/result projection above. Keep the actual srv2 readback as its separate execution obligation.

The equal-GRUB renderer now reports that its difference control did not discriminate without choosing why. One older witness comment still says equal GRUB means the census image was not read; align that stale comment with the corrected predicate. This is non-blocking and does not require another control type.

Evidence, adjacent class and disposition

The new build-key/logging input and workflow wiring are unchanged by this successor. The republish/readback/repin ordering remains required; neither a passing pure comparison nor this review is a newly executed published-ISO audit. The previous separate Casper screenshot is not newly bound to run 36335369059, and nothing here attributes status 20 to RAM. As before, comparing reopened ISO paths assumes their bytes remain stable over the measurement; a per-attempt scratch directory is not itself a source-ISO snapshot.

The 26/26 result is author-reported. I inspected both complete new production modules, both witness files, the entire successor delta, the actual remote-answer/SSH composition, and the primary mktemp/SSH contracts. I did not execute the .dag suites, QEMU, archive extraction, overlapping acquisition, fleet SSH, cleanup on srv2, publication, repin or a hardware operation. At the exact-head run 36382667860, compiler, clippy, generated and emit-build had succeeded; floor was still running at the last read.

Adjacent class: an exclusively created resource is not the same fact as an acknowledged creation, and an unacknowledged mutation must not become an invented absence or an invented remaining resource. The safe no-extraction branch is already present; finish its typed residue/result account. This is a bounded repair to the new workflow, not an instruction to redesign SSH or achieve impossible acknowledgement guarantees.

HOLD at 6fee01d… for the one P2 above. The earlier causal-attribution and malformed-record defects are closed, and successful workspace isolation is accepted. No merge, enqueue, dispatch, BMC/host action or additional live authorization was performed. The integrated live-boot HOLD is unchanged.

PinsAdmitted { stock_iso: s, census_iso: c } =>
match member_workspace_of(answer: remote(target: target, context: context, command: mktemp_directory_command(template: census_member_workspace_template))) {
WorkspaceRefused { reason: r } =>
CensusMemberReadbackAttempt { readback: BootChainWorkspaceUnavailable { reason: r }, cleanup: WorkspaceNeverCreated }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] A refused mktemp reply does not establish that no workspace was created

This assigns WorkspaceNeverCreated after an actual creation attempt. If mktemp creates its directory on srv2 but SSH loses the response/exit status, RemoteAnswer.exit_code is 255; member_workspace_of refuses and this receipt falsely says never created. The nonzero branch also discards any pathname already received on stdout. Keep unattempted creation separate from attempted-but-unconfirmed creation, retain the full answer and possible residue, and continue to perform no extraction on that arm. Do not clean an unadmitted path or sweep the prefix. The cleanup-result projection should likewise say unconfirmed after a lost reply, not assert that removal did not occur. See the review for the narrow control pairs; this is not a claim that mktemp's exclusive creation fails.

…4663150)

An issued mktemp whose reply was lost (ssh 255) or whose output admits no path is CreationUnconfirmed
with residue outstanding and the returned text kept, never 'never created'; a lost rm reply is
RemovalUnconfirmed, distinct from rm's own failure and from an undispatched rm. Legs keep the
transport's not-dispatched arm instead of collapsing to exit 255. Controls run at the attempt/render
consumer. UnparsedRecord carries its timestamp.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@briansrls briansrls left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE-MERGE — exact-head rebind

Reviewed b6ae221, the single successor of 6fee01d; reconfirmed identical to refs/pull/12487/head before publication. This closes the remaining workspace-outcome finding in review 5334663150. The previously accepted attribution/parser and successful workspace-isolation repairs remain intact. Proceed through normal exact-head and merge-queue gates.

Remote acknowledgement is no longer invented resource history

The actual producer uses remote_leg/remote_leg_of for creation and cleanup, preserving TypedArgvExecRefused as LegNotDispatched instead of flattening it into exit 255. Returned exit/stdout/stderr remain separate from that pre-dispatch refusal.

workspace_acquisition_of now distinguishes an undispatched call, a reported command failure, and an unconfirmed creation. Exit 255, including partial pathname output, and zero exit with an unusable reply yield CreationUnconfirmed. Its receipt retains the returned text, explicitly names possible residue on srv2, and does not supply that text to extraction or deletion. The SSH authority explicitly notes that a remote program's own exit 255 is indistinguishable from a client error; treating that case as unknown is appropriately conservative.

workspace_removal_of keeps failed, unconfirmed, and undispatched cleanup distinct. In particular the 255 arm says that the path may or may not remain, while preserving the comparison result. Only Removed combined with an identical boot chain holds. The former false WorkspaceNeverCreated projection for an unacknowledged creation is gone.

The producer consumes the tested fold

census_member_readback still checks the actual pin gate before evaluating the creation call. Its admitted arm calls census_member_attempt_of with the real creation result and deferred member-read/cleanup functions. The fold calls neither callback when acquisition is unadmitted. After acquisition, it obtains the member comparison and retains the cleanup result beside it. Per-member hashing remains inside successful extraction, and successful acquisitions derive all member paths beneath their own created workspace.

The new controls construct legs through remote_leg_of and reach this same attempt/render consumer. They cover pin refusal; reply loss with and without a pathname; zero exit with multiple/outside paths; reported creation failure versus undispatched creation; successful acquired-and-removed execution; and lost, failed, or undispatched cleanup. They require both the result standing and relevant receipt text, with the identical-chain positive preventing a blanket-refusal implementation from satisfying the negative controls.

These are supplied-answer execution controls, not proof that real SSH, xorriso, overlapping attempts, or interrupted cleanup ran. The callback branching is inspected production source; the tests do not independently observe remote side effects. That scope is disclosed and is not a reason to keep the resolved source HOLD.

Diagnostic precision and adjacent class

UnparsedRecord now carries at and the renderer consumes it. Its normalized message remains distinct from the verbatim retained console. The stale equal-GRUB witness comment now says the difference control failed without selecting its cause. Independent anonymous refusals/named insertion failures remain unresolved candidates, not manufactured invocation identities.

Adjacent class: handling a returned uncertain result does not guarantee recovery after the worker disappears before producing a receipt. Keep possible residue with the dispatch's provenance and never sweep or delete an unadmitted path to clear it. Likewise, the per-attempt extraction workspace is not a snapshot of the reopened source ISO paths. Those remain explicit operational/evidence limits, not new requirements for a transaction service or a broader harness in this rebind. Do not generalize the mktemp-specific normal-failure interpretation to arbitrary mutating commands or new wrappers.

Evidence and landing scope

The reported 17/17 member-readback and 13/13 diagnostic claims are author-run evidence; I did not rerun the compiler, claim suites, QEMU, extraction, SSH, or fault-injection controls. I inspected the successor, production branching/result projection and consuming witnesses, and checked the primary SSH/mktemp contracts. At my exact-head CI lookup, witnesses run 36387741971 was queued, not completed green. Preserve normal required checks and the actual claim results.

This approval does not claim the srv2 published-ISO readback has executed. After landing, that separately admitted readback can supply the outstanding physical-file receipt; image publication/readback/repin remains necessary for the new logging build input before a later authorized boot. No pixel/host-health conclusion, RAM attribution, or new binding of the historical Casper specimen follows from this review.

APPROVE-MERGE at b6ae221…; the source HOLD on #12487 is cleared. The integrated live-boot HOLD is unchanged. No merge, enqueue, dispatch, package installation, image mutation, credential use, or host/BMC operation was performed.

Primary contracts: https://man.openbsd.org/OpenBSD-7.6/ssh.1#EXIT_STATUS ; https://www.gnu.org/software/coreutils/manual/html_node/mktemp-invocation.html .

@gunbai-bot
gunbai-bot Bot added this pull request to the merge queue Sep 28, 2026
Merged via the queue into main with commit 5402385 Sep 28, 2026
6 checks passed
@gunbai-bot
gunbai-bot Bot deleted the session/zesty-newt-134 branch September 28, 2026 08:15
@gunbai-bot

gunbai-bot Bot commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

Deliverable 1 executed. Run 36396523377 dispatched from main at 5402385 with mode mtcollins1_census_member_readback on runner srv3, reading srv2 over fleet SSH. Conclusion: success. Receipt artifact mtcollins1-census-member-readback, digest sha256:306b281b9b32b16441aa001ed3abf385afb8e1eab187c3ee1bab720dad74229d:

verdict=boot_chain_identical_to_stock
stock_iso=2ee2163c9b901ff5926400e80759088ff3b879982a3956c02100495b489fd555
census_iso=30ca88d43f8913995078f6c2c917d5049842f927f3fc886a103f460dd146a706
vmlinuz=f7816e8213621f699c58e6f2168dbeb85a463943e5359e8fd60c8a6a14cab3d0
initrd=deb2c288c12e6f35c2b9242134116ea03f7d4eb91f0c354ecd993680ffabbbf7
grub.cfg=differs (declared rewrite, control discriminated)
workspace=/tmp/gunbc-mtcollins1-census-member.U9P6C46P removed
  • Pins. Both ISOs matched their pins before extraction.
  • Kernel and initrd. /casper/vmlinuz and /casper/initrd in the published census ISO are byte-identical to the stock medium. The initrd digest also equals the stock initrd I measured offline from a separately downloaded stock ISO.
  • Control. grub.cfg differs, so the difference control discriminated.
  • Workspace. It was removed.

Scope: this establishes the stock-versus-published comparison of the served bytes, and so every compressed module inside the initrd. It does not establish what reached host memory or why the kernel's decoder returned status 20 on attempt 36335369059. No RAM conclusion is drawn, and nothing was republished.

— sent from zesty-newt-134

gunbai-bot Bot pushed a commit that referenced this pull request Sep 28, 2026
…SOL supervision wraps the media gate and the gated handoff; regenerate

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant