Repository navigation
mtcollins1 census: published-ISO boot-chain readback + module-identity instrument + decompression-refusal observation - #12487
Conversation
… instrument, decompression-refusal observation Deliverable 1: gunbc.machine_intake_mtcollins1_census_member_readback reads vmlinuz, initrd and grub.cfg out of the pinned stock and census ISOs on srv2 over fleet SSH (fleet-converge mode mtcollins1_census_member_readback), with grub.cfg as the in-run discriminating control. Deliverable 2: mtcollins1_census_module_identity_args (KMOD_LOG=info udev.log_level=debug SYSTEMD_LOG_TARGET=kmsg printk.devkmsg=on) as a census build input; option A per eager-owl-205. Upstream facts homed in extdeps.linux.module_decompress, extdeps.kmod.libkmod, extdeps.systemd.udevd. Deliverables 3-4: gunbc.machine_intake_kernel_module_decompression_observation rides ConsoleRetained; controls from runs 36335369059 / 36253081549 captures and an offline qemu receipt. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…hecker) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…t_command is the one -extract spelling (review 72051) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Review 72051 (xorriso argv fork): agreed and fixed in f0350b0. — sent from zesty-newt-134 |
briansrls
left a comment
There was a problem hiding this comment.
HOLD — exact-head module-identity instrument and published-ISO readback
Reviewed f0350b0, reconfirmed identical to refs/pull/12487/head immediately before publication. The diagnostic direction, declared cmdline input, shared xorriso builder and separate read-only-on-the-ISOs mode are accepted. Three source defects remain in the observation/admission consumers. These can be repaired without another physical boot or completion of the general BMC simulator.
1. P1 — any EINVAL insertion in the capture becomes the identity of an anonymous ZSTD failure
kernel_module_decompression_observation first collects all kernel decompression messages. Once that set is nonempty, it scans the ENTIRE capture for loader insert failures whose strerror equals Invalid argument, places every one in FailedModuleIdentity, and labels the result ModuleIdentityKnown. There is no invocation, worker/request, time, or other evidence join between these populations. A loader failure before the first ZSTD message is admitted just as readily as one afterwards.
Source-derived counterexample: prepend an unrelated module insertion failure with errno EINVAL to failed_attempt_excerpt. The two originally anonymous ZSTD failures now acquire that unrelated module as their known identity. Add two such records and both become the alleged identities, irrespective of their order or the number of actual decompression attempts. Reusing the current uas control but changing its errno from No such device to Invalid argument also promotes it to the known set without any evidence it decompressed badly.
The implication is reversed: a ZSTD decoder error returns EINVAL, but EINVAL does not establish a ZSTD decoder error. Upstream Linux v6.8 kernel/module/main.c returns EINVAL for other loader conditions (for example symbol version/namespace checks and invalid flags); module initialization can also return an error. libkmod v31's path-bearing INFO record reports the insertion's returned errno, not the internal failing kernel stage. Sources: https://raw.githubusercontent.com/torvalds/linux/v6.8/kernel/module/main.c ; https://raw.githubusercontent.com/kmod-project/kmod/v31/libkmod/libkmod-module.c ; https://man7.org/linux/man-pages/man2/init_module.2.html .
Keep the named insertion records as useful independent observations, but keep their association with anonymous decompression events unestablished/candidate unless a discriminating invocation-level join exists. A nearest-line or arbitrary time-window heuristic is not that join. The controlled QEMU case can independently know which file was deliberately corrupted; that experimental ground truth does not make the same attribution rule valid on an unconstrained real boot.
The narrow repair need not add kernel tracing immediately: represent anonymous kernel refusals plus the actually named insertion failures, and report the unresolved relationship honestly. An automatic known decompression identity claim needs the stronger producer. Add unrelated-EINVAL before/after, interleaved workers, multiple anonymous failures with one named insertion, and the controlled positive, through the real console/bundle consumer. Preserve the existing different-errno discrimination, but do not treat it as sufficient.
2. P2 — a recognized but unreadable refusal is dropped into 'no refusal in this capture'
decompression_refusal returns the same none for an unrelated line and for a line containing the exact failure prefix whose code cannot be parsed. The enclosing fold drops both and returns ModuleDecompressionNotRefused / no findings when the resulting list is empty.
For example [ 10.374590] ZSTD-decompression failed with status , or the same complete prefix followed by unreadable, renders as no refusal in this capture. Partial/truncated evidence is particularly relevant on the observation channel being repaired here. When a valid record is beside a malformed one, only the valid record survives, silently undercounting the recognized failure population.
Separate nonmatching lines from recognized-unparsed/pending records, preserving the original line and the failed field. Do not invent a numeric status or certify an exact zero/count over unparsed members. Likewise keep a malformed loader record as unread evidence rather than claiming that no loader record reached the capture. Unknown valid numeric ZSTD codes should continue to use the existing Other arm; there is no request to enumerate all codes.
Add malformed/truncated and valid-plus-malformed controls at mtcollins1_boot_console_reading / bundle rendering, with unrelated text and the census-reaching excerpt as positives. A parsing uncertainty must not become a physical hardware assertion either.
3. P1 — concurrent readbacks share and delete the same remote scratch files
member_scratch_path is a constant per side/member, for example /tmp/gunbc-mtcollins1-census-member-census-initrd. Every readback runs separate SSH operations for rm -f, extract, hash, and rm -f on that pathname. There is no per-attempt owned directory or admission of existing ownership.
The new mode is ExecutorDomain: fleet_converge_mutation_group_for keys it by the selected runner host, while census_member_readback_target() ALWAYS addresses srv2. Dispatches on srv1 and srv3 can therefore operate concurrently on the same remote scratch namespace. One can remove another's extracted member before its hash (a false read failure). Across revisions/pins, an extraction can replace the object between the other attempt's extraction and hash; all four command statuses can be zero while the returned digest belongs to the other attempt. Cleanup can delete a sibling's in-progress observation. The GRUB-difference control does not protect the subsequent kernel/initrd scratch objects.
This is a source-derived interleaving, not a claimed live srv2 incident or an independently executed xorriso race. Calling the operation read-only on /srv/bmc does not make its remote scratch effects isolated.
Use an actually created, exclusively owned temporary directory/identity on srv2 and derive every member path beneath it. A run/attempt label without exclusive creation is not sufficient if it can be reused. Admit prerequisite operations before dependent extraction/hash, and clean only this attempt's resources; preserve the initiating failure and cleanup result separately. The current read_member_remote executes extraction even after the initial clear fails and gives a later cleanup failure priority over the initial problem. A failed setup must not authorize writing an occupied path. Reuse existing filesystem/typed-argv authorities; no new remote agent or host job is requested.
Add a local/controlled extraction-route test with two simultaneous invocations, different member contents, and delayed hash/cleanup, plus occupied/setup-failure and successful repeat cases. The current seven readback tests supply abstract digest values to the pure comparison; they do not exercise this acquisition/ownership boundary. The actual srv2 observation may still wait until after merge once the source is corrected.
Accepted instrument and wiring
- The four arguments are declared input to
mtcollins1_seeded_image_input:KMOD_LOG=info udev.log_level=debug SYSTEMD_LOG_TARGET=kmsg printk.devkmsg=on. Their authorities are separated; the new build-key expectation is 5828991ca3dcace9, and the old output-digest pin is not silently relabeled as the new image. Keep the republish/readback/repin requirement and the live hold. No hand GRUB modification is needed. - The reported QEMU A/B is the right kind of transport/log-retention evidence: same intended setup, one deliberately corrupted module, missing pathname without the arguments and a path-bearing loader failure on ttyAMA0 with them. Its receipt excerpts clearly distinguish designed uas/later-driver additions from measured lines. I did not download/rehash the full QEMU logs or execute that experiment, so it remains author-run evidence. A manual A/B receipt is not an enrolled reproducible QEMU regression lane; preserve that distinction without making a general QEMU workflow a prerequisite for these local repairs.
- Option A's omission of per-device debug from SOL is explicit. The requesting device/modalias is not inferred. INFO logging can name failed insertion files but does not by itself guarantee correct attribution to a kernel event. Extra userspace logging and the inherited PID1 environment are diagnostic side effects, not changes to decompression validation.
ignore_loglevel/forced loading remain absent. - The new observation is consumed by the real ConsoleRetained constructor, diagnostic rendering and findings. It is not an inert note. It adds diagnostic evidence, not a claim that a required device/live root failed or a new power action.
- The readback mode is wired through the mode roster, wire/scope/key-demand/mutation folds, CI target roster, invocation, actual step and always-on-mode receipt upload. The target is the declared srv2 fleet endpoint/principal. It contacts no BMC and does not intentionally write
/srv/bmc; remote scratch still needs the ownership repair above. - The duplicate uncalled ExtractPathBestEffort operation is gone.
xorriso_extract_commandis the consumed shared exact-member builder, reached through typed argv rather than an independently copied SSH shell command. - Equal verified initrd bytes are sufficient for the scoped stock-vs-published payload comparison without rehashing every member. They are not evidence of the bytes delivered to the physical host or a diagnosis of RAM/kernel faults.
Other evidence qualifications / adjacent class
The readback's pure join rejects off-pin/unmeasured ISOs, but the wet producer evaluates all member reads before invoking that join. Therefore its pin verdict precedes comparison, not the remote extraction effects. Do not describe the current pure control as executing 'no extraction after pin refusal'. Move admission before dependent reads if that is the intended protocol.
An equal grub.cfg correctly fails the declared difference control, but the census image was not read overstates the cause. The same observation could mean the actual census artifact omitted the intended rewrite. Report failure of the control without deciding which producer was wrong. Similarly, hashing each ISO before reopening its path for member extraction assumes that publication keeps that object unchanged; cite/use that existing immutability contract or retain a coherent read transaction rather than inferring path equality is object stability.
The adjacent class is joining independent observations into a stronger causal/identity claim, or measuring a shared mutable name while reporting an attempt-owned result. Preserve evidence of an unexpected result rather than turning it into either a hardware diagnosis or an artificial clean reading. The previous Casper specimen is still not newly bound to run 36335369059 by this PR, and the older run is census-reaching, not fully qualified merely because this excerpt has no ZSTD message.
Verification and disposition
I inspected the complete new observation/readback modules, their witness specimens, the image-input/build-key delta, shared command builder, workflow/console consumers and the primary Linux/libkmod error contracts. Exact-head run 36375121892 has all six reported checks successful: generated, compiler, clippy, floor, emit-build and the witnesses aggregate. That does not discriminate the counterexamples above. The 7/7 local readback result is author evidence; no .dag suite, full initrd audit, QEMU boot, xorriso extraction, remote command, image publication, repin, merge or hardware operation was executed in this review.
Source: HOLD at f0350b0… for these three boundaries. Diagnostic direction accepted. Integrated live-boot HOLD unchanged. The changes requested are in the new local folds and readback transaction, not a demand to finish the whole acceptance matrix before gaining useful module-load visibility.
| Absent => [] | ||
| Present { value: f } => | ||
| if f.error == refused_decompression_strerror { | ||
| [FailedModuleIdentity { |
There was a problem hiding this comment.
[P1] EINVAL is not an invocation join
This selects every EINVAL insertion in the entire capture whenever any anonymous ZSTD refusal exists, including insertions preceding the refusal or from unrelated workers. EINVAL is also returned by other module-load stages. Preserve named insertion failures as independent/candidate observations unless an invocation-level producer links them to the anonymous decompression event. Add an unrelated EINVAL record to failed_attempt_excerpt: it must not become the known decompression identity merely because its strerror matches.
| let prefix = linux_module_zstd_decompress_failed_prefix as String | ||
| if !string_contains(s: l.rest, pattern: prefix) { none } else { | ||
| match parse_int(s: trim(join(split(s: l.rest, delimiter: prefix).skip(n: 1), prefix))) { | ||
| Absent => none |
There was a problem hiding this comment.
[P2] Preserve a recognized refusal whose code is unreadable
Returning none here makes the complete failure prefix plus a missing/malformed code indistinguishable from unrelated text. The parent then reports ModuleDecompressionNotRefused or silently undercounts alongside valid rows. Retain a recognized-unparsed/pending reading with the raw line/cause, and keep it visible through the console/bundle consumer; do not invent a code or a zero-refusal result.
| remote_answer_of(outcome: typed_argv_exec_over_fleet_ssh(target: target, context: context, argv: argv_words(command: command))) | ||
| } | ||
|
|
||
| fn read_member_remote(target: SshTarget, context: FleetSshExecutionContext, iso_path: NonEmptyStr, side: IsoSide, member: CensusIsoMember) -> MemberReading { |
There was a problem hiding this comment.
[P1] This scratch path is shared across independently admitted dispatches
The mode's concurrency is keyed by executor host, but every invocation targets srv2 and uses the same per-side/member /tmp name. Two executions can remove/overwrite one another's extracted members between SSH legs, so a successful hash need not refer to this attempt's extraction. Create an exclusive remote workspace, carry its identity through extract/hash/cleanup, refuse failed setup before dependent effects, and test overlapping attempts with different contents. No live srv2 mutation is needed for that control.
…ords kept; owned srv2 workspace 1. EINVAL insert failures are unresolved candidates; every named insert failure is kept as its own observation; no nearest-line rule promotes one to the refusal's identity. 2. A recognised refusal or loader record that cannot be parsed is retained with its line and the failed field; the refusal count is not stated complete beside an unparsed record. 3. The readback gates extraction on the ISO pins, acquires an exclusively created mktemp -d workspace on srv2, extracts under it, never hashes a failed extraction, and removes that workspace once, reporting cleanup beside the verdict. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
briansrls
left a comment
There was a problem hiding this comment.
HOLD — exact-head rebind; attribution/parser repairs accepted, one remaining workspace-outcome finding
Reviewed 6fee01d, the single successor of f0350b0. Reconfirmed identical to refs/pull/12487/head immediately before publication. The two diagnostic blockers and the fixed shared-scratch path are repaired. Keep those changes. One P2 remains at the new workspace acquisition/result boundary; this is not a request for a new transaction service, a physical boot, or the general BMC matrix.
P2 — an unacknowledged remote creation is reported as a workspace that was never created
member_workspace_of returns WorkspaceRefused for every nonzero RemoteAnswer and for an unreadable/rejected output path. census_member_readback then unconditionally assigns WorkspaceNeverCreated, whose renderer says workspace=never created (nothing extracted). That is justified when the pin/context gate prevented the creation call. It is not justified after the creation call was issued but its outcome was not established.
Source-derived counterexample:
- The admitted fleet-SSH command reaches srv2 and
mktemp -dsuccessfully creates its new directory. - The SSH connection fails before the local caller receives a complete result/exit status. It can also fail after some or all of the pathname reached stdout.
- The local RemoteAnswer has exit 255.
member_workspace_ofrefuses and discards stdout in this branch. - The returned attempt says never created, and has no owned workspace cleanup or explicit unresolved-residue record, although a directory can remain on srv2.
This is not an allegation that mktemp's exclusive creation is broken, nor an independently reproduced live SSH incident. OpenSSH documents 255 as a client/transport error, not proof that the remote command did not run. The inspected transport preserves a ran leg's exit/stdout/stderr before remote_answer_of and member_workspace_of collapse it. remote_answer_of also represents a refused leg as exit 255, so the same integer cannot establish whether creation was attempted. A zero-exit but malformed/truncated reply likewise establishes no usable owned path, not non-creation.
The cleanup side has the dual wording problem: workspace_cleanup_of converts any nonzero result into WorkspaceNotRemoved and renders NOT REMOVED, although a lost SSH result can follow a successful remote removal. Keep removal unconfirmed distinct from a proved remaining path. The current fail-closed attempt verdict is appropriate and must remain; the finding is false residue/history, not that this branch currently returns ExitSuccess.
Narrow repair: preserve no-creation-attempt, acquired workspace, and attempted-but-unconfirmed creation as different outcomes. Retain the full command observation, the target/attempt context and any returned pathname as evidence; do not discard it or treat an unadmitted name as authority to delete. Reserve never created for a path whose creation was not issued or whose absence is actually established. An unconfirmed acquisition may continue to refuse with possible residue explicitly outstanding; this review does not require inventing ownership or automatic recovery after a lost reply. Similarly report attempted-but-unconfirmed cleanup without claiming the directory definitely remains. No extraction on an unadmitted workspace, and no cleanup by guessing or sweeping the shared prefix.
Add controls at the actual attempt/render consumer for: no creation requested after pin refusal; issued creation with exit 255 both with and without partial pathname; exit 0 with unusable/multiple path output; successful acquired path; and cleanup reply loss beside a retained comparison result. They must require the unknown/residue statement and non-holding verdict, not just a nonzero process result. Keep the existing legitimate mktemp failure and successful cleanup cases. Where execution was refused before dispatch, preserve that fact instead of making it indistinguishable from a lost response.
Primary contracts checked: https://man.openbsd.org/OpenBSD-7.6/ssh.1#EXIT_STATUS ; https://www.gnu.org/software/coreutils/manual/html_node/mktemp-invocation.html . This is about the remote acknowledgement boundary, not a claim that an ordinary local mktemp output error necessarily leaks a directory.
Accepted: the anonymous and named populations no longer manufacture an identity
RefusalIdentityCandidates is explicitly UNRESOLVED; the known-identity arm is gone. All named insertion failures remain their own population, including different errno values and loader-only captures. The earlier unrelated-EINVAL specimen is a candidate, not an attributed decompression failure. No nearest-line heuristic replaces the missing invocation link. The QEMU experiment's controlled corruption remains separate experimental ground truth, not a general inference about real host failures.
The existing console consumer still calls this observation and renders/finds it. It does not create a new device, live-root, RAM or SOL verdict. The request/modalias remains explicitly unread at the selected SOL logging level. The later-driver arm means a matching prefix printed later in the retained sequence; it is not successful driver-load or device-availability evidence.
Accepted: recognized unparsed records survive and the count is qualified
The new stem recognizers distinguish nonmatching text from recognized-but-unparsed status/path/errno records. A malformed-only refusal is not dropped into NoRefusalRecorded; a valid-plus-malformed population keeps both and does not claim a complete refusal count. Uncatalogued numeric status stays in the Other arm. The whole named-failure population is retained even without a kernel refusal.
Non-blocking evidence precision: UnparsedRecord.line currently contains l.rest, after the timestamp and CR/NUL normalization have been stripped, not the full original raw console line. The retained console remains the raw-evidence reference, but do not describe this field alone as a verbatim timestamped record. Carry the timestamp/source location or original line when using it for cross-event correlation. The new tests assert the message portion, not preservation of the original instant.
Accepted: successful acquisition no longer shares the fixed scratch namespace
The actual wet producer checks census_pin_gate before calling mktemp or xorriso. The admitted mktemp result supplies the workspace for all six side/member paths; the new builder is admitted at the existing typed-argv seal. read_member_remote issues sha256sum only inside successful extraction, and cleanup occurs once after the normal returned readback, with its result kept beside the comparison. The held verdict requires both identical boot-chain readings and successful workspace cleanup. The old per-member shared /tmp names and pre-extraction rm are gone.
GNU mktemp's create-new-directory contract is a valid basis for separation of successful concurrent acquisitions. The supplied-answer controls are honestly scoped as pure decision controls: they do not execute xorriso, overlapping fleet transactions, or an interruption between remote operations. I did not run those either. I am not retaining the old collision finding merely because an empirical collision test has not run; the remaining finding is the uncertainty/result projection above. Keep the actual srv2 readback as its separate execution obligation.
The equal-GRUB renderer now reports that its difference control did not discriminate without choosing why. One older witness comment still says equal GRUB means the census image was not read; align that stale comment with the corrected predicate. This is non-blocking and does not require another control type.
Evidence, adjacent class and disposition
The new build-key/logging input and workflow wiring are unchanged by this successor. The republish/readback/repin ordering remains required; neither a passing pure comparison nor this review is a newly executed published-ISO audit. The previous separate Casper screenshot is not newly bound to run 36335369059, and nothing here attributes status 20 to RAM. As before, comparing reopened ISO paths assumes their bytes remain stable over the measurement; a per-attempt scratch directory is not itself a source-ISO snapshot.
The 26/26 result is author-reported. I inspected both complete new production modules, both witness files, the entire successor delta, the actual remote-answer/SSH composition, and the primary mktemp/SSH contracts. I did not execute the .dag suites, QEMU, archive extraction, overlapping acquisition, fleet SSH, cleanup on srv2, publication, repin or a hardware operation. At the exact-head run 36382667860, compiler, clippy, generated and emit-build had succeeded; floor was still running at the last read.
Adjacent class: an exclusively created resource is not the same fact as an acknowledged creation, and an unacknowledged mutation must not become an invented absence or an invented remaining resource. The safe no-extraction branch is already present; finish its typed residue/result account. This is a bounded repair to the new workflow, not an instruction to redesign SSH or achieve impossible acknowledgement guarantees.
HOLD at 6fee01d… for the one P2 above. The earlier causal-attribution and malformed-record defects are closed, and successful workspace isolation is accepted. No merge, enqueue, dispatch, BMC/host action or additional live authorization was performed. The integrated live-boot HOLD is unchanged.
| PinsAdmitted { stock_iso: s, census_iso: c } => | ||
| match member_workspace_of(answer: remote(target: target, context: context, command: mktemp_directory_command(template: census_member_workspace_template))) { | ||
| WorkspaceRefused { reason: r } => | ||
| CensusMemberReadbackAttempt { readback: BootChainWorkspaceUnavailable { reason: r }, cleanup: WorkspaceNeverCreated } |
There was a problem hiding this comment.
[P2] A refused mktemp reply does not establish that no workspace was created
This assigns WorkspaceNeverCreated after an actual creation attempt. If mktemp creates its directory on srv2 but SSH loses the response/exit status, RemoteAnswer.exit_code is 255; member_workspace_of refuses and this receipt falsely says never created. The nonzero branch also discards any pathname already received on stdout. Keep unattempted creation separate from attempted-but-unconfirmed creation, retain the full answer and possible residue, and continue to perform no extraction on that arm. Do not clean an unadmitted path or sweep the prefix. The cleanup-result projection should likewise say unconfirmed after a lost reply, not assert that removal did not occur. See the review for the narrow control pairs; this is not a claim that mktemp's exclusive creation fails.
…4663150) An issued mktemp whose reply was lost (ssh 255) or whose output admits no path is CreationUnconfirmed with residue outstanding and the returned text kept, never 'never created'; a lost rm reply is RemovalUnconfirmed, distinct from rm's own failure and from an undispatched rm. Legs keep the transport's not-dispatched arm instead of collapsing to exit 255. Controls run at the attempt/render consumer. UnparsedRecord carries its timestamp. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
briansrls
left a comment
There was a problem hiding this comment.
APPROVE-MERGE — exact-head rebind
Reviewed b6ae221, the single successor of 6fee01d; reconfirmed identical to refs/pull/12487/head before publication. This closes the remaining workspace-outcome finding in review 5334663150. The previously accepted attribution/parser and successful workspace-isolation repairs remain intact. Proceed through normal exact-head and merge-queue gates.
Remote acknowledgement is no longer invented resource history
The actual producer uses remote_leg/remote_leg_of for creation and cleanup, preserving TypedArgvExecRefused as LegNotDispatched instead of flattening it into exit 255. Returned exit/stdout/stderr remain separate from that pre-dispatch refusal.
workspace_acquisition_of now distinguishes an undispatched call, a reported command failure, and an unconfirmed creation. Exit 255, including partial pathname output, and zero exit with an unusable reply yield CreationUnconfirmed. Its receipt retains the returned text, explicitly names possible residue on srv2, and does not supply that text to extraction or deletion. The SSH authority explicitly notes that a remote program's own exit 255 is indistinguishable from a client error; treating that case as unknown is appropriately conservative.
workspace_removal_of keeps failed, unconfirmed, and undispatched cleanup distinct. In particular the 255 arm says that the path may or may not remain, while preserving the comparison result. Only Removed combined with an identical boot chain holds. The former false WorkspaceNeverCreated projection for an unacknowledged creation is gone.
The producer consumes the tested fold
census_member_readback still checks the actual pin gate before evaluating the creation call. Its admitted arm calls census_member_attempt_of with the real creation result and deferred member-read/cleanup functions. The fold calls neither callback when acquisition is unadmitted. After acquisition, it obtains the member comparison and retains the cleanup result beside it. Per-member hashing remains inside successful extraction, and successful acquisitions derive all member paths beneath their own created workspace.
The new controls construct legs through remote_leg_of and reach this same attempt/render consumer. They cover pin refusal; reply loss with and without a pathname; zero exit with multiple/outside paths; reported creation failure versus undispatched creation; successful acquired-and-removed execution; and lost, failed, or undispatched cleanup. They require both the result standing and relevant receipt text, with the identical-chain positive preventing a blanket-refusal implementation from satisfying the negative controls.
These are supplied-answer execution controls, not proof that real SSH, xorriso, overlapping attempts, or interrupted cleanup ran. The callback branching is inspected production source; the tests do not independently observe remote side effects. That scope is disclosed and is not a reason to keep the resolved source HOLD.
Diagnostic precision and adjacent class
UnparsedRecord now carries at and the renderer consumes it. Its normalized message remains distinct from the verbatim retained console. The stale equal-GRUB witness comment now says the difference control failed without selecting its cause. Independent anonymous refusals/named insertion failures remain unresolved candidates, not manufactured invocation identities.
Adjacent class: handling a returned uncertain result does not guarantee recovery after the worker disappears before producing a receipt. Keep possible residue with the dispatch's provenance and never sweep or delete an unadmitted path to clear it. Likewise, the per-attempt extraction workspace is not a snapshot of the reopened source ISO paths. Those remain explicit operational/evidence limits, not new requirements for a transaction service or a broader harness in this rebind. Do not generalize the mktemp-specific normal-failure interpretation to arbitrary mutating commands or new wrappers.
Evidence and landing scope
The reported 17/17 member-readback and 13/13 diagnostic claims are author-run evidence; I did not rerun the compiler, claim suites, QEMU, extraction, SSH, or fault-injection controls. I inspected the successor, production branching/result projection and consuming witnesses, and checked the primary SSH/mktemp contracts. At my exact-head CI lookup, witnesses run 36387741971 was queued, not completed green. Preserve normal required checks and the actual claim results.
This approval does not claim the srv2 published-ISO readback has executed. After landing, that separately admitted readback can supply the outstanding physical-file receipt; image publication/readback/repin remains necessary for the new logging build input before a later authorized boot. No pixel/host-health conclusion, RAM attribution, or new binding of the historical Casper specimen follows from this review.
APPROVE-MERGE at b6ae221…; the source HOLD on #12487 is cleared. The integrated live-boot HOLD is unchanged. No merge, enqueue, dispatch, package installation, image mutation, credential use, or host/BMC operation was performed.
Primary contracts: https://man.openbsd.org/OpenBSD-7.6/ssh.1#EXIT_STATUS ; https://www.gnu.org/software/coreutils/manual/html_node/mktemp-invocation.html .
|
Deliverable 1 executed. Run 36396523377 dispatched from main at 5402385 with mode
Scope: this establishes the stock-versus-published comparison of the served bytes, and so every compressed module inside the initrd. It does not establish what reached host memory or why the kernel's decoder returned status 20 on attempt 36335369059. No RAM conclusion is drawn, and nothing was republished. — sent from zesty-newt-134 |
…SOL supervision wraps the media gate and the gated handoff; regenerate Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Work item: mtcollins1 census — name the module that fails ZSTD decompression + published-ISO initrd readback. The live-boot HOLD is untouched. This PR boots no hardware, edits no image by hand, force-loads nothing and disables no validation.
What the evidence already says (no RAM conclusion)
mtcollins1-boot-receiptshas digestsha256:16dcc1b34233cea3b338104a6cff5d88a46f8b6c2e815443653da5271062dd9b. The SOL capture inside it has sha256761e79ca…fcee87. It shows twoZSTD-decompression failed with status 20lines at 10.374590 and 10.380475.sha256:ca38324ebcad6477911b0f56a86f91a0785a77dc31163ede6b9afc616fde3496; SOL capture sha25683e818fb…8f5161. It has no refusal, and it has the renesas "fallback to ROM" warning.kernel/module/decompress.c. The printk carries only the decoder status and never names a module. Status 20 is zstdcorruption_detected, which the kernel returns as-EINVAL.1. Published-image boot-chain readback (built; executes after merge)
gunbc.machine_intake_mtcollins1_census_member_readback, fleet-converge modemtcollins1_census_member_readback. It reaches srv2 over fleet SSH, likemtcollins1_census_medium_readback, because srv2 has had no runner slots since 2026-09-19. Each attempt is one transaction:mktemp -d /tmp/gunbc-mtcollins1-census-member.XXXXXXXX, which creates a new 0700 directory and never reuses a name. Concurrent dispatches share srv2 but never share a path; the mode's concurrency key is the runner host. If the workspace can't be created, nothing is extracted./boot/grub/grub.cfg,/casper/vmlinuz,/casper/initrd), xorriso extracts into the workspace, and only a successful extraction is hashed.rm -rf --one-file-system. The outcome is reported beside the verdict, never in place of it.Workspace outcomes at the remote acknowledgement boundary. Each outcome states only what is established:
CreationNotRequested: a gate stopped the attempt beforemktemp.CreationNotDispatched: the transport refused the leg, so the invocation was never made.CreationFailed:mktempreported its own non-255 status, so nothing was created.CreationUnconfirmed: ssh returned 255, which is its client status and does not show the command didn't run; ormktempexited 0 with output that isn't exactly one path under the prefix. Residue may remain. The returned text is kept, and no unadmitted name is used to extract or delete.Removed,RemovalFailed(rm's own failure),RemovalUnconfirmed(lost reply, not proof the directory remains), orRemovalNotDispatched.The attempt holds only on an identical chain and
Removed. Nothing under/srv/bmcis written.The verdict names the first member in boot order that differs.
grub.cfgis the member the remaster rewrites, so if it compares equal the run refuses withcontrol_did_not_discriminate. That means the difference control failed; its cause is not chosen: the census file may not have been the one read, or the published artifact may lack the rewrite. Equal kernel and initrd bytes establish only the scoped comparison of stock against published bytes, not what reached host memory or why a decoder refused. If the initrds are equal, every module inside is equal by construction, so modules are not re-hashed one by one. A differing initrd is reported as the first differing boundary, and descending into its layers is the declared next step. For reference, the stock/casper/initrdsha256 measured offline isdeb2c288c12e6f35c2b9242134116ea03f7d4eb91f0c354ecd993680ffabbbf7.Scope of the acquisition controls. They run at the attempt/render consumer (
census_member_attempt_of, the fold the producer runs), with legs built through the real transport adapter. The cases are:mktempfailure, and an undispatched creation;They do not execute overlapping attempts against a real host. Exclusivity of a successful acquisition rests on mktemp's create-new contract.
Not yet executed: the mode only exists once this PR is on main, and the fleet key is federated to main runs. After merge: dispatch
fleet-convergewith modemtcollins1_census_member_readback, then attach the receipt here.2. Module-identity instrument (modeled cmdline input)
The instrument is
mtcollins1_census_module_identity_args=KMOD_LOG=info udev.log_level=debug SYSTEMD_LOG_TARGET=kmsg printk.devkmsg=on. It is appended to the censusgrub_kernel_cmdlinebuild input, so it changes the build key to5828991ca3dcace9and therefore the digest. The boot's medium readback refuses until the image is republished, read back and repinned. No one edits GRUB by hand.udev.log_level=debugalone does not reach SOL. Measured against the versions in this initrd (systemd-udevd255.4-1ubuntu8.10, libkmod31, kernel6.8.0-71):init-top/udevruns udevd withSYSTEMD_LOG_LEVEL=info;udev.log_level=debugoverrides it, because the cmdline is parsed after the environment. But daemonized udevd logs to stderr =/dev/console, and both captures' cmdlines end--- console=tty0. So/dev/consoleis the KVM, not SOL. Fix:SYSTEMD_LOG_TARGET=kmsg.Failed to insert module '<path>': <strerror>is logged at INFO, and libkmod's default priority is ERR. Fix:KMOD_LOG=info. The kernel passes it into/init's environment, initramfsinitdoes not scrub the environment, and udevd inherits it.printk.devkmsg=on.ignore_loglevelwas declined: it would flood the 115200 line and perturb the timing of a possible race.Upstream facts are each homed in their own module:
extdeps.linux.module_decompress,extdeps.kmod.libkmod,extdeps.systemd.udevd, andextdeps.linux.kernel(printk.devkmsg).Offline control (condition 2 of the ruling). Setup:
-M virt, oneqemu-xhcidevice, TCG./casper/vmlinuz, plus stock/casper/initrdwith exactly one edit: 64 bytes inverted inside the zstd body ofxhci-pci.ko.zst, frame header intact. Edited initrd sha25683fd2a47…6f96.--- console=tty0kept.Results:
7e8db2f8…5552):ZSTD-decompression failed with status 20, and nothing names the module. This reproduces the hardware signature.b8b33393…6bcf), on ttyAMA0:The control was run by hand in a session. It has no entry point yet; a modeled qemu instrument is the follow-up if this needs to be re-derivable.
Side effect to know about. The environment variables are inherited by the live system's PID 1, so the census boot's SOL will carry somewhat more INFO output after the initramfs as well.
3. The fact at its real strength
gunbc.machine_intake_kernel_module_decompression_observationkeeps two independently observed populations:ZstdErrorCode, with no module named;Nothing in a capture links one invocation to the other. A zstd refusal returns
-EINVAL, butEINVALdoes not establish a decompression failure: the v6.8 loader also returns it for symbol-version and namespace checks and for invalid flags. So anInvalid argumentinsert failure is a candidate, and the identity readsRefusalIdentityCandidates(UNRESOLVED) orRefusalIdentityUnread. There is no "known" arm. A known identity needs an invocation-level link, which is the declared next rung.Records that can't be parsed are kept. A recognized refusal line whose status can't be read, or a loader record truncated before its path or errno, is retained with its line and the field that failed. A refusal count is never stated as complete while an unparsed refusal record stands beside it, and uncatalogued numeric statuses stay
ZstdErrorCodeOther.The observation is carried by
ConsoleRetainedinmtcollins1_boot_diagnostic_bundle, the attempt-bound console route, and contributes one finding when a refusal record is present. It claims nothing about device availability, live-root acceptance, SOL health or hardware, and the renesasfallback to ROMline is not read.4. Executing controls
kernel_module_decompression_observation_witness_test(13 claims):corruption_detectedand identity UNREAD.EINVALfailure is only a candidate.mtcollins1_census_member_readback_witness_test(13 claims) covers the comparison verdicts, plus the pin gate that precedes extraction, workspace admission, a failed extraction never hashed, and cleanup retained beside the verdict.Coordination: swift-wolf-904 (#12434, the SOL collector) and eager-koi-811 (KVM). This observation reads the retained capture on the existing console route and adds nothing to the collector. With this instrument, udevd writes to kmsg, which the kernel prints on every console, so the KVM (tty0) shows the same loader records as SOL.
🤖 Generated with Claude Code