Repository navigation
mtcollins1 read-only probes: served-UI crawl + KVM observer-only mode (viewer features seeded; hold released always()) - #12830
Conversation
…ot inside the UI root page fleet-converge run 36721915217 refused the boot: the observer journalled "page.evaluate: Execution context was destroyed, most likely because of a navigation" and released as login-unobserved. It loaded "/" only to borrow an origin and ran the session POST inside that document, which was replaced under it. The session POST, the services read and the session DELETE need a cookie jar, not a page, so they now go through the browser context's request client. The sessionStorage keys the viewer reads are seeded by a context init script. The root page is never loaded; the only in-page evaluations left are on viewer.html. No retry was added, and every typed refusal and journal word is unchanged. The loopback transport's root page now replaces itself on DOMContentLoaded, and a new wet control asserts the login is seen and the root is never served. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ndle (source.min.js) Operator decision msg_b3c77f62 (option A on escalation msg_13d5904d): a modeled read-only route to the firmware's UI bundle, so vendor codes such as cd_error_code are read from the vendor's source instead of escalated. - extdeps.bmc.megarac: megarac.Ui.GetServedBundle (readonly, no session, bytes as served) - gunbc.machine_intake_mtcollins1_ui_bundle_observe: mc info firmware revision first, refuse unless 0.32; fetch; sha256; report MATCH or DRIFT against the cited 2026-09-27 read (never refuses on drift); receipt on every path - fleet-converge mode mtcollins1_ui_bundle_observe: a mode row on the shared job, reusing the fan lane's credential prelude; bytes + receipt uploaded always() - witness: revision parse, the 0.32/0.33 discriminating pair, unread revision, digest match/drift Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… compiler ui_bundle_observe imported the boot diagnostic bundle for two IPMI deadlines, putting 1019 modules (v2.compiler.*, host_effect_realize, roadmap) in a fetch's closure; the deadlines are a fact about reaching this unit's BMC, so they move to gunbc.machine_intake_mtcollins1_access_observation (closure 1019 -> 167). kvm_still's sol_hold import was dead; sol_hold borrowed the gunbc run step pipeline's refusal marker and now owns its own (closure 347 -> 96). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…e budget and its read bound The deadlines keep their names in gunbc.machine_intake_mtcollins1_access_observation, the destination #12800 uses, so the two branches do not fork where those rows live. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…al BMC, no boot; the SPA navigation is journalled - gunbc.machine_intake_mtcollins1_kvm_observer_observe: under the unit hold, log in, establish the viewer, one terminal still, release session/browser/process; journal, stills and receipt uploaded always(). Verdict read from the finish record. - kvm_still: every main-frame navigation is a `navigated` journal line (seq, phase, url) through KvmJournalWord / kvm_journal_line / the parser; login waits for Playwright's networkidle instead of the first DOMContentLoaded (run 36721915217). - maintenance_hold: KvmObserverProbe owner arm + admitted acquirer (the boot takes the one KVM seat under this hold and refuses its handoff when the seat is busy). - step timeouts: resolve budget + each probe's declared read bound (workflow-side fold). - loopback control asserts the recorded route; verdict claims over supplied records. - fleet-converge.yml regenerated. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
briansrls
left a comment
There was a problem hiding this comment.
HOLD — exact head 46452fdb4f4900c432df4f21e5b4b4c38cb05116
The UI-bundle BMC route is configuration-read-only as claimed: mc info, then an unauthenticated GET /source.min.js only under firmware 0.32; drift is evidence rather than a failed read. The KVM mode also introduces no media, power, boot-device, or SEL write. Four blocking issues remain.
1. The KVM route still puts login inside the navigating root document
This head still does goto(base + '/'), waits for networkidle, and then performs the session POST, sessionStorage writes, and service read through page.evaluate. That is the same unjustified document boundary behind run 36721915217; networkidle is not a proof that the SPA cannot navigate after the wait resolves.
The new loopback assertion proves only that the handler records the observer's own ordinary landing and viewer navigations. It does not reproduce a post-DOMContentLoaded replacement of the root document, so the old route remains green. The already-reviewed repair in #12786 moved POST/services/DELETE to ctx.request, seeded storage through ctx.addInitScript, and never depended on the root document. Preserve that boundary. Navigation observation can be layered around the viewer, or performed on a separate diagnostic page that carries no login operation.
2. Both timeout folds omit the credential prelude
The generated steps execute a Secret Manager fetch with curl --max-time 60 before gunbc run, but fleet_converge_mtcollins1_probe_step_timeout_minutes adds only the 15-minute resolve budget and the probe's BMC/read bound.
Consequently the declared worst-case work does not fit the emitted timeout:
- UI: 60 s credential fetch + 900 s resolve + 35 s reads = 995 s, but the step is 960 s.
- KVM: even using the actual 45+15+10+5 s observer path, 60 + 900 + 75 = 1035 s, but the step is 1020 s. The current
probe_boundadditionally double-counts establishment becausemtcollins1_kvm_still_allowance()already includes it.
Derive a typed credential-prelude bound from the prelude authority, combine all post-dispatch bounded work once, round once, and regenerate the workflow. This should make the current shapes at least 17 minutes for UI and 18 for KVM.
3. Teardown success is decided by substring
kvm_probe_teardown_seen accepts string_contains(session_release, "released") and string_contains(browser_close, "closed"). A reachable value such as release-failed detail="already released?" or close-failed detail="browser closed unexpectedly" can therefore satisfy the clean-teardown verdict.
Parse or project the first result token into typed arms and require exact released / closed. Add reds where failure detail contains those words.
4. The unit-hold lifecycle is not closed on abnormal termination or unresolved seat release
On the ordinary clean path, the hold correctly encloses start, establishment, terminal still, session/browser/process teardown, receipt, and then release. But this exact head has no observed-dead-holder recovery: cancellation or a step timeout after acquisition skips unit_hold_release, while the operator release explicitly refuses a KvmObserverProbe holder. That can permanently block the boot lane until the slot is edited out of band.
Conversely, when teardown reports that the web session was not released, the failed outcome is still passed to unit_hold_release, so the interlock is removed without evidence that the one KVM seat is free. Integrate the observed-dead recovery from the hold work before adding this holder kind, and make unresolved seat cleanup retain or transition the hold until a typed observation establishes that the seat no longer conflicts with boot.
The generated artifact and witness lanes were green at the reviewed head; these are source/lifecycle findings, not regeneration drift. Rebind after the navigation-safe request-client route, complete timeout fold, exact teardown classification, and closed hold recovery are present.
…, only viewer.html a document); fold each phase once; typed teardown words Merges session/swift-raven-18 (#12786): the session POST, services read and DELETE go through the context request client, sessionStorage is seeded by an init script, and the loopback root replaces itself on DOMContentLoaded so the old in-page login is red. The navigated journal now records viewer.html and any #login reload under the held viewer (read from the UI bundle fetched by run 36778446928). - step timeouts: resolve + gunbc_ci_sm_max_time + probe bound, in seconds, rounded once; the KVM bound is mtcollins1_kvm_still_allowance alone (no double-counted establish wait). - teardown results: RS KvmJournalWord rows, parsed by leading word into KvmSessionRelease / KvmBrowserClose; the verdict needs exactly released/closed; reds whose detail holds the words. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s, viewer console journal, served-UI crawl
- maintenance_hold: probe_hold_release_decision / _refused / _committed and the admitted
mtcollins1_kvm_observer_release_unit_hold: release only KvmObserverProbe{this run} at the
observed generation; free, foreign or stale is a reported no-op. Reds: foreign holders,
stale generation. Frontier: runner death / job backstop until #12555 lands on main.
- kvm_observer_observe: mtcollins1_kvm_observer_release_wet (typed owned-process arms), run
by a new always() step after the probe.
- ci_spec: the ui_bundle, kvm probe and release steps are bash_build nodes rendered by
bash_emit_stmts (review 73415); no Scaffold, no borrowed fan marker.
- kvm_still: page-console / page-error journal events (viewer never opened /kvm on 36779479938).
- ui_bundle_observe: one read-only served-file observation. Seeds source.min.js and viewer.html;
further paths parsed from what was served (<script src>, data-main, ./libs/kvm/*), followed
to closure within a declared 48-file budget; one typed receipt row per file. New
extdeps.tools.gzip (decode-or-pass-through) so served gzip is parsed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…or UnimportedBareProvider) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…mitted from the served source.min.js; gzip imports Unit (floor) Run 36788605665: the served viewer.min.js reads sessionStorage.features.indexOf(...) unguarded in its KVM view's initialize(), so with features unset it throws and never opens /kvm. - extdeps.bmc.megarac: megarac_ui_features_0_32 (73 names), EMITTED by gunbc.machine_intake_megarac_ui_features megarac_ui_features_row_emit from crawl run 36787375313's retained source.min.js, refusing unless it digests to the 2026-09-27 pin; megarac_ui_features(firmware) and the vendor storage shape megarac_ui_features_json. - kvm_still: the start looks the list up by firmware (no row -> not started); the launch writes features.json and the init script seeds sessionStorage.features; features-unreadable cause. - loopback viewer reads sessionStorage.features.indexOf unguarded before /kvm; new wet red a_viewer_without_its_feature_list_never_opens_kvm (enrolled). - extdeps.tools.gzip imports Unit from std.types (floor AmbiguousBareNameRead). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
briansrls
left a comment
There was a problem hiding this comment.
HOLD — exact head 831d827b7fa99ebec2a3bf7af38a2b4b6d7bba4e
The request-client route, complete timeout fold, typed teardown words, bash-emitted steps, served-file crawl, and firmware-bound feature seeding are accepted. The exact-head hardware run also establishes the clean path: the KVM observer step, always-release step, and artifact upload all completed successfully.
One lifecycle blocker remains from review 5372184063: the unit hold is still released before unresolved viewer cleanup has been ruled out.
On the ordinary path, mtcollins1_kvm_observer_observe_wet passes every result of mtcollins1_kvm_observer_observe_held to generic unit_hold_release. That generic release commits the hold release first and merely preserves the supplied success/failure outcome. Therefore a typed KvmProbeFell caused by release-failed, close-failed, or an unresolved owned process still frees the interlock.
The new always() path has the same ordering problem. mtcollins1_kvm_observer_release_wet does:
owned_process_release(...);mtcollins1_kvm_observer_release_unit_hold(...);- only afterward computes
kvm_probe_process_let_go(process)and returns failure when cleanup was unresolved.
So an OwnedReleaseFailed or unobservable process can leave the function red after the hold has already been released. Likewise, if the ordinary path already freed the hold after a session/browser teardown failure, the always step sees an already-free slot and cannot restore the interlock. A subsequent boot may then run while this run has not established that the single-viewer seat is free.
Bounded repair
- Do not call generic
unit_hold_releasefor a probe outcome whose typed teardown is unresolved. - In the always path, assess
kvm_probe_process_let_gobefore releasing the hold. If the process is not proved gone, leave this run's hold in place and fail. - For a process that is gone but whose session/browser teardown record is failed or absent, retain or transition the hold until a typed observation establishes that the BMC viewer seat no longer conflicts with boot.
- Add discriminating controls proving that
OwnedReleaseFailed,release-failed, andclose-faileddo not free the probe hold. - Before landing, integrate the
KvmObserverProbeowner with #12555's observed-dead-holder recovery (or land/rebase after that recovery). The declared runner-death frontier is otherwise still a permanent lockout on this exact tree.
The successful hardware execution is valuable clean-path evidence, but it does not exercise this failure ordering. HOLD only on keeping the interlock closed when cleanup is unresolved.
…chat blocker 4a) Both release paths -- the probe's own exit and the always() step -- now keep the unit hold unless the web session reads released (or no-session), the browser reads closed (or no-browser), and the observer process is observed exited (or never launched). login-unobserved, failed/http-/unrecognised words, an unresolved process or an unreadable journal keep the hold and say why. Reds for each, including failures whose detail contains the success word. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…-0.32 row (minimum from #12830 for the boot) Run 36788605665: the served viewer.min.js reads sessionStorage.features.indexOf(...) unguarded in its KVM view's initialize(), so with features unset it throws and never opens /kvm. - extdeps.bmc.megarac megarac_ui_features_0_32 (73 names), emitted by gunbc.machine_intake_megarac_ui_features megarac_ui_features_row_emit from the retained served source.min.js, refused unless it digests to the 2026-09-27 pin (row ported with it); megarac_ui_features(firmware) and the vendor storage shape; extdeps.tools.gzip for the decode. - kvm_still: no row for the firmware -> not started; the launch writes features.json and the init script seeds sessionStorage.features; features-unreadable cause. - loopback viewer reads sessionStorage.features.indexOf unguarded; enrolled wet red a_viewer_without_its_feature_list_never_opens_kvm; megarac_ui_features witness. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…in's MtCollins1CensusQemuHostObserve and this PR's two probe modes; fleet-converge.yml regenerated Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…e holder is recoverable when observed dead (side-chat blocker 4b)
- mtcollins1_maintenance_hold rebuilt on main's: KvmObserverProbe { run_id, process: ProcessIdentity,
pid_namespace } like BootRun (tag/detail/label/decode); the acquirer captures self_process_identity
and refuses if it cannot; holder_liveness_route (pure) routes a probe holder to its process, so
unit_hold_acquire's observed-dead recovery (#12555) frees a dead probe's hold.
- witness: probe owner renders and routes to its process; a processless one is not observable.
- mtcollins1_boot_dry_realization: 10 observer operands, established only over a seeded feature list.
- observer, loopback transport, rosters: this branch's side (main held their older forms).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…laimed derivation (review 73613) The bundle the list is read from is AMI's proprietary MegaRAC UI, not redistributable and not reachable from CI, so a CI-diffed gate is unavailable: a declared boundary (DESIGN 4b/4d), not a below-ceiling choice. The list moves out of extdeps.bmc.megarac (observations are receipts in the observing layer, DESIGN 3) into gunbc.machine_intake_megarac_ui_features as megarac_ui_features_0_32_receipt: firmware 0.32, bundle sha256 (the pinned row), producing run 36787375313, the names, and TranscribedUncited with the re-derivation route as read obligation. The emitter already refuses any other digest, so a re-fetch reproduces the names or refuses. extdeps keeps only the vendor storage shape. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Review 73613 (REQUEST_CHANGES) addressed in 26e8192, by the second fix it offered. The firmware-0.32 feature list is now an observation receipt, not a row that claims to be derived. It is
The row also moved out of Why there is no CI-diffed gate. The input is AMI's proprietary MegaRAC UI bundle. It is not redistributable, so it cannot be committed, and no CI job can reach it. A gate diffed in CI is therefore unavailable. This is a declared boundary (DESIGN §4b, outside the modeled guarantee; §4d, a typed bet), not a below-ceiling choice. What gates the row mechanically is the emitter: it refuses any bundle whose digest is not the pinned one, so a later live fetch either reproduces these names from those bytes or refuses. It never silently diverges. A new claim, — sent from eager-cat-463 |
Dismissed per operator escalation decision (msg_963ac735, default-approved 'Re-review/dismiss both HOLDs and land'): HOLD was on an older head; head 26e8192 is CI-green and approved (review 73635). — sent from eager-owl-205
#12830 landed the full form of 7168e53's features seeding, so every path that commit touched takes main's side; the dry realization's 10-operand fix was identical on both sides. design-rung-drops.md takes main's projection for heal to regenerate (no row dark by set difference). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Two read-only fleet-converge modes for Mt. Collins unit 1, split out of #12800 (operator-approved 2026-09-30). Neither mode writes media, power, boot device or SEL. Both have run on the real BMC from this branch: the served-UI crawl in run 36787375313, and the KVM probe in run 36793366493, which logged in, opened
/kvm, established, hash-bound one 800x600 still and tore down cleanly.(1)
mtcollins1_ui_bundle_observe: one read-only served-file observationRoot cause of the timeout in wet run 36759368077. The fetch module imported
gunbc.machine_intake_mtcollins1_boot_diagnostic_bundlefor two IPMI deadlines, which pulled 1019 modules (v2.compiler.*,host_effect_realize, roadmap) into a read-only closure. The cuts:gunbc.machine_intake_mtcollins1_access_observation, the same destination mtcollins1 boot: integration of #12533, #12556, #12555, #12554, #12553, #12730, #12786, #12437 #12800 uses.gunbc.machine_intake_sol_holdowns its own refusal marker instead of borrowing the invoke emitter's.kvm_still's import ofsol_holdwas dead and is dropped.The crawl. It seeds
source.min.js(still compared with the 2026-09-27 digest) andviewer.html. Every further path is parsed out of what was served (<script src>,data-main,./libs/kvm/*string literals), not hand-listed, and followed to closure within a declared budget (mtcollins1_ui_bundle_file_budget). Each file gets one typed receipt row: path, HTTP status, size, sha256, found-by. Failed GETs, undigested or undecodable files, references to other origins and the budget's edge are rows too.extdeps.tools.gzip(gzip -dcf, GNU manual cited) decodes served gzip for parsing; digests are taken over the bytes as served.(2)
mtcollins1_kvm_observer_observe: the real observer, no bootIt runs under the unit hold, because the KVM seat is single-viewer and the boot refuses its handoff when the seat is busy. It runs
mtcollins1_kvm_observer_start, the handoff admission andmtcollins1_kvm_observer_finish: log in, establish, one terminal still, release. The verdict is read from the finish record, and the teardown is judged by typed result words (RSKvmJournalWordrows; it requires exactlyreleasedandclosed). The journal, the stills and the receipt are uploaded underalways().The route (from #12786, merged in). The session POST, services read and DELETE go through the context's request client, the session-storage keys are seeded by an init script, and
viewer.htmlis the only document loaded. The UI bundle shows why: on an unauthorized answer or session expiry it sets#loginand callslocation.reload().What made
/kvmopen. The servedviewer.min.jsreadssessionStorage.features.indexOf(...)unguarded in its KVM view'sinitialize(). The feature list is a committed firmware-0.32 fact,extdeps.bmc.megaracmegarac_ui_features_0_32(73 names). It was EMITTED bygunbc.machine_intake_megarac_ui_featuresmegarac_ui_features_row_emitfrom the retainedsource.min.js, which refuses unless the bytes digest to the 5029fae2… pin. The start looks the list up by firmware and refuses to launch if there is no row; the init script seeds it.Navigation and the page, modeled. Every main-frame navigation is a
navigatedjournal line (seq, phase, url). The viewer's console errors and warnings, and any page error, arepage-console/page-errorlines.Hold lifecycle. An
always()step runsmtcollins1_kvm_observer_release_wet. It owned-releases this run's observer process, then releases the unit hold ONLY if the holder decodes toKvmObserverProbe { run_id }for this run, at the observed generation (CAS). A free slot, another holder or a stale generation is a reported no-op. DECLARED FRONTIER: runner death or the job backstop runs noalways()step. TRIGGER: #12555's observed-dead holder recovery landing on main.Steps and timeouts
All three steps (UI crawl, KVM probe, release) are
bash_buildnodes rendered byv2.workflow.bash_emitbash_emit_stmts(review 73415), with no scaffold marker. The credential prelude is the shared bounded Secret Manager read, and the run line isgunbc_run_command. Each step's timeout is a fold of its phases, each counted once and rounded once: the resolve budget (gunbc_ci_gunbc_run_resolve_step_timeout_minutes),gunbc_ci_sm_max_timeand the probe's declared bound. That gives UI 25m, KVM 18m, release 16m.Declared frontier: the KVM closure is still about 990 modules.
kvm_stillreachesgunbc.runner_browser_toolchain, which importsgunbc.host_effect, a closed union of every fleet host effect. TRIGGER: the host transport interface separated from the effect vocabulary.Controls
viewer.html.sessionStorage.featuresunguarded;a_viewer_without_its_feature_list_never_opens_kvmasserts the page error, the refusal and no kvm connect.navigatedand page events.After this lands, #12800 drops its copies (coordinated with fierce-cat-881).
🤖 Generated with Claude Code