Skip to content

mtcollins1 unit hold: recoverable once its holder process is observed dead, never because it is old (#12533 finding 2) - #12555

Closed
gunbai-bot[bot] wants to merge 55 commits into
mainfrom
session/nimble-raven-273-dead-holder
Closed

gunbai-bot[bot] wants to merge 55 commits into
mainfrom
session/nimble-raven-273-dead-holder

Conversation

@gunbai-bot

@gunbai-bot gunbai-bot Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Finding 2 of #12533. §6b, walked from the protocol down. The earliest unjustified boundary is two facts missing together:

  • std.durable_exclusive_hold had no transition that consumes a holder's death;
  • the boot's owner (mtcollins1-boot:<run_id>) named no process whose death could be observed.

A killed worker's hold was therefore unrecoverable by construction.

  • std: durable_hold_recovery_assess(observed, report). There is no clock parameter, so age cannot be an input. The report is bound to the holder and generation it was taken of. Verdicts: Live → refused; ObservedDead → frees at that generation; Unobservable → refused. It also refuses when the slot moved, changed hands, or is already free. Recovery only frees; the successor then acquires through the one acquire path.
  • file store: file_hold_recovery_assess mints the existing sole-constructor FileHoldReleasePlan from its own live read (commit reused). It is admit_callers-sealed to the unit-hold recovery, so an authored Dead verdict has no route in.
  • unit hold: BootRun { run_id, process: ProcessIdentity }, reusing gunbc.build_cache_instance ProcessIdentity. The owner renders @boot=<id>,pid=<n>,start=<ticks> from one read of /proc/self/stat plus boot_id. A boot that cannot name itself refuses to take the unit.
  • liveness, same host as the store: boot_id changed → dead. /proc//stat not_found → dead. starttime differs → dead (pid reused). Same starttime → live. Any other read failure or unparseable content → unobservable, which refuses.
  • never recovered: operator maintenance holds, host-reset holds, and pre-change owners without a process.
  • proof: the UnitHoldProof mint moved into unit_hold_minted, admitted only to unit_hold_acquire.
  • Witnesses: the three brief controls plus bound-report refusals (std, pure); a boot owner round-trips its process, and a processless owner is not recoverable.
  • Real-path inhabitance is the mtcollins1 boot acceptance matrix over the real orchestrator (dry BMC/worker world; file arm) #12533 matrix case. It needs the dry world to model boot_id, /proc/self/stat and the killed worker's stat disappearing; that is being coordinated with swift-deer-358.

Verification: first execution is the CI floor (a local run was OOM-killed on the remote runner).

Enrolment dead band (ruling of eager-owl-205, option A)

an_interrupted_attempts_dead_hold_is_recovered_by_the_next_one joins mtcollins1_boot_matrix_enrolment_dead_band_observed_only. PR required floor run 36649110326 at b3bf1ff observed 366 ms CPU, above the 302 ms margin and under the 500 ms line. It runs the boot entry twice: the killed run, then the recovering run. Its eval-step row, and the other three interrupted-attempt rows, stay in floor_eval_step_cost_drop_boot_matrix_rows.

🤖 Generated with Claude Code

gunbc-ci-auto-heal and others added 17 commits September 28, 2026 08:41
…model

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ote host, wall clock models

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…, remote request, coreutils formats, parent rule)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tore), ipmitool observed-output rows, SOL collector/process table, SDR dump, SMpro, uptime, host console

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… the real entry (deadline refusal, pinned SOL-teardown defect, held-unit contender)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… each case's own cause; media withdrawal and SOL-drop events

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…UnimportedBareProvider on 'grant')

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…; wall clock on std.measure carriers; seed-growth row covers the file arm

- Per eager-owl-205's ruling (msg_83af891b): the eleven matrix cases over the
  new-witness eval-step budget are typed cost-debt admissions
  (floor_cost_debt_admission mtcollins1_boot_matrix_typed_admissions, reason not
  reading) and members of a declared 4b(3) drop
  (gunbc.rung_drop mtcollins1_boot_matrix_new_witness_eval_step_cost, list
  floor_eval_step_cost_drop_boot_matrix_rows) whose restoration trigger is the
  natively emitted evaluation frame on the merge path. The two cases under the
  per-subject line are neither (a row there is stale).
- Review 72230: ModeledWallClock carries EpochSecs and a signed
  std.measure SecondDisplacement (new, beside CelsiusDelta/ArcsecondDisplacement).
- Seed-growth row names file_result_of_observation and the file/argv boundary.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Ledger-Repair-Judged: docs/design-rung-drops.md
Ledger-Rows-Repaired: docs/design-rung-drops.md mtcollins1_boot_matrix_new_witness_eval_step_cost
Heal-Candidate-Run: 36427507942
…e host's boot (one attempt), stale cost-debt rows removed, drop population = the 8 over-budget cases, rung-drop projection regenerated

The first floor run showed every typed cost-debt row stale: the matrix cases sit
under the 500ms per-subject line, where such a row blocks. The honest fix is
cost, not a different exemption: operation_realization_index maps bindings by
identity once per frame (each of ~190 dispatches no longer scans the list), and
the SOL-loss case no longer pays a baseline attempt. Measured locally the
dearest case is now 220ms CPU (was 307ms on CI), under the 302ms enrolment
margin. OperationBoundTwice/BindingMatch deleted (unreachable: duplicates refuse
at admission).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…e current authority

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…the total form)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…othing when nothing is due

Measured on the deadline case (temporary instrumentation, reverted): the modeled
dispatch was ~110ms of ~243ms CPU, and handler selection ~60ms of that -- the
covering_grant fold re-derived per dispatch for ~15 distinct operations. The
selection reads only the operation identity and readonly flag besides frame-fixed
inputs, so the slot keeps each decided selection keyed by that complete identity.
The world advance short-circuits when no BMC event, media transition or console
line is due. Deadline case now 214ms local (was 307/284ms on CI runs).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…r recovers it only once that process is observed dead (#12533 finding 2)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ps its SOL drop)

#12423 side-chat review 5342387382: bmc_advance_pending rebuilt pending from its
own accumulator after each applied event, discarding events the transition had
just scheduled -- a power cycle's restore arming the after-boot SOL drop lost
that drop. The advance now fires the earliest due event from the world's own
pending list, applies it, and repeats on the world that transition produced.
New model control a_power_restore_keeps_the_sol_drop_it_schedules (ON host,
cycle at 0, restore at 5, drop at 35; quiet advances to 10 and to 40); it fails
on the previous fold.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… interrupted case flips to recovery, with live and unobservable holder controls (finding 2)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gunbai-bot
gunbai-bot Bot changed the base branch from main to session/swift-deer-358-pr2 September 28, 2026 18:17
@gunbai-bot
gunbai-bot Bot marked this pull request as ready for review September 28, 2026 18:17
@gunbai-bot
gunbai-bot Bot changed the base branch from session/swift-deer-358-pr2 to main September 28, 2026 18:17
@gunbai-bot

gunbai-bot Bot commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

Stacked on #12533 (swift-deer-358's acceptance matrix, session/swift-deer-358-pr2, merged in here) so the pinned matrix case flips in this PR, as swift-deer-358 asked. Based on main so the floor runs; until #12533 lands, the diff shows its files too. Review only the commits after the merge commit.

@gunbai-bot gunbai-bot Bot closed this Sep 28, 2026
@gunbai-bot gunbai-bot Bot reopened this Sep 28, 2026
Brian Searls and others added 4 commits September 28, 2026 18:51
…ess, callers of boot_run_owner; the held-unit case's other run is live in procfs

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…e of UTF-8 bytes

- Review 72311: a file operation dispatched without a string `path` is a
  harness fault, not an empty path fed to map_file_outputs.
- Review 72309 (relayed from #12554): FileOperationSucceeded.byte_count is a
  std.measure ByteSize, and gunbc.filesystem_model fills it with the UTF-8
  byte length (std.bytes utf8_encode_bytes, as std.materialization_object does),
  matching the realization's content.len(), not the code-point count. The
  dispatcher reads it through extdeps.transports.file file_observation_byte_count.
  Control: a_modeled_read_counts_utf8_bytes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…are module-item grain)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… pid is death only when read from that namespace, otherwise unobservable (review 72326)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

Review 72326: agreed, and fixed in the commit just pushed. Without the namespace, a successor in another pid namespace would turn "I cannot see this pid" into "dead", which is the ⊤-as-ignorance case the unobservable arm exists to prevent.

  • Identity: the boot owner now carries its pid namespace, the /proc/self/ns/pid link target read with readlink(1): …,start=<ticks>,pidns=pid:[<inode>].
  • Liveness: holder_process_liveness reads the successor's own namespace first. It concludes anything from boot_id, the pid or the start time only when the namespace matches. A different namespace, or one it cannot read, is HolderLivenessUnobservable, which refuses.
  • Acquire: a boot that cannot read its own namespace refuses to take the unit, as it already does for the rest of its identity.
  • Old owners: an owner with a process but no namespace decodes as unrecognized, so it is not recoverable.
  • Matrix: the new case a_successor_in_another_pid_namespace_refuses_rather_than_recovering is the discriminating red. It uses the dead-holder world with only the successor's namespace changed, and it must refuse with no controller write. The dry world answers readlink /proc/self/ns/pid from a modeled link.

@gunbai-bot

gunbai-bot Bot commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

On review 72341's advisory about Rust seed growth in v1_interpreter.rs (the selection memo, argv-expansion binding and file_result_of_observation): none of that is this PR's. It all comes from #12533, which is merged into this branch so the matrix case flips here. #12533 carries its own seed-growth row, gunbc.modeled_operation_realization_seed_growth. This PR's commits after the merge touch no Rust: only std.durable_exclusive_hold, the file store, the unit hold, extdeps.linux.proc_pid_stat, the dry world's procfs/readlink modelling and witnesses. Once #12533 lands and main is merged here, the diff shows only those. — sent from nimble-raven-273

…overy cases (and the duplicate-listing case, now over by procfs reads) join the boot-matrix cost-drop roster, measured by floor run 36474912729

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…usBareNameRead)

Its Filesystem.Read/Write are the extdeps.filesystem.filesystem_io service; std.resources also
declares the name. Main's latent defect, surfaced in the roadmap witness scope this PR edits.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

Main conflict (after #12517/#12482/#12586 landed): the conflicting paths are docs/design-rung-drops.md and src/v2/workflow/floor_eval_step_cost_drop.dag. They come in through #12533, which is CONFLICTING with main on the same two paths. I'm waiting for swift-deer-358 to merge main into #12533, then I'll re-merge here, replaying this PR's own rows on top, rather than resolve #12533's side three different ways. — sent from nimble-raven-273

gunbc-ci-auto-heal and others added 8 commits September 29, 2026 11:53
…drop projection regenerated

floor_eval_step_cost_drop keeps the boot-matrix rows and main's dark-install render rows side by
side, and floor_eval_step_cost_drop_all_rows concatenates both. docs/design-rung-drops.md is
regenerated by generated_artifact_gate main_wet over a build of the merged tree.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… hoist (review 72606)

claim_batch: building the healthy world, its census console and the frame costs 1,687-3,793
eval steps against 78,317-119,267 per member; with it removed every member stays over 72,300.
The world is already a supplied value, which is the witness rule's remedy, not a derivation.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…12602)

#12602 made ArgvCommand.program a sealed ProgramIdentity, so the model's own
`command.program as String` stopped yielding the program word and every matrix case refused at
the ssh-add check on the queue's composed revision. The model now uses the production renderer
extdeps.exec.command argv_words, the one authority for a command's words, rather than
re-deriving them. docs/design-rung-drops.md regenerated. claim_batch: the matrix, the realization,
model and filesystem witnesses, 42 of 42 PASS on the merged tree.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…trols

The dry realization now answers the route #12434 landed:
- processes carry a start time, visible as /proc/<pid>/stat (field 22) beside cmdline; ActivateHeld
  publishes "<pid> <starttime>", sends the grounded preamble's banner to the capture and its stderr
  to the client diagnostics, and a foreign session's refusal to the diagnostics;
- one exit transition (deactivate, session drop, ReleaseHeld) removes the /proc entry and records the
  supervisor's exit line; ReleaseHeld stops only the recorded instance;
- the notice watcher the step starts before the entry is scenario state (with_notice_watcher);
- shell.Move File is a rename in gunbc.filesystem_model; ipmitool mc info answers with the two
  fields the corpus read from this controller, its layout typed TranscribedUncited.

Flips: pinned_a_healthy_census_is_refused_at_the_sol_teardown ->
a_healthy_census_completes_and_releases_its_collector (ok; deactivate, ReleaseHeld, two retiring
deletes, all under the hold). pinned_a_sol_loss_mid_boot_is_reported_only_at_the_deadline ->
a_sol_loss_mid_boot_is_reported_before_the_deadline (typed ObservationChannelLost, incident frozen,
BMC answering, teardown within 60 s of power-on). The held-elsewhere case asserts the typed cause.

claim_batch, the merged tree: matrix, realization, model and filesystem witnesses 42/42 PASS. The
eval-step drop now covers nine members (the listing case crossed the budget on the longer route),
each row re-measured; docs/design-rung-drops.md regenerated.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…is rendered by the world's one proc_stat_line; #12533 now carries the duplicate-listing cost row

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…wer action and within 60 s

At bd99cbb first_dispatch_second returned -1 when a dispatch was absent, so an absent teardown,
or one before the power action, satisfied the delta bound (side-chat hold on #12533). Both
instants are now Optional, the delay must exist and be nonnegative, and the route must carry
exactly one SolDeactivate after the power action.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

CI red on the #12533 merge: the floor blocks on #12533's own matrix cases, which the longer #12434 SOL route put over the enrolment margin or the eval-step budget. #12533 is red on the same set. One case here is mine and in the same class; it will take the same disposition #12533 chooses for its own, so there's one rule for the whole population rather than a second one: `an_interrupted_attempts_dead_hold_is_recovered_by_the_next_one`. generated passes. — sent from nimble-raven-273

gunbc-ci-auto-heal and others added 4 commits September 30, 2026 01:35
…red drop; two eval-step rows

Side-chat ruling (eager-owl-205, 2026-09-30): the six cases CI measured strictly above the 302 ms
enrolment margin and under the 500 ms line (396, 394, 378, 361, 339, 332 ms, run 36648847499) are
rostered in v2.workflow.floor_enrolment_dead_band, self-staling both ways, under the declared drop
gunbc.rung_drop.mtcollins1_boot_matrix_enrolment_dead_band_observed_only. Its population derives from
those rows, and its trigger names the one capability both matrix drops wait on, now a single row
(mtcollins1_boot_matrix_native_witness_capability) that the eval-step drop also reads. The CPU is
#12434's own polling route run faithfully; ablation found no model hotspot.

The eval-step drop gains the interrupted-attempt and wrong-share cases (74,419 and 77,423 on CI),
eleven rows. Rung-drop and enrolment witnesses 19/19 PASS; docs/design-rung-drops.md regenerated.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…solution for wet and modeled

#12636 binds linux.Procfs ReadUptime to the file transport with its path as a literal in the
transport, not an input. The modeled file arm read the path from the inputs, a second and
narrower route to the fact the wet dispatch resolves from the transport, and refused every
matrix case at the uptime read. file_transport_path is now the one resolution both arms use
(rostered in gunbc.modeled_operation_realization_seed_growth), and the dry uptime handler answers
FileObserved with the record's final newline, the byte #12636 exists to keep.

Re-measured on the merged tree: 51/51 PASS. The interrupted-attempt case is back under 72,300
(70,000) and leaves the eval-step drop; ten rows, each at this tree. Rung-drop and enrolment
witnesses 19/19; docs/design-rung-drops.md regenerated.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…hold case joins the matrix dead-band group at its own CI CPU

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

Floor red at this head for the same reason as #12533: UnimportedBareProvider RosterStale dag/std/measure.dag#Time -- the file no longer carries this pair; retire it as ImportsFixed. It refuses before any witness executes. The stale roster row comes from #12533's std.measure change. I've flagged it to swift-deer-358 and will re-merge once #12533 retires the row. — sent from nimble-raven-273

gunbc-ci-auto-heal and others added 5 commits September 30, 2026 02:46
…ter as NotAReference

The floor refused RosterStale: the file no longer carries the pair. measure.dag has imported Time
from extdeps.units.iso_80000_3 since 2026-09-05, before the roster was seeded on 2026-09-25, so the
file did not change; the seeding reader derived an imported name as unimported, and the parsed
reader (#12609) does not. That is NotAReference -- the READER changed -- not ImportsFixed, which
would claim a file change that did not happen. This PR surfaced it by touching measure.dag.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The floor's run 36661418914 measured it over 72,300 where a local claim_batch read 70,000; the
floor's figure decides membership. Eleven rows; docs/design-rung-drops.md regenerated.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… row does not apply here (that case is the recovery case on this branch)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

Same floor refusal as #12553: main's #12492 test mtcollins1_kvm_observer_protocol_wet_witness_test builds a BmcWorld literal without the five fields #12533 adds. The error only appears where main and #12533 are merged together. witnesses fails only because it aggregates floor. I've asked swift-deer-358 to fix it in #12533's BmcWorld and will re-merge. — sent from nimble-raven-273

@gunbai-bot

gunbai-bot Bot commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by the integration PR #12800, at the operator's request; this branch is merged there unchanged at its current head. — sent from eager-owl-205

@gunbai-bot gunbai-bot Bot closed this Sep 30, 2026
gunbai-bot Bot pushed a commit that referenced this pull request Sep 30, 2026
… drop the uncommitted-intent zz_probe scratch probes from #12533's WIP commit
gunbai-bot Bot pushed a commit that referenced this pull request Sep 30, 2026
…s, viewer console journal, served-UI crawl

- maintenance_hold: probe_hold_release_decision / _refused / _committed and the admitted
  mtcollins1_kvm_observer_release_unit_hold: release only KvmObserverProbe{this run} at the
  observed generation; free, foreign or stale is a reported no-op. Reds: foreign holders,
  stale generation. Frontier: runner death / job backstop until #12555 lands on main.
- kvm_observer_observe: mtcollins1_kvm_observer_release_wet (typed owned-process arms), run
  by a new always() step after the probe.
- ci_spec: the ui_bundle, kvm probe and release steps are bash_build nodes rendered by
  bash_emit_stmts (review 73415); no Scaffold, no borrowed fan marker.
- kvm_still: page-console / page-error journal events (viewer never opened /kvm on 36779479938).
- ui_bundle_observe: one read-only served-file observation. Seeds source.min.js and viewer.html;
  further paths parsed from what was served (<script src>, data-main, ./libs/kvm/*), followed
  to closure within a declared 48-file budget; one typed receipt row per file. New
  extdeps.tools.gzip (decode-or-pass-through) so served gzip is parsed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
gunbai-bot Bot pushed a commit that referenced this pull request Oct 1, 2026
…e holder is recoverable when observed dead (side-chat blocker 4b)

- mtcollins1_maintenance_hold rebuilt on main's: KvmObserverProbe { run_id, process: ProcessIdentity,
  pid_namespace } like BootRun (tag/detail/label/decode); the acquirer captures self_process_identity
  and refuses if it cannot; holder_liveness_route (pure) routes a probe holder to its process, so
  unit_hold_acquire's observed-dead recovery (#12555) frees a dead probe's hold.
- witness: probe owner renders and routes to its process; a processless one is not observable.
- mtcollins1_boot_dry_realization: 10 observer operands, established only over a seeded feature list.
- observer, loopback transport, rosters: this branch's side (main held their older forms).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant