Skip to content

Fleet runner deployment process: idempotent host-self-selecting fleet-converge emit + swap/no-oomd memory model (8G-RAM standalone contract) + per-host systemd-timer upsert. COORDINATE with cool-swift-127 (srv1 deploy script); full brief from snappy-tern-720. - #6096

Closed
briansrls wants to merge 24 commits into
mainfrom
session/merry-carp-256-fleet-runner-deploy

Conversation

@briansrls

Copy link
Copy Markdown
Contributor

Auto-opened by session-dashboard for session merry-carp-256.
Pushing to session/merry-carp-256-fleet-runner-deploy advances this PR.

Worker attestation

Before flipping this PR to ready for review, confirm each item:

  • Title describes the change (not the session id or branch).
  • PR body summarises what and why (replace the TODO below).
  • Tests run: name the command (e.g. npm test, cargo test) and the result.
  • If this closes a work item, the body contains a Closes #N directive.
  • No commits on this branch are surprises (no fork/cherry-pick I did not make).
  • No secrets / credentials / large binaries staged.

Summary

TODO: replace this paragraph with one or two sentences naming the change and its motivation. Reviewers read this first.

Test plan

  • TODO: list the commands that ran (or "no tests changed; relied on CI") and the outcome.

Brian Searls and others added 7 commits July 1, 2026 16:00
… contract.

Regenerate per-slot memory knobs to stopgap-validated burst values (24G max, 14G high, 8G swap) and update ci_budget_tree witness so floor spawn-width stays on the 8G cgroup cap while RunnerSlotMemoryContract carries the higher burst ceiling.

Co-authored-by: Cursor <cursoragent@cursor.com>
@gunbai-bot
gunbai-bot Bot force-pushed the session/merry-carp-256-fleet-runner-deploy branch from 05279f5 to b8bed9f Compare July 1, 2026 17:13
@gunbai-bot
gunbai-bot Bot marked this pull request as ready for review July 1, 2026 17:13
Brian Searls and others added 4 commits July 1, 2026 17:18
Derive converge apply mode from BmcOnboarding lifecycle via runtime map lookup (srv3 fresh_standup), emit absolute operator paths with User= in the timer unit, and replace slot×swap aggregate with RAM-overcommit-vs-host-swap backing check.

Co-authored-by: Cursor <cursoragent@cursor.com>
@gunbai-bot

gunbai-bot Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Addressed both blocking findings in c8d89da:

1. Systemd $HOME in timer ExecStart (valid) — gunbc_host_ctrl_checkout_path / gunbc_host_gunbc_root_path now derive from gunbc_fleet_operator_unix_home (/home/briansrls), so emitted units get an absolute ExecStart plus User=briansrls. Regenerated .github/fleet-converge-timer-upsert.sh and .github/fleet-converge.sh; witness fleet_converge_timer_emit_holds asserts no $HOME in the service unit.

2. runner_swap_aggregate_demand slot×swap multiply (valid) — per-slot MemorySwapMax is a cgroup cap on a shared host swap pool, not an additive fleet demand. Replaced with runner_ram_overcommit_per_host + runner_swap_backs_ram_overcommit: swap only backs (slot_count × memory_max) − slice_cap overcommit, compared against gunbc_host_swap_available. Witness updated accordingly.

Also fixed CI compile (fabric_runner_slice_cap_per_host Measure/Branch unify) and lifecycle-derived apply mode runtime lookup (map_get on host id — pattern-match on HostIdentity data constants was compile-time only, so srv3 was incorrectly quiescent).

— sent from merry-carp-256

Brian Searls and others added 2 commits July 1, 2026 17:55
…esses.

runner_deployment_plan gates on fabric_memory_accounting_sound; manifest emits fabric_effective_heavy_job_concurrency. Placement witness checks per-slot caps vs slice/swap pool (burst oversubscription), not aggregate slot×max.

Co-authored-by: Cursor <cursoragent@cursor.com>
@gunbai-bot

gunbai-bot Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Addressed opus APPROVE findings in 461882e + follow-up:

1. fabric_memory_accounting_sound / fabric_effective_heavy_job_concurrency (valid) — runner_deployment_plan() now fail-closes on !fabric_memory_accounting_sound before host fold (swap escape: per-slot memory_swap_max ≤ host_swap when Σ contracts exceed Σ allocatable). fabric_effective_heavy_job_concurrency is emitted on the runner-deploy manifest (runner-deploy fabric_effective_heavy_job_concurrency=…) — non-test consumer. Witnesses updated.

2. bmc_phase_has_running_os dead helper (valid) — removed; apply mode is direct BmcOnboardingPhase → ConvergeApplyMode match in gunbc_apply_mode_for_host (no predicate pair).

3. Timer emit string-literal shell (advisory, acknowledged) — added fleet_converge_timer_emit_disposition scaffold marker naming project_timer_upsert_to_doc dissolution target; shared bash-emit-capability marker with fleet_converge_emit tracked for clever-raven orchestration follow-up.

— sent from merry-carp-256

The predicate gates burst overcommit on per-slot swap_max fitting the shared host pool, not aggregate swap soundness — rename matches operator intent and avoids §5 overclaim.

Co-authored-by: Cursor <cursoragent@cursor.com>
@gunbai-bot

gunbai-bot Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

fabric_memory_accounting_sound overclaim (valid, fixed in latest push) — Renamed to fabric_memory_accounting_admissible. The OR-branch admits burst overcommit when memory_swap_max ≤ host_swap (any single slot may page up to its cgroup cap against the shared pool), not when aggregate RAM overcommit is swap-backed — witness witness_ram_overcommit_exceeds_host_swap_backing documents that aggregate gap explicitly. Basis string updated to say ADMISSIBLE vs aggregate-sound. runner_deployment_plan error message now says "inadmissible".

Did not narrow to slice_cap + host_swap ≥ demand — that would reject the operator's intentional burst oversubscription model; rename is the honest fix.

— sent from merry-carp-256

@gunbai-bot

gunbai-bot Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

composer-2.5 APPROVE (no findings) — Verified against 04b058fed: no action required. Findings list is empty; burst contract, fabric_memory_accounting_admissible gate, lifecycle-derived apply mode, timer-upsert emit + drift witnesses, and generated shell artifacts all match the described scope.

— sent from merry-carp-256

Brian Searls and others added 2 commits July 1, 2026 18:30
fleet_host_lifecycle_phase_map_derived folds fleet_host_onboarding_lifecycle (single Scaffold authority). operator_fleet_unix_user/home live in fleet_intent_network; host_converge aliases for emit paths.

Co-authored-by: Cursor <cursoragent@cursor.com>
@gunbai-bot

gunbai-bot Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

opus APPROVE non-blocking findings (04b058f → latest):

1. Lifecycle list/map dual authority (valid, fixed) — Dropped parallel fleet_host_lifecycle_phase_map data. Lookup now goes through fleet_host_lifecycle_phase_map_derived(rows: fleet_host_onboarding_lifecycle) (fold + map_insert); fleet_host_onboarding_lifecycle remains the only Scaffold-marked authority.

2. fabric_memory_accounting_admissible weak discrimination (valid, already addressed) — Intentional operator burst model: per-slot memory_swap_max ≤ host_swap admits aggregate RAM overcommit; basis string + witness_ram_overcommit_exceeds_host_swap_backing document the gap. Renamed from _sound to _admissible in prior commit. Revisit when shared-pool contention model lands (dissolve-on: product.compute_fabric).

3. Hardcoded briansrls operator identity (valid, partially fixed) — Lifted to gunbc.fleet_intent_network as operator_fleet_unix_user / operator_fleet_unix_home (Scaffold disposition). host_converge aliases for emit; generated artifacts unchanged (/home/briansrls). Full per-operator extdeps row is dissolve-on follow-up.

4. Timer emit string-shell (truncated in relay; advisory) — Already carries fleet_converge_timer_emit_disposition scaffold binding project_timer_upsert_to_doc; shared bash-emit dissolution tracked with fleet_converge_emit for clever-raven orchestration.

— sent from merry-carp-256

fleet_converge_shell_emit_dissolution_trigger (Slice 2 / #6092) is single authority bound by timer Scaffold; OnBootSec derives from gunbc_fleet_converge_timer_on_boot_delay like OnUnitActiveSec.

Co-authored-by: Cursor <cursoragent@cursor.com>
@gunbai-bot

gunbai-bot Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

opus APPROVE non-blocking findings (addressed in latest push):

1. Timer emit Scaffold bind: missing bash-emit dissolution trigger (valid, fixed) — Added fleet_converge_shell_emit_dissolution_trigger on fleet_converge_emit.dag (Slice 2 / #6092 / shell_emission_model). Timer Scaffold now binds to that trigger (not project_timer_upsert_to_doc alone); fleet_converge_timer_emit_shell_authority aliases it so both walls share one dissolve-on.

2. OnBootSec=2min literal asymmetry (valid, fixed) — gunbc_fleet_converge_timer_on_boot_delay (120s) in host_converge; timer emit derives OnBootSec= via second_count like OnUnitActiveSec. Regenerated .github/fleet-converge-timer-upsert.sh; witness updated.

— sent from merry-carp-256

Tag bmc_phase_is_pre_os_installed Terminal (canonical ladder partition); import host_fixed_overhead_bytes from fleet_host_budget only; update runner-count disposition to operator-declared width 10.

Co-authored-by: Cursor <cursoragent@cursor.com>
@gunbai-bot

gunbai-bot Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

composer-2.5 REQUEST_CHANGES (all three findings fixed in latest push):

1. bmc_phase_is_pre_os_installed missing disposition (valid) — Added bmc_phase_is_pre_os_installed_disposition: Terminal documenting exhaustive closed-enum partition (pre-OsInstalled vs installed+); mirrors gunbc_apply_mode_for_host FreshStandup boundary.

2. host_fixed_overhead_bytes dual authority (valid) — Removed duplicate from ci_runner_placement.dag; imports sole copy from fleet_host_budget (witnesses already used that path).

3. gunbc_ci_runner_count_from_budget_disposition lies about derivation (valid) — Terminal reason updated: runner_count is operator-declared gunbc_declared_runner_slot_count (10), not runner_pool / memory_max.

— sent from merry-carp-256

Brian Searls and others added 2 commits July 1, 2026 18:51
…get.

Self-binding Scaffold read as the decl claiming its own authority; bind now
points at product.compute_fabric.connect per the basis dissolve-on string.

Co-authored-by: Cursor <cursoragent@cursor.com>
@gunbai-bot

gunbai-bot Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

@claude-opus-4-7 — disposition nit (non-blocking)

Valid: gunbc_runner_slot_memory_contract_disposition self-bound to gunbc_runner_slot_memory_contract, which read as transient scaffold claiming its own authority.

Fixed in 778955179: bind now points at product.compute_fabric.connect — the dissolve target named in gunbc_runner_standalone_memory_contract_basis (dissolve-on: product.compute_fabric connect() memory axis). The row stays Scaffold { dissolves_to: SingleAuthority } until that axis derives per-workload contracts; it no longer self-references.

— sent from merry-carp-256

@gunbai-bot

gunbai-bot Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

@claude-opus-4-7 APPROVE (no blocking findings) — Verified against 778955179:

  • Unit modeling: RunnerSlotMemoryContract, gunbc_host_swap_available, gunbc_ci_heavy_job_working_set, gunbc_runner_slice_cap are ByteSize; gunbc_fleet_converge_timer_on_boot_delay is Second. No bare-scalar unit fields.
  • Hand-shell gate: .github/fleet-converge-timer-upsert.sh emitted from fleet_converge_timer_emit.dag with GENERATED … DO NOT HAND-EDIT header; scaffold binds fleet_converge_shell_emit_dissolution_trigger (shared De-bash orchestration emit: named-edge registry dispatch #6092 seam with fleet_converge_emit.dag).
  • Operator identity: operator_fleet_unix_user / operator_fleet_unix_home in fleet_intent_network.dag (Scaffold → SingleAuthority); timer unit uses User=briansrls and absolute /home/briansrls/... ExecStart — no $HOME in system scope.
  • Apply mode: gunbc_apply_mode_for_host is lifecycle-derived (BmcOnboardingPhase 4-arm match); srv3 CredentialsRotated → fresh_standup, srv1/srv2 OsInstalled → quiescent reload.
  • Budget tree: session_pool child dropped from host_allocation_tree; fabric_memory_accounting_admissible gates runner_deployment_plan(); witness tests (host_allocation_conservation_test, runner_memory_contract_witness_test) cover the reframe.
  • Disposition nit (prior review): gunbc_runner_slot_memory_contract_disposition now binds product.compute_fabric.connect, not self.

No code changes required for this review.

— sent from merry-carp-256

Per-host burst path now requires memory_swap_max ≤ host_swap and
single-slot memory_max ≤ slice_cap + host_swap (non-tautological).
Add fleet_converge_timer_emit to medium_structure_exception_roster
and emitted_medium_emit_fns; discriminating witness for excess swap.

Co-authored-by: Cursor <cursoragent@cursor.com>
@gunbai-bot

gunbai-bot Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

@composer-2.5 REQUEST_CHANGES — addressed in 2bdb7a7ac

1. fabric_memory_accounting_admissible fail-open (valid) — The OR disjunct memory_swap_max ≤ host_swap was tautological with live literals (both 8GiB) and unrelated to overcommit, so the plan gate could never go red.

Replaced with per-host host_memory_accounting_admissible: aggregate-sound (demand ≤ slice_cap) or host_memory_burst_admissible requiring both memory_swap_max ≤ host_swap and memory_max ≤ slice_cap + host_swap (single-slot peak against shared pool — matches operator burst intent; orthogonal to runner_swap_backs_ram_overcommit which documents simultaneous aggregate demand is not swap-backed).

Added witness_burst_gate_rejects_excess_per_slot_swap (16GiB per-slot swap → gate red). Basis string and PlanUnsound reason updated.

2. Timer emit roster gap (valid) — Added dsl/gunbc/fleet_converge_timer_emit.dag, dsl/test/claim/fleet_converge_timer_emit_test.dag, and fleet_converge_timer_upsert_sh_for / expected_fleet_converge_timer_upsert_sh to medium_structure_exception_roster / emitted_medium_emit_fns alongside the sibling fleet_converge_emit entries.

— sent from merry-carp-256

Brian Searls and others added 3 commits July 1, 2026 19:21
gunbc_deploy_runner_provisioned=false emits deploy_dashboard_srv1 if: false;
when flipped, ANDs main-push guard with vars.GUNBC_DEPLOY_RUNNER_READY.

Co-authored-by: Cursor <cursoragent@cursor.com>
Drop gunbc_deploy_runner_provisioned model flag; ci_deploy_srv1_job_if
always ANDs main-push with the observed repo var (unset skips job, no re-emit flip).

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant