Repository navigation
CI: move required runners to Ubicloud standard (ubicloud-standard-16-arm) - #6107
Merged
Merged
Conversation
…arm) Stabilization move (operator directive): take required CI off the self-hosted arm64 fleet, which is the source of the sccache flakes, OOM-kills, and the spawn-width budget pain. Each Ubicloud job runs on its OWN dedicated VM, so the 10x-oversubscription / admission problem that motivated the whole fleet budget tree dissolves by construction. Modeled, not hand-hacked: - gunbc_ci_runner_spec -> RunsOnExpression "ubicloud-standard-16-arm" (16 vCPU / 48 GB, arm64 to keep the current arch — zero arch-change risk). New single-authority data decl gunbc_ci_ubicloud_runner_label. - ci_runner_seam_witness flipped: asserts the runs-on is the Ubicloud hosted label (was: self-hosted arm64). Green by execution. - ci.yml regenerated (byte-drift-gated) — runs-on on all 3 jobs. Scope: this is the RELIABILITY half. The ~9x floor speedup (spawn_width 1 -> ~9 on a 48 GB VM) needs a separate §3 cap de-conflation: the floor width fold's cap (gunbc_ci_runner_cgroup_memory_cap, 8 GiB) is the load-bearing authority for the self-hosted fleet budget tree (fleet_host_budget, ci_runner_placement, ci_budget_tree + witnesses), so it is genuinely two concepts now (dormant self-hosted cap vs active Ubicloud VM cap). Splitting it is deliberate follow-up, kept out of this stabilization PR so it does not touch the floor scheduler. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
briansrls
added a commit
that referenced
this pull request
Jul 1, 2026
Scope-creep fix per review: the runner_spec_from_offer + ci_runner_seam witness edits are unrelated to boundary-coverage; they live in #6107 (CI -> Ubicloud). Restore both files to main so this PR is boundary-coverage only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
briansrls
added a commit
that referenced
this pull request
Jul 1, 2026
…n/cargo The self-hosted fleet had cargo pre-baked at the isolated $CARGO_HOME/bin; a fresh Ubicloud VM has cargo on PATH (via setup-rust-toolchain + the pinned default toolchain in the isolated RUSTUP_HOME), not at that hardcoded path. ci_release_build_line -> 'cargo build ...'; propagates to both jobs and all eagain retries. ci.yml regenerated; drift + serializer witnesses green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
briansrls
added a commit
that referenced
this pull request
Jul 1, 2026
…dard-16-arm) (#6107)" (#6111) Operator directive: put CI back on srv1/srv2. The ubicloud move bought no wall-clock — the dominant cost is the serial batch-1 dsl_compile_clean_gate (~47min, 1 node / 1 resolve-group, spawn_width-immune), so a dedicated 16-vCPU VM runs the same ~50min as the fleet. No use renting VMs for it; the real levers (affected-set-scoped compile-clean, cross-run resolve memoization, sharding the gate) are tracked on the roadmap (#6110 §2). Full revert of the #6107 bundle (spec + seam witness + $CARGO_HOME cargo path, ci.yml regenerated via main_wet; drift gate + ci_runner_seam_holds PASS by execution). Also heals the pre-existing .gitignore drift (stale srv3 dnsmasq row) — same one-line heal as #6110, merges cleanly either order. Known residual, accepted by operator: fleet sccache flakes / OOM exposure return, and deploy_dashboard_srv1's LocalShell premise ("runner is ON srv1") is only restored when srv1 claims the job — [self-hosted, linux, arm64] can land on srv2 (wrong-host deploy risk; tracked as 2-cd-transport + G1 placement on the roadmap). Co-authored-by: Brian Searls <briansrls@gunb.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
gunbai-bot Bot
pushed a commit
that referenced
this pull request
Jul 11, 2026
Self-hosted fleet has no free slots (operator, 2026-07-11); route this PR's iteration to ubicloud-standard-16-arm so the lens work is not gated on fleet capacity. Same move as the #6107 -> #6111 precedent. gunbc_ci_runner_spec() returns the ubicloud label; the fleet derivation is preserved untouched as gunbc_ci_runner_spec_fleet_derived() and ubicloud_iteration_routing_note carries the revert obligation. Workflows regenerated from the authority via main_wet (drift gate stays coherent): runs-on [ubicloud-standard-16-arm] at ci.yml build_release/floor/emit-determinism and falsifier.yml. Revert = git revert this single commit + regen (or just revert; the regenerated ci.yml/falsifier.yml are included here).
briansrls
pushed a commit
that referenced
this pull request
Jul 12, 2026
… cost/routing work (#6489) * witness-cost-locality lens v0: one typed verdict for the two ambient-coupling axes (receipt-only) One law: witness cost must be denominated in its subject (the DESIGN 5 denomination law). Two detectable projections at module grain: DataBreadth (closure reaches an ambient-read carrier: filesystem_read callers + the decl_facts reflection home) and LayerDepth (closure spans >=2 compiler pipeline stages - the round-trip shape). Verdict algebra + census receipt + precision-frontier disclosures as data; six fixture witnesses incl. the single-stage boundary and a red control (law perturbed to >=1 flips single_stage_closure_stays_local red - receipted). Census 2026-07-11 over the 732-entry roster: Local 483 / LayerDepth 229 / DataBreadth 9 / both 11; derives the operator-ruled-offline enforcement rows and the execution/ exclusions; falsified two naive laws (decl_facts reflection invisible to filesystem_read grep; hermetic fixture rows over-flagged - the declared InputEnvelope is the override authority). LayerDepth is receipt-only until fn-grain call-graph reachability lands (ci_yaml counter-receipt: stage-type imports are not stage execution). Dissolve-on: fn-grain reachability + InputEnvelope declarations wired into floor admission; the witness_exclusion hand-rows and this v0 roster retire together. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add the required construction_justification row (cursor review) WallAfterGrounding dissolving to SingleAuthority: the verdict is validation-tier until fn-grain reachability + InputEnvelope declarations let floor admission consume it, at which point non-Local rows are structurally absent from the per-PR plan (the single authority) and the lens dissolves per its precision-frontier note. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * [ITERATION-ONLY - REVERT BEFORE MERGE] Route CI to ubicloud runners Self-hosted fleet has no free slots (operator, 2026-07-11); route this PR's iteration to ubicloud-standard-16-arm so the lens work is not gated on fleet capacity. Same move as the #6107 -> #6111 precedent. gunbc_ci_runner_spec() returns the ubicloud label; the fleet derivation is preserved untouched as gunbc_ci_runner_spec_fleet_derived() and ubicloud_iteration_routing_note carries the revert obligation. Workflows regenerated from the authority via main_wet (drift gate stays coherent): runs-on [ubicloud-standard-16-arm] at ci.yml build_release/floor/emit-determinism and falsifier.yml. Revert = git revert this single commit + regen (or just revert; the regenerated ci.yml/falsifier.yml are included here). * Closure facts + per-PR admission law: the transitive-carrier case becomes provable, and verdict x envelope decides the lane Extends the v0 lens with the fact half its census note promised: - ModuleImportFact + import_closure: module-grain closure over declared import facts, the |E|-sweeps fixpoint shape sanctioned by v2.lens.affected_set (pass + frontier-stability + doubled-facts fuel, early-stable). Terminates on cyclic FACT rows - the substrate refuses cyclic imports, but the fold must not trust its input. - witness_cost_locality_from_facts: closes over the facts so the transitive carrier (the enforcement_live falsification - the corpus read two hops away) is provable in fixtures, not just prose. - WitnessInputEnvelope + per_pr_admission: the admission matrix (verdict x declared envelope), fail-closed with declaration-wins-when-stricter. DeclaredCorpusInput always routes to the scheduled lane; DeclaredBoundedFixture is the hermetic override (reachability over-approximates; the scheduled falsifier lane is its audit); EnvelopeUndeclared admits only Local - the refused_undeclared_when_envelope_unknown shape from InputEnvelope P0. RouteScheduledLane is a typed routing verdict, never a silent drop. Witnesses (7 new, 13 total, all green by execution; the suite itself stays Local-class: 393ms resolve, 0ms eval, fixture-only): - two-hop transitive carrier couples on DataBreadth; severing the mid hop flips it Local (the discriminating pair) - cyclic facts terminate and still reach the carrier - admission matrix cells incl. the RED control: undeclared + coupled never rides the continuous floor * Resolve the cargo binary at toolchain-pin time; stop baking the fleet's shim path into the build line The build line invoked "$CARGO_HOME/bin/cargo" - a fleet fact fused into the job model. On a runner image whose rustup pre-exists (ubicloud, GitHub-hosted), setup-rust-toolchain never runs rustup-init, so no shim lands under the isolated $CARGO_HOME and the build dies with "No such file or directory" (run 29161556017). The isolated RUSTUP_HOME/CARGO_HOME env still governs whichever shim exists - only the shim's LOCATION is realization-specific. Fix: the pin step (already the "make the isolated toolchain invocable" step, name unchanged so the cost-floor roster keeps resolving; no new run-step so the raw-script count pin holds) resolves CARGO_BIN once - isolated shim if present, else PATH - and refuses loudly when neither exists (fail-closed, DESIGN 5). ci_release_build_line invokes "$CARGO_BIN"; the retry escalations' env-prefix composition is unchanged since it stays one word. Falsifier inherits via the shared prelude. Workflows regenerated via main_wet. witness_rustup_run_plain_not_quoted repinned as witness_rustup_run_block_scalar_not_quoted: the pin step is now multi-line so its run correctly serializes as a block scalar; the property the witness guards (quote-laden content never YAML-escaped, negative control kept) is unchanged. ci_yaml_serializer_keystone_holds green by execution; 18-entry consumer sweep green except 3 witnesses red on unedited baseline too (pre-existing/environmental). * [ITERATION-ONLY - REVERT BEFORE MERGE] Reconcile the runner seam witness with the ubicloud override Cursor catch (review on the folded #6490, restated on #6489): the routing override left ci_runner_seam_holds red - it pinned gunbc_ci_runner_spec() to the fleet derivation and the yaml to [self-hosted, linux, arm64]. Reconciliation, discriminating in both directions during the window: the fleet-derivation witnesses repoint to gunbc_ci_runner_spec_fleet_derived() (the derivation authority stays proven and un-drifted), the live-spec witnesses pin the override label exactly (drift of the override itself is caught), and the workflow==live-spec seam witness is unchanged. ci_runner_seam_iteration_note carries the revert obligation; this commit reverts together with the routing commit (3f203d8). ci_runner_seam_holds green by execution. * Revert the iteration-only ubicloud routing (operator go 2026-07-11: iteration window closed, focus on merge) Restores gunbc_ci_runner_spec() to the fleet derivation as the single runner authority and the seam witnesses to pinning it (reverts 3f203d8 + 36e4cd3 as one motion); workflows regenerated via main_wet back to [self-hosted, linux, arm64]. The CARGO_BIN resolution (a03142d) stays - fleet-neutral portability fix, proven by execution on both runner classes (fleet floors unchanged; ubicloud build green run 29162064482). ci_runner_seam_holds and the serializer keystone green by execution on the reverted head. What the iteration window bought, for the record: the CARGO_BIN portability bug (fixed), the 12.5-min full floor + 13.5GiB-uncapped counterfactual receipt that pinned the fleet wedge on the memory.high throttle (now fixed at the budget edge in #6495), and the surfacing of the #6459 batch-2 panic (fixed in #6493). * Runner offers modeled + one selection authority: labels, memory regime, and budgets project from the selected target Operator asks 2026-07-11: model the runner rows (github, ubicloud; fleet existed) and make the mappings clear - then "stay on ubicloud for this change; self host runners are still contended." extdeps (cited, zero fabrication): - extdeps.cloud.ubicloud: the 10 documented runner shapes (runner-types.md anchor; arm 3GB/vCPU, x64 4GB/vCPU kept as vendor facts; the standard-16-arm observed MemTotal receipt recorded SEPARATELY from the catalog claim, discrepancy noted not reconciled) - extdeps.github.hosted_runners: the Ubuntu family with repository VISIBILITY as a first-class axis (public 4vCPU/16GB vs private 2vCPU/8GB for the same label - folding it away would be a state-space conflation); version-pinned labels, ubuntu-latest alias deliberately not a row gunbc.ci_runner_target (the missing edge the iteration window proved): - CiRunnerTarget = FleetSelfHosted | UbicloudRunner{row} | GithubHostedRunner{row}; selected_ci_runner_target() is THE switch - projections: runs-on spec (fleet stays DERIVED from the fleet offer; cloud rows carry the provider's cited token), memory regime (SlotCarved{desired} with the 15GiB throttle line vs WholeMachine{ram}), RAM-speed budget (fleet -> slot ceiling; cloud -> catalog ram) - ci_workflow, falsifier_workflow, floor budget, falsifier width all read the projections: flipping the selection row moved runs-on AND CARGO_BUILD_JOBS (4 -> 11, from the 48GB row) AND floor width in one regen - the propagation the label-only model could not do - selection = ubicloud-standard-16-arm (operator routing, fleet contended); flip back = FleetSelfHosted + regen, nothing else moves - falsifier now rides the selected shape too: its pinned fleet-slot capacity deficit witness stays parameterized on the FLEET ceiling (the deficit is a slot fact), while on 48GB its envelope fits with 3x headroom - the nightly's dissolve-on path Witnesses: ci_runner_target_witnesses (fleet-derivation identity, ubicloud/github label projections, both regimes, cited arm ratio held across the family, visibility-axis discrimination) + seam witnesses repinned to the selection authority with the fleet derivation kept live for the flip back. * Width-fit witness measures against the machine the plan runs on (the selected target budget), not the fleet slot unconditionally * Regen after folding the runner-offer selection into this branch (ubicloud labels + JOBS=11 + CARGO_BIN compose) * Resolve the cargo binary at toolchain-pin time; stop baking the fleet's shim path into the build line The build line invoked "$CARGO_HOME/bin/cargo" - a fleet fact fused into the job model. On a runner image whose rustup pre-exists (ubicloud, GitHub-hosted), setup-rust-toolchain never runs rustup-init, so no shim lands under the isolated $CARGO_HOME and the build dies with "No such file or directory" (run 29161556017). The isolated RUSTUP_HOME/CARGO_HOME env still governs whichever shim exists - only the shim's LOCATION is realization-specific. Fix: the pin step (already the "make the isolated toolchain invocable" step, name unchanged so the cost-floor roster keeps resolving; no new run-step so the raw-script count pin holds) resolves CARGO_BIN once - isolated shim if present, else PATH - and refuses loudly when neither exists (fail-closed, DESIGN 5). ci_release_build_line invokes "$CARGO_BIN"; the retry escalations' env-prefix composition is unchanged since it stays one word. Falsifier inherits via the shared prelude. Workflows regenerated via main_wet. witness_rustup_run_plain_not_quoted repinned as witness_rustup_run_block_scalar_not_quoted: the pin step is now multi-line so its run correctly serializes as a block scalar; the property the witness guards (quote-laden content never YAML-escaped, negative control kept) is unchanged. ci_yaml_serializer_keystone_holds green by execution; 18-entry consumer sweep green except 3 witnesses red on unedited baseline too (pre-existing/environmental). * Unbreak the post-merge-wave floor: absent declared roots skip loudly, the split-spanning gate universe, and the disposition-marker adjacency pin Three content reds every completed floor now hits, all proven on clean main (the 60m-timeout and batch-2-panic masks are gone, so the content layer behind them is finally visible): 1. anchor_source_root panicked mid-floor on "dag/compiler" (run for PR #6497, cli_run.rs:365) - a DECLARED root from the medium-structure roster that does not exist on disk yet (modeled-before-implemented). The layer walk's own is_dir guard proves absence is a legitimate skip state there, but the anchorer panicked before the guard could run. New non-panicking try_anchor_source_root for declared-root walks: absence skips with a counted [layer-import] line (loud, never silent); CLI-provided roots keep the strict panicking contract. 2. affected_set_universe_gate_processes_match_declared_gates compared the Gate coproduct's 10 arms against gunbc_ci_gates - the ci JOB's slice (7) since the #6472 job split. The witness now unions floor+regen+emit_determinism (the same union its sibling witness_gate_roster_matches_coproduct_arms already used); probe receipt: the three slices partition the 10 arms exactly. 3. witness_floor_disposition_marker_initialized_before_docs_only_branch (landed in today's wave) pinned the stamp DIRECTLY adjacent to the docs-only branch, but the emitted script has a blank line between them - red on clean main from its first run. The pin now matches the emitted adjacency (stamp, blank, _ci_changed) and still proves the ordering it exists for. Receipts: ci_spec_witnesses, all 4 affected_set_universe witnesses, and the layering clean-tree witness green by execution with the fixed binary. Remaining known main reds (predate today, tracked separately): s1_closure_parses_holds (#6459 wrapper-retained diagnostic arm), ci_deploy_witnesses. * GithubHostedRunner projects the hosted scalar form (cursor catch) Ground GithubHostedRunnerCatalogRow.runs_on_label in the actions grammar's RunnerLabel closed vocabulary (extended with the cited ubuntu-24.04-arm / ubuntu-slim spellings; runner_label_string stays the single spelling authority) and route the GithubHostedRunner target through HostedRunner, whose serialization is the scalar runs-on form GitHub uses for hosted labels. A labels-list projection would have emitted the self-hosted label-matching form. Witness repinned to the HostedRunner variant plus a by-execution runner_yaml discrimination (scalar hosted vs flow-list label-routed). Ubicloud stays SelfHosted label routing (provider tokens, proven live 2026-07-11). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ci_deploy witness: repin deploy invoke to the rooted source-root spelling witness_deploy_run_script_invokes_gunbc_wet pinned the unrooted '--source-root dag' while gunbc_ci_deploy_invoke emits the rooted '--source-root "$ROOT/dag"' (witness_layer_source_flags_rooted, #6453) - red on clean main, the third masked layer of the rooted-argv family (#6493 repinned the scheduler-argv pair). Proven by execution: suite false -> true on the repin, conjunct-bisected first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * try_anchor_source_root: seed-retained scaffold receipt (cursor #6489 catch) Hand-Rust expansion now carries the DESIGN §7 shape the same file requires: authority rows (cli_run_source_root_anchor_scaffold sibling of the workspace-root scaffold, bind to the declaration), named dissolve-on (roots walk GENERATED via cli-run-reconcile-defork Chunk F, absence a typed roster-layer diagnostic; or 5-dissolve-patches), checkable receipt anchor + counted loc delta, scaffold witnesses green by execution, and a discriminating unit pair (declared-present -> Some, declared-absent -> None). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * s1_closure floor panic: wrapper-retained diagnostics must inhabit the Diagnostics coproduct body_lower_wrapper_retained_shell constructed Accepted with a RAW NonEmptyDiagnostics in the diagnostics field where the Diagnostics coproduct (None | Some) is declared - the first downstream match over Diagnostics (diagnostics_merge in the normalize child fold) panicked non-exhaustive on every file with a wrapper-retained fn body, killing the s1_closure witness whenever the affected set selected it. Fix wraps the singleton in Some. Reproduced red (exact CI panic) and green by execution via s1_file_parses on dag/std/error_primitives.dag; new witness pins the shape and returns false (no panic) on the raw form. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Post-merge regen: plan docs re-derive the Lane D links from the merged authorities This branch's earlier regen ran while the .dag plan authorities lacked the link rows (pre-#6503), so it had written the docs linkless; the merge kept that side while main brought the link rows. main_wet now re-derives the linked docs - pair consistent, drift gate green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Two latent fleet-lane witness reds, first executed by this head's wider selection Both red on clean main by direct execution, latent behind affected-set skips and the batch-2 red era: host_standup_spine: the spine gained the P0:host-identity-converge step in #6253 (11 steps) while spine_step_tag matched exactly 10 phases with no catchall - non-exhaustive panic on Assimilation{HostIdentityConverge} whenever executed. Witness now describes the 11-step spine (tag arm + ordinal shift + length/ledger pins 10->11, gap count 4 unchanged, fn renamed ten->eleven to keep the name truthful). fleet_show_effective_read: declared_runner_count() derives from pool_budget / slot cap, so the #6495 re-carve (24->16GiB) moved it 10->5 and the 2026-07-03 width fixtures went stale. The parse-typing witness now pins the literal it parses (a parse fixture must not depend on a live-derived count - that fusion is how this went latent-red); the srv1 convergence witness flips to assert the REAL fact, drift (the hosts still run pre-carve width - the same oversubscription the falsifier exit-137 receipts located), with a dissolve-on for a post-re-carve readback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Brian Searls <briansearls1@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Switches required CI (
ci.yml, all 3 jobs) off the self-hosted[self-hosted, linux, arm64]fleet onto Ubicloudubicloud-standard-16-arm(16 vCPU / 48 GB) dedicated runner VMs.Why (stabilize)
Per operator directive: take required CI off the self-hosted fleet "for now." The fleet is the source of the recurring pain — sccache flakes, OOM-kills, and the spawn-width budget mismatch. On Ubicloud each job gets its own VM, so the 10× oversubscription / memory-admission problem that motivated the whole fleet budget tree disappears by construction (no co-residence).
Modeled, not hand-hacked
gunbc_ci_runner_spec→RunsOnExpression { expression: gunbc_ci_ubicloud_runner_label }. New single-authority data declgunbc_ci_ubicloud_runner_label = "ubicloud-standard-16-arm". (Used the existingRunsOnExpressionvariant rather than polluting GitHub's hostedRunnerLabelenum with a third-party label.)ci_runner_seam_witness_testflipped to assert the runs-on is the Ubicloud hosted label (was: self-hosted arm64). Green by execution..github/workflows/ci.ymlregenerated — byte-drift-gated against the model.arm64 chosen to keep the current arch (zero arch-change risk); 48 GB clears the measured
rust_tests~37 GiB peak. One knob to resize:gunbc_ci_ubicloud_runner_label.Scope — reliability now, speed as a clean follow-up
This is the reliability half. The ~9× floor speedup (
spawn_width1 → ~9 on a 48 GB VM) is deliberately not here: the floor width fold's capgunbc_ci_runner_cgroup_memory_cap(8 GiB) is the load-bearing authority for the self-hosted fleet budget tree (fleet_host_budget,ci_runner_placement,ci_budget_tree+ witnesses). It is genuinely two concepts now (dormant self-hosted cap vs active Ubicloud VM cap); a §3 de-conflation is the right move but shouldn't ride in a stabilization PR that touches the floor scheduler.Merge coordination
Hold merge behind the in-flight re-revert of #6078's resurrected roster (adhoc-9575c77c) — this PR adds one self-tested data decl the resurrected roster would flag as unrostered.
🤖 Generated with Claude Code