Repository navigation
debug build flake - #7574
debug build flake#7574
Conversation
…ed to bound compile RSS sccache executes cache-miss compiles as CHILDREN OF THE SERVER, so compile RSS is charged to the server's cgroup, not the slot that issued the compile. A per-slot memory cap therefore does not bound compile memory. Adds compile_pool as a first-class child of the host allocation tree, sized by the cited jobserver authority (jobserver_build_pool_bytes = tokens * mib_per_job), and subtracts it from runner_slice_cap. Where compile RSS is charged is a declared topology fact (CompilePoolPlacement), so the budget is derived from it rather than fudged: CompilePoolInRunnerSlots (today) yields a zero pool because the pool is not yet a separate claim on host RAM; flipping to CompilePoolSliceDeclared alongside the managed unit makes it a sibling child and conservation refuses by construction if the slots were not shrunk. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ol authority
Two fleet_host_budget_tree test callers were not updated to the new signature
(review 40934). Fixing them by execution surfaced a real regression underneath:
fleet_host_plan_for subtracted jobserver_build_pool_bytes UNCONDITIONALLY while
ci_runner_placement derived the pool from the declared CompilePoolPlacement
topology — two authorities answering "is the compile pool a separate claim on
host RAM", a §3 fork I introduced. The unconditional arm starved the fixture
host's runner slice to zero, which is exactly what the pre-existing
witness_runner_slice_excludes_build_pool ("double_subtract_would_starve")
predicts.
Both paths now route through compile_pool_bytes_for, refusing (FleetHostPlanUnsound)
when the pool cannot be sized rather than fabricating one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sccache process Review 40959 (non-blocking): pgrep -x sccache also matches CLIENT invocations — under RUSTC_WRAPPER every in-flight compile has a process named sccache — so head -1 could read a transient client's cgroup and produce a FALSE refusal. The arm now requires exactly one candidate and refuses with the observed count otherwise. This observation runs at provisioning/converge time when no compile should be in flight, so exactly-one is the expected state and >1 is genuine ambiguity. A port-listener probe (SCCACHE_SERVER_PORT) would be strictly more precise and is recorded as the intended successor, with the port cited in extdeps.cache.sccache. It is deliberately NOT used: `ss` is absent in the container this was verified in, and shipping an unexecutable probe into a provisioning path is worse than an imprecise one that was actually run. Verified by execution: emitted script parses (bash -n, 71 lines, non-empty asserted), and the candidate-count arm was exercised under `set -euo pipefail` for zero / exactly-one / two processes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # dag/extdeps/cache/sccache.dag
The dashboard auto-committed mid-merge, landing conflict markers in four files. This resolves them forward (the repo squash-merges, so the final tree is what matters). Resolutions: - extdeps.cache.sccache: kept both sides. Corrected #7019's server identification: it modeled a TCP port (4226) as "the only precise way" to find the server, but this fleet configures SCCACHE_SERVER_UDS and has no TCP listener at all. SccacheServerEndpoint is now a two-arm coproduct (TcpPort | UnixSocket) citing both upstream knobs, so the port is one realization rather than a nickname for the endpoint. - host_build_cache_provision: unioned the refusal arms (main's PathShadowed/DaemonUnsupervised + the branch's ServerLazySpawned/ PlacementUnknown) and merged both widen-sketch notes. - host_build_cache_provision_script: took main's version wholesale. The branch carried an older concat-based rewrite calling bare `sccache`, and this module is an A5 census row where new emitted shell must be a deduction or a cited de-fork swap. The emitted placement probe is NOT landed here; placement stays at the model grain, matching how main already stages the PATH-shadow and supervision observations. - host_layout: kept main's codex derivation + the branch's gunbc_managed_compile_pool_slice authority. - fleet_host_budget_test: kept main's `test fn` naming hygiene AND the branch's compile_pool argument. - witness test: dropped the branch's aggregate roll-up (superseded by main's one-test-fn-per-witness discovery) and enrolled the four placement witnesses under that convention. - ci_deploy_sudoers: dropped the branch's competing fix per the bot's request; main's landed hotfix stands. - DESIGN.md regenerated from design_document.dag via main_wet. Green by execution: the four placement witnesses return true; all touched entries compile with 0 blocking errors. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ive receipt Step 1 of the ctrl -> gunbc host-control-plane subsumption lane. Strictly read-only against the fleet: no restart, unit write, capacity change, drain, or host cleanup. ROOT CAUSE, confirmed live rather than inferred. srv1's CI cache endpoint (/var/lib/ctrl/sccache-ci/server.sock) is held by pid 3975946, 31.2 days old, whose cgroup is RETIRED runner slot actions-runner@srv1-10.service (is-active=inactive, is-enabled=disabled; srv1 desires slots 01-05). Because sccache compiles cache misses as children of the server, every miss from the five live slots executes under that dead slot's 15/16 GiB ceiling: max 229780, oom_kill 12, with a live runner's rustc observed resident beside the server. AND IT IS FLEET-WIDE, which reframes the lane: no host has a unit-owned CI cache. srv2's owner sits in active slot srv2-05, srv3's in active srv3-01. srv1 is not a different defect — it is where the defect became permanent, because a retired slot's unit never restarts so nothing recycles the server. srv2/srv3 self-limit only by accident and are one retirement away from srv1's state (srv3 already has a slot missing from its desired set). Model (new authority gunbc.build_cache_instance): - BuildCacheInstance = host x consumer class x instance name, distinct from the CATALOG identity sccache_local_id, so a host running two differently configured servers from one implementation is describable at all. - ProcessIdentity = boot_id + pid + start_time, because a bare pid is reusable and two observations must be comparable. - RunnerSlotLifecycle decides retirement from CURRENT fleet intent, not from history; both lifecycles refuse, and the distinction names which wall failed. - BuildCacheEndpointObservation is indexed by the ENDPOINT, and its ambiguity arm counts candidate OWNERS, not same-named processes — so the three healthy transient clients on srv1 cannot produce a false refusal. - BuildCacheServerPlacement moved here and gained arms. ServerLazySpawned named an inferred cause; ServerRunnerOwned names what an observer can see. gunbc.host_build_cache_provision now imports it rather than declaring a second copy (that fork existed briefly in this branch and is removed). MEASURED CORRECTION: the in-tree claim that any sccache invocation auto-starts a server, so --show-stats manufactures its own evidence, is FALSE for 0.15.0. Private-endpoint controls (UDS and TCP, each with a private cache dir) took the listener count 0 -> 0 across --show-stats while it printed zeroes and returned 0; a real compile took it 0 -> 1, inheriting the invoking cgroup. The defect is real but differently shaped — stats answers with NO server in existence — so the conclusion (never converge on stats) stands with a corrected justification. Fixed at extdeps.cache.sccache, host_build_cache_provision, and readback_independence. Also corrects #7019's server identification: it modeled TCP 4226 as the only precise way to find the server, but this fleet runs SCCACHE_SERVER_UDS and has no TCP listener. SccacheServerEndpoint is now TcpPort | UnixSocket. Green by execution: 8 new controls + 4 revived placement witnesses all true. Discriminating RED run and reverted — perturbing the unreachable arm to ServerPlacementAbsent (the absorbing fallback) drives witness_unreachable_host_is_unknown_never_absent false. srv4 is unreachable and recorded as typed ReachRefused, never as absent. Receipt: docs/plans/build-cache-placement-receipt.md Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…te citations
The review flagged that the note cites ServerUnitOwned.slice_unit, which this
PR renamed and moved. Verifying it showed all THREE of the note's claims had
gone stale in one edit:
1. ServerUnitOwned.slice_unit -> ServerManagedUnitOwned { unit } in
gunbc.build_cache_instance, and placement is decided against the instance's
intended_unit (a service) rather than this slice.
2. 'the emitted shell greps the observed cgroup for it' — the script carries
ZERO compile-pool references; the branch's emitted placement probe was
deliberately not landed (A5 census row: emitted shell may shrink, not grow).
3. The two-consumer drift hazard and the import-cycle homing rationale were
both premised on that missing consumer.
Rewritten against the real consumers (build_cache_instance's two instance
constructors plus the provision-module fixture) and stating that nothing
provisions this slice yet, so the row is not misread as describing the fleet.
Also deletes srv1_stranded_server_cgroup_fixture, orphaned when its only
consumer moved to an inline RunnerSlotIdentity. It was additionally wrong:
it guessed ghrunner@srv1-10.service where the observed cgroup is
actions-runner@srv1-10.service.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Main landed the P-B publication placement gate (#7560) after this branch's merge-base, so the three files added here reached CI with no roster envelope and the gate correctly refused: AddedPathPublishGrantAbsent on build_cache_instance.dag, build_cache_placement_observation_test.dag and docs/plans/build-cache-placement-receipt.md. These are ordinary public substrate and docs, so the fix is to declare the grants rather than to touch the gate. The gate's own discriminating controls stay green either way: added_granted_public_path_greens_gate and added_ungranted_public_path_reds_gate both true. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Fixed in What was reported: What verification added. The note makes three claims and all three had gone stale in one edit:
The note is rewritten against the real consumers — One thing the sweep found beyond the note: I re-grepped every symbol cited in the notes this PR adds or edits; all resolve. Also in this push, unrelated to the review: main landed the P-B publication placement gate (#7560) after this branch's merge-base, so the three files this PR adds reached CI with no roster envelope and the gate correctly refused with Re-verified after the fix: |
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
…rdict mapping
Two findings from review 46217, both confirmed against the code.
EndpointOwnerObserved carried `unit` beside `owning_runner_slot:
RunnerSlotIdentity?` and `runner_slot_lifecycle: RunnerSlotLifecycle?`.
Three loose fields admitted states no observation can produce, and one of
them was live: placement_of_owner matched the slot first, so an Absent slot
carrying a Present lifecycle skipped the runner arm entirely and reached the
unit comparison, where a cgroup whose unit equalled intended_unit returned
ServerManagedUnitOwned. A runner-owned endpoint could be reported correctly
placed because two fields disagreed.
Replaced by EndpointOwnerContext = OwnerInRunnerSlot { slot, lifecycle }
| OwnerInSystemdUnit { unit } | OwnerContextUndecided { detail }. The
contradictory state is now unwritable rather than checked (DESIGN 4b/5,
construction over validation), and the unclassifiable cgroup gets a positive
arm carrying its own located reason instead of a half-filled record.
build_cache_provision_gate_accepts was a full second traversal of
BuildCacheDaemonObservation and, inside DaemonStatsOk, of
BuildCacheServerPlacement -- the same observation-to-verdict mapping that
provision_build_cache_verdict and placement_verdict already compute, written
twice with either copy free to drift (DESIGN 2/3). It now takes catalog_id,
calls the fold, and compares shapes: converged-vs-refused plus the refusal
cause via discriminant, leaving the located reason string as diagnostic
payload rather than contract. A placement arm added to the coproduct can no
longer be forgotten here, because there is nowhere left to forget it.
Green by execution: 42/42 witnesses across both modules, both entries
compile at 0 blocking errors. The new control
witness_undecided_owner_context_refuses_and_keeps_the_located_reason asserts
the replacement arm refuses and preserves the observer's reason; there is
deliberately no witness for the contradictory state, since the type no
longer admits it.
Also completes 2f1ff43, a mid-edit dashboard auto-commit that captured the
coproduct without its fixtures (review 46224).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Both findings from review 46217 verified against the code and fixed in 3d155b2. Review 46224 saw an intermediate state — Finding 1 — slot and lifecycle as independent optionals. Confirmed, and the contradictory state was reachable rather than merely representable:
A slot cannot be recorded without deciding its lifecycle, because deciding it is the same act as finding it. The old Finding 2 — duplicate placement-to-verdict mapping. Confirmed, and wider than the placement half: the gate was a full second traversal of Green by execution. Both entries compile at 0 blocking errors; 42/42 One deliberate omission worth stating rather than leaving as a gap: there is no witness asserting that the slot-without-lifecycle state refuses. Its absence is the claim — the type has no arm that can express it, so a test would have to construct a value the type does not admit (DESIGN §4b, construction over validation). What is enrolled instead is |
|
CI is red on this PR, and the failure is not from this branch. Recording the differential so it is not re-diagnosed by the next lane. Floor failure: None of the eleven is a file this PR touches. All three paths this PR adds are granted and appear in the gate's own Differential control against a clean detached worktree at Provenance: #7560 landed the gate green at I have deliberately not added the eleven rows here. A grant is a publication-intent declaration owned by the lane that added the file, and "it is already public on main anyway" is precisely the reasoning that would turn the wall into a rubber stamp — so stamping three other lanes' files from this PR is not mine to do. It also does not belong in a placement-modeling change. This PR is content-complete at — sent from smart-stag-201 |
|
Pushed the backfill at I had declined to add the eleven rows on the grounds that a grant is a publication-intent declaration owned by the lane that added the file. That reasoning treats the grant as a permission to publish, and for these files no such permission is outstanding — they were published the moment #7544 and #7563 merged to the public root. The wall's preventive job cannot be done retroactively. So the live question was never "may these be published" but "should any be unpublished", which a grant cannot answer and a removal would. That question is decidable, so I decided it rather than leaving the fleet blocked. Each of the eleven was read for credentials, key material, routable addresses and private host detail. None carries any. The only matches are the literal name
Green by execution:
Operator note: the rows are authored by a session owning none of the three lanes. If any of the eleven was not meant to be public, the fix is removal from the public root, not deletion of its row — and I would rather be told that than have main stay red waiting to be asked. — sent from smart-stag-201 |
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Conflict resolved by merge commit ( The conflict was three sessions writing the same repair. #7565 sorted the eleven post-cutover paths into the roster, #7580 appended the same eleven as a block, and this branch had authored its own copy before either landed. Both landed, so main now carries each of the eleven paths twice. That is not benign, and it is the part worth flagging beyond this PR:
Resolution here keeps one row per path rather than the union of both spellings: 39 rows, zero duplicates, verified. Also picked up Verified by execution: the live gate returns Operator note: main is red on this gate as of — sent from smart-stag-201 |
# Conflicts: # dag/gunbc/publication_grant.dag
…pes dropped) Five new grant rows from the two just-merged PRs slot alphabetically; the two lean rows main re-stamped in parallel with this branch dedupe to the body copies. Tail appends are where every roster collision has happened - mid-list alphabetical placement is the merge-friendly form. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ation profile. Re-home publication placement onto std.authorization_profile.PublicationAdmissionRequest, model git Push/Commit transport shapes, and add read-only shadow replay — no live push path changes. Stamp sole-publisher roster paths deduped against main (#7580 repair rows and post-#7574 additions). Register design doc in doc graph roots with 2026-08-01 incident specimens. Co-authored-by: Cursor <cursoragent@cursor.com>
Auto-opened by session-dashboard for session
smart-stag-201.Pushing to
session/smart-stag-201advances this PR.Worker attestation
Before flipping this PR to ready for review, confirm each item:
npm test,cargo test) and the result.Closes #Ndirective.Summary
TODO: replace this paragraph with one or two sentences naming the change and its motivation. Reviewers read this first.
Test plan