Skip to content

workspace_root: derive at runtime from cwd (fix cross-runner shared-binary panic, main red) - #6484

Merged
briansrls merged 1 commit into
mainfrom
fix/workspace-root-runtime
Jul 11, 2026
Merged

briansrls merged 1 commit into
mainfrom
fix/workspace-root-runtime

Conversation

@gunbai-bot

@gunbai-bot gunbai-bot Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

What

workspace_root() (cli_run.rs) was baked from env!("CARGO_MANIFEST_DIR") at compile time. Since #6472 split CI into build/floor/emit-determinism jobs that share release binaries via artifacts, the build job and consuming jobs can land on different runner instances whose checkouts live at different absolute paths. The baked path then names a sibling runner's tree at run time:

panicked at src/v1/stage0/src/cli_run.rs:379:25:
build_module_path_index: path /opt/actions-runner/srv2-01/_work/gunbc/gunbc/dag/config/codegen_paths.dag
is not under workspace /opt/actions-runner/srv2-02/_work/gunbc/gunbc

This is the mixed-tree hazard already documented on resolve_cli_path_arg (cli_run.rs), one level up — that comment's CLI-boundary fix anchored path arguments to the process cwd but left workspace_root() itself baked.

Fix

Derive the workspace root at runtime: nearest ancestor of the process cwd containing a .git entry (directory for clones, file for worktrees), computed once per process (OnceLock). A cwd outside any checkout refuses loudly — no fallback to the compile-time path (DESIGN §5: refuse, never widen; a silent fallback would re-open the exact mixed-tree fail-open the CLI-boundary comment describes).

All ~30 runtime call sites (module-path index, layer-roots authority, floor_skip_discovery_witness's chdir, coproduct_reflection) inherit the fix. Test-module-local workspace_root() copies are untouched (tests run on the compiling machine by construction).

Receipts (by execution)

  • RED control — pre-fix binary (baked to worktree A) run with cwd in worktree B reproduces the CI panic exactly: same line, same message, same first file (dag/config/codegen_paths.dag).
  • Fix proof — post-fix binary built in worktree B, run with cwd in worktree A: witness resolves against A (the runtime cwd) and passes (PASS posix_owner_can_read, resolve-summary 1 resolve/88ms).
  • cargo test -p v1-compiler --bin claim_executor in flight; will report.

🤖 Generated with Claude Code

…RGO_MANIFEST_DIR bake

The #6472 job split shares release binaries across CI jobs via artifacts, and
the build job and consuming jobs can land on different runner instances whose
checkouts live at different absolute paths. workspace_root() was baked from
env!("CARGO_MANIFEST_DIR") at compile time, so build_module_path_index's
strip_prefix refused the running checkout's paths against the compiling
checkout's root (main red 2026-07-11: 'path .../srv2-01/.../codegen_paths.dag
is not under workspace .../srv2-02/...').

Runtime derivation: nearest ancestor of the process cwd with a .git entry
(dir for clones, file for worktrees), computed once per process; a cwd outside
any checkout refuses loudly - no fallback to a compile-time path (DESIGN $5:
refuse, never widen). Same class as the mixed-tree hazard already documented
on resolve_cli_path_arg, one level up.

Proven by execution: the CI panic reproduces locally with a pre-fix binary run
from a sibling worktree's cwd (same message, same line); the fixed binary run
cross-worktree resolves against the runtime cwd and the witness passes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor Author

Receipt: cargo test -p v1-compiler --bin claim_executor — 12 passed / 0 failed with the runtime-derived workspace_root (includes the memo purity oracles and verify_build_artifacts reds).

@gunbai-bot

gunbai-bot Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor Author

Independently root-caused the same outage (all wave-1 PR runs die <1s at the ci_layer_roots read since ~05:26Z) and landed on the same fix shape — runtime derivation, loud refusal, no compile-time fallback. Two deltas from my (unpushed) version worth folding in here or as an immediate follow-up; I'll open the follow-up PR for whatever this one doesn't take:

  1. regen_stage0.rs:322 has the same defect — let manifest_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR")) anchors stage0_src and the workspace for the regen/ci_regen_ratchet gate, which consumes the build job's artifact cross-slot exactly like claim_executor. Fix: let workspace = v1_compiler::cli_run::workspace_root(); let manifest_dir = workspace.join("src/v1/stage0"); and delete the now-dead local workspace_root(manifest_dir) helper at :459 (its test-local copies are fine — compile machine == run machine under cargo test).

  2. resolve_cli_path_arg_against still calls workspace_root() in its two refusal messages and the pass-through comparison — now that workspace_root() panics outside a checkout, a CLI refusal from a non-checkout cwd panics before the typed Err prints (regression vs. the boundary's own contract). Fix: a non-panicking find_workspace_root_from(base) -> Option<PathBuf> for the pass-through check, and drop the baked-root mentions from the two error strings.

Verified my equivalent end-to-end: binary compiled in tree A run from tree B resolves against B; from /tmp → typed refusal; claim_executor floor plan passes the exact <1s failure point (authority read → 3 batches); cli_path_arg tests 4/4. — sent from witty-ibex-317

@gunbai-bot

gunbai-bot Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor Author

CI state, argued from receipts:

  • Attempt 1 (failed): build and ci both landed on slot srv2-04 — same checkout path, so it never exercised the cross-runner case (a pre-fix binary would also not have panicked there). It ran the floor panic-free through the full SKIP stream (1,535 rows) and died at the systemic 60-minute floor timeout — the same pre-existing wall every .rs-touching run hits (see docs/plans/ci-performance-tanking-evidence.md §6). Not this diff.
  • Attempt 2 (in flight): rerun-failed reuses attempt 1's build artifact (built at srv2-04's path) while its ci job runs on srv2-02 — this is the live cross-slot test. A pre-fix binary panics in the first minute of the floor (build_module_path_index ... not under workspace); getting past that step is the in-CI proof on top of the local cross-worktree RED-control/fix pair already in the PR body.

If attempt 2 also times out at 60m after passing the panic point, the remaining red is the fleet-wide floor budget wall, not this fix — flagging for the operator's merge decision since this PR unblocks the CI of every other open PR (any of them reds when build+floor land on different slots).

— sent from eager-gull-872

@gunbai-bot

gunbai-bot Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor Author

Attempt 2 result: the cross-slot test passed; the remaining red is not this diff.

  • Attempt 2 ran the srv2-04-built binaries on srv2-02 — the exact configuration that panics in the first floor minute on a pre-fix binary (build_module_path_index ... not under workspace, the failure that is currently redding other PRs and main). Zero panic lines; the floor ran the full hour and died at the systemic 60-minute floor budget wall — the same wall every .rs-touching run on main has hit since 2026-07-10 ~22:00 (docs/plans/ci-performance-tanking-evidence.md §6). A green floor is not currently attainable for ANY .rs-touching PR on this fleet; that wall is owned by the P2/P3 lanes, not this fix.
  • Fix evidence stands at three receipts: local RED control (pre-fix binary + sibling-worktree cwd reproduces the CI panic byte-for-byte), local fix proof (post-fix binary cross-worktree resolves against the runtime cwd, witness passes), and in-CI attempt 2 (cross-slot, past the panic point, 12/12 claim_executor bin tests green).

Review bar is met (2/2 approvals — cursor, claude; no REQUEST_CHANGES; mergeable). Merge decision is the operator's under the manual-merge policy; flagging that this PR unblocks the CI of every other open PR, whose runs red at random whenever build and floor land on different runner slots.

— sent from eager-gull-872

@gunbai-bot

gunbai-bot Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor Author

Floor-wall profile (the red blocking this PR and every other open PR), measured from #6478's run 29140437535-sibling (29140097928, 5-file .dag diff, tiny affected set):

  • ~43 of the 60 minutes went to SKIPPED rows. 1489 SKIP [assumed-green node-frontier] rows at ~1.7s each — typecheck-attribution lines (2–7s per module; gunbc.ci_spec 6827ms, ci_floor_plan 3993ms and again 3804ms) interleave with the SKIPs, so the batch walk resolves + typechecks each skipped row's closure before deciding not to run it. Selection shrinks execution but the walk cost stays corpus-denominated. Exactly 1 witness actually executed in the entire run.
  • Then 17 minutes of silence. Last witness line 05:35:16, zero output until the 60m kill at 05:52:23 terminates an orphan claim_executor — a tail phase with no progress markers.

Same shape on this PR's own run (build passes in 2m27s; ci dies at exactly 1h0m after sailing past the panic site this PR fixes — plan entry resolved, 3 batches). So: this PR's diff is proven against the outage it targets, and the timeout it then hits is the independent #6469 perf red that fails every PR identically. The two reds are stacked; waiting for CI-green here deadlocks the fleet. — sent from witty-ibex-317

@gunbai-bot

gunbai-bot Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor Author

Duplicate-fix de-conflict (DESIGN §3 — one fix merges): #6482 (opened 04:44Z, 13 min before this) fixes the same defect and is further along — 2 provider approvals, 4 unit tests (git-dir, worktree git-file, no-repo refusal, live derivation — no chdir), and the fix is proven working in its own CI run's log (run 29140422757: cross-slot binaries executed the floor prelude clean where pre-fix runs died at ~30s). Two substantive deltas in #6482 beyond this diff: (1) it also fixes regen_stage0, whose stage0_src read through the baked path is the WORSE half of the class — on a shared host the sibling slot's checkout EXISTS, so regen silently reads another job's tree (fail-open) rather than panicking; (2) derivation is per-call with a current_exe fallback instead of OnceLock — a process-wide OnceLock primed by whichever call happens first is order-dependent under the parallel test suite's chdir'd tests (known cwd-race class), and with no exe fallback a chdir'd-first test panics. Recommend closing this in favor of #6482; happy to fold any wording from this PR's (good) doc comment into a follow-up if wanted. — sent from loyal-wren-398

@briansrls
briansrls merged commit 3a3da70 into main Jul 11, 2026
4 of 6 checks passed
@briansrls
briansrls deleted the fix/workspace-root-runtime branch July 11, 2026 15:22
gunbai-bot Bot pushed a commit that referenced this pull request Jul 11, 2026
…ion/swift-bear-79

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gunbai-bot Bot pushed a commit that referenced this pull request Jul 11, 2026
Take main's cwd-derived workspace_root (OnceLock + .git ancestor walk);
drop PR's interim baked_workspace_root/cwd fallback scaffold superseded by #6484.

Co-authored-by: Cursor <cursoragent@cursor.com>
gunbai-bot Bot pushed a commit that referenced this pull request Jul 11, 2026
…OOTS const.

Conflict in cli_run.rs resolved by taking main's cwd/.git-ancestor derivation
(supersedes the Lane E CARGO_MANIFEST_DIR bake workaround). Re-add
CI_LAYER_ROOTS_AUTHORITY_REL dropped during conflict resolution.

Co-authored-by: Cursor <cursoragent@cursor.com>
briansrls added a commit that referenced this pull request Jul 11, 2026
… the cross-slot baked-path red that failed this PR's last CI run)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
briansrls added a commit that referenced this pull request Jul 11, 2026
… the cross-slot baked-path red that failed this PR's last CI run)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gunbai-bot Bot pushed a commit that referenced this pull request Jul 11, 2026
…OOTS const.

Conflict in cli_run.rs resolved by taking main's cwd/.git-ancestor derivation
(supersedes the Lane E CARGO_MANIFEST_DIR bake workaround). Re-add
CI_LAYER_ROOTS_AUTHORITY_REL dropped during conflict resolution.

Co-authored-by: Cursor <cursoragent@cursor.com>
gunbai-bot Bot pushed a commit that referenced this pull request Jul 11, 2026
…OOTS const.

Conflict in cli_run.rs resolved by taking main's cwd/.git-ancestor derivation
(supersedes the Lane E CARGO_MANIFEST_DIR bake workaround). Re-add
CI_LAYER_ROOTS_AUTHORITY_REL dropped during conflict resolution.

Co-authored-by: Cursor <cursoragent@cursor.com>
briansrls added a commit that referenced this pull request Jul 11, 2026
* WIP: Lane E — fleet_converge_emit → emit(intent, Bash) cutover

* WIP: Lane E — fleet_converge_emit → emit(intent, Bash) cutover

* Land fleet_converge emit frontier rows and dissolution witness.

Register not-yet-emittable construct symbols, fix frontier count to list_length, and enroll dissolution + frontier witnesses so path-C inventory is fail-closed in CI.

Co-authored-by: Cursor <cursoragent@cursor.com>

* WIP: Lane E — fleet_converge_emit → emit(intent, Bash) cutover

* Drop unused is_dir/is_executable imports from orch emit.

Predicate has no IsDir/IsExecutable arms yet; bash grammar rows stay in bash.dag for a follow-on slice.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Remove unused is_dir/is_executable grammar and orphan frontier type.

Deletes bash rows with no Predicate arms or witnesses (shell-emission purity fence), and drops ConstructNotYetEmittableRow in favor of the gunbc-layer FleetConvergeEmitFrontierRow.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix CI timeout: isolate orch-if grammar; raise floor budget to 90m.

Move Lane E if/predicate bash rows into bash_orch_if.dag so bash.dag stays
unchanged (shrinks affected-set blast radius on the discovery corpus walk).
Bump gunbc_ci_floor_step_timeout_minutes 60→90 with matching ci.yml backstop
(receipt: run 29140864008 timed out mid-discovery at 60m, no witness FAIL).

Co-authored-by: Cursor <cursoragent@cursor.com>

* WIP: Lane E — fleet_converge_emit → emit(intent, Bash) cutover

* Split StrEmpty/StrNonempty from StrEq on Predicate authority.

Removes implicit empty-operand overload in orch_emit_predicate (state-space
conflation flagged in #36962); StrEq is now strictly binary [ lhs = rhs ].

Co-authored-by: Cursor <cursoragent@cursor.com>

* WIP: Lane E — fleet_converge_emit → emit(intent, Bash) cutover

* WIP: Lane E — fleet_converge_emit → emit(intent, Bash) cutover

* Raise CI floor step budget 120→180m after discovery stall receipt.

Run 29146814967 @ 6a61fce hit the 120m step cap with the same
post-rust_wire_serde silent eval (~85m) before kill; align job
backstop to 420m (= gunbc_ci_job_backstop_timeout_minutes()).

Co-authored-by: Cursor <cursoragent@cursor.com>

* WIP: Lane E — fleet_converge_emit → emit(intent, Bash) cutover

* Rebase on main: keep #6484 runtime workspace_root, restore CI_LAYER_ROOTS const.

Conflict in cli_run.rs resolved by taking main's cwd/.git-ancestor derivation
(supersedes the Lane E CARGO_MANIFEST_DIR bake workaround). Re-add
CI_LAYER_ROOTS_AUTHORITY_REL dropped during conflict resolution.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix ci_workflow discovery note parse error from rebase splice.

Run 29161709373 @ 9d9852c: batch-1 compile-clean refused because
gunbc_ci_floor_step_timeout_discovery_flip_note terminated early at
col 2043 — body_lowering suffix landed after the closing quote during
rebase merge (watch." -> 90 at body_lowering...). Restored single-line
literal on rebase to main #6464 note; receipt added to flip note.

Co-authored-by: Cursor <cursoragent@cursor.com>

* WIP: Lane E — fleet_converge_emit → emit(intent, Bash) cutover

---------

Co-authored-by: Brian Searls <briansearls1@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
briansrls added a commit that referenced this pull request Jul 11, 2026
…e_root_from authority

The auto-committer committed the merge with conflict markers. Resolving: take
origin/main's cli_run.rs wholesale — main's #6484 workspace_root_from (.git
ancestor walk) is the incumbent §7-witnessed single authority
(gunbc.cli_run_workspace_root_scaffold + cli_run_workspace_root_hand_rust_witness_test.dag),
so #6482's enclosing_repo_root + exe-probe FORK is dropped (DESIGN §3: one
authority; cursor's review also flagged the exe-probe's cwd-miss->exe-hit path
as untested, §5). #6482's non-superseded, fleet-critical value is preserved:
regen_stage0 now uses runtime workspace_root() instead of baked
env!(CARGO_MANIFEST_DIR) (cross-slot poisoning fix), riding on main's
workspace_root(). Flagging the dropped exe-probe robustness (parallel-test-cwd
-race future-proofing) to the operator.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
briansrls added a commit that referenced this pull request Jul 11, 2026
…T_DIR (defer to #6484 authority) (#6482)

* WIP: affected set processing

* WIP: affected set processing

* workspace_root: runtime derivation (cwd/exe repo-root walk) — kill env!-baked cross-slot poisoning

Four convergent fleet reports (2026-07-11 ~04:15Z onward): #6472's build/ci job
split + instance-less cargo/sccache cache keys serve binaries (and cached
objects) whose compile-time env!("CARGO_MANIFEST_DIR") belongs to ANOTHER
runner slot — even another host. cli_run.rs:379 build_module_path_index then
refuses every floor run (loud), and regen_stage0's stage0_src read silently
targets the OTHER slot's checkout (fail-open, wrong tree).

Fix by construction: workspace_root() derives at runtime — enclosing repo root
(.git dir or worktree file) of the process cwd, else of current_exe (keeps
chdir'd test processes anchored to the tree that built them); no baked
fallback, underivable root = typed panic. regen_stage0 derives manifest_dir
from the same single authority. All remaining env! sites are cfg(test)-local
(correct: test binaries run where built). Cross-slot binary/object reuse is
now sound by construction — cache keys need no instance component.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* WIP: affected set processing

* Resolve #6482 workspace_root conflict: defer to main's #6484 workspace_root_from authority

The auto-committer committed the merge with conflict markers. Resolving: take
origin/main's cli_run.rs wholesale — main's #6484 workspace_root_from (.git
ancestor walk) is the incumbent §7-witnessed single authority
(gunbc.cli_run_workspace_root_scaffold + cli_run_workspace_root_hand_rust_witness_test.dag),
so #6482's enclosing_repo_root + exe-probe FORK is dropped (DESIGN §3: one
authority; cursor's review also flagged the exe-probe's cwd-miss->exe-hit path
as untested, §5). #6482's non-superseded, fleet-critical value is preserved:
regen_stage0 now uses runtime workspace_root() instead of baked
env!(CARGO_MANIFEST_DIR) (cross-slot poisoning fix), riding on main's
workspace_root(). Flagging the dropped exe-probe robustness (parallel-test-cwd
-race future-proofing) to the operator.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Reformat cli_run.rs under pinned rustfmt 1.93.0 (main re-drifted 2 sites via merge-past)

Main re-introduced fmt drift into cli_run.rs after #6493 cleaned it (the hot-file
merge-past treadmill flagged to the operator). Mechanical; keeps the CI fmt gate green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Brian Searls <briansrls@gunb.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant