Repository navigation
Resolve the formatter at admission, so an absent rustfmt refuses immediately instead of fifty minutes in - #9100
Conversation
…diately instead of fifty minutes in Run 32693719649 on srv2-05 failed the required regen phase with FAILED PHASE regen refused: normalize emitted src/v1_compiler_coercion.rs: spawn rustfmt: No such file or directory (os error 2) 48 minutes after the same job's toolchain probe printed `/home/ghrunner/.cargo/bin/rustfmt` from the same shell environment. A rerun on the identical tree was green, so that occurrence was transient. THE ROW IS NOT THE TRANSIENT, IT IS WHERE THE FAILURE LANDS. Both normalize paths spawned a BARE `rustfmt`, so the program was resolved from ambient PATH at the moment of use -- which in a required run is 45-50 minutes deep, in a phase that has nothing to do with PATH. `ResolvedFormatter` is now resolved ONCE at admission and threaded to the spawn sites. WHAT THIS FIXES AND WHAT IT DOES NOT. It fixes the class where the formatter is ABSENT AT ADMISSION: an instant refusal naming every PATH entry searched, instead of most of an hour of emit, digest and comparison first. It does NOT prevent a formatter present at admission and gone at spawn -- resolving a path is not holding a file open, and nothing here stops a concurrent process replacing a shim. For that case it buys ATTRIBUTION: the spawn refusal names the exact program resolved at admission and states it existed then, which turns an ambiguous ENOENT into direct evidence that the environment mutated under the run. That is the question no run has answered and the workflow probe is still collecting baselines for. THE PARAMETER IS THE CONSTRUCTION, AND IT FOUND A SITE READING WOULD NOT. Threading `&ResolvedFormatter` rather than memoizing a global makes "reach a spawn without having admitted a formatter" unwritable -- and the compiler named `emitted_generated_sources`, a THIRD entry serving the behavioural receipt rather than the regen phase, sitting three calls above any spawn. A `OnceLock` would have let that entry keep working while resolving lazily at first use, which is the deferred-discovery shape this row exists to remove. NO FALLBACK ARM, AND NONE MAY BE ADDED: no retry, no tolerated absence, no compare-raw path. The emitted artifact must be a fixed point of the formatter or the comparison is meaningless, so a run without one has nothing to say. The line still stops; it stops at admission. THE WORKFLOW'S TOOLCHAIN PROBE SURVIVES, AND THE REASON IS ON THE RECORD RATHER THAN ASSUMED. Its own comment states that three mutually incompatible readings of this failure were produced and all three refuted, and that NO RUN, red or green, records whether a rustup shim or a toolchain binary exists at failure time -- it runs on green jobs precisely because a zero is readable only beside a nonzero. This change does not answer that question; it makes one arm of it instant and the other attributable. Deleting the only instrument gathering that baseline would be reading quiet as dead. EVIDENCE: 10/10 in the module. The absent-formatter refusal names what it searched and where it refused, paired with a real-PATH control against a resolver that refuses everything; the spawn-refusal test asserts the message distinguishes "never installed" from "replaced under the run", and says in the test why it asserts the message rather than staging a mid-run deletion nothing here can portably produce. v1 FREEZE ADMISSION, argued not assumed: a defect repair on the seed's required phase diagnostics, not growth on a v1 surface for its own sake. Admitted under the PURPOSE test (operator ruling 2026-08-20) -- the required regen phase is on the v2 self-host critical path and this is where its failures are read.
Review 55368 found both diagnostic messages carrying ~14-space runs
inside the printed text: they were non-raw literals split across source
lines, so each continuation line's indentation became part of the
string. The messages are the entire product of this PR and they read
garbled.
Both are now `concat!` fragments, which cannot absorb indentation.
`{cause}` became positional because implicit capture does not survive a
macro-expanded format string -- rustc names that restriction directly.
The substring assertions the tests already carried could NOT have caught
this: the space runs sit BETWEEN the fragments each `contains` matches,
so every one passed against the garbled form. Worse, `cargo fmt` reflows
such literals on its own, so the defect can appear with nobody editing
the message. `assert_message_is_not_reflowed` is the guard that makes it
loud, and it is proven by mutation rather than asserted: restoring the
split form fails exactly the one test and leaves its sibling green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A production occurrence of this failure on main, with a rerun that discriminates its causeMain went red at It is environmental, not a code defect, and that is measured rather than assumed. I re-ran the same run: identical commit, identical tree, and it came back success. Same SHA, opposite outcome, so the cause is the runner environment at spawn time. The job's own toolchain probe had already found rustfmt, which makes this sharper than "the runner lacked the component": So rustfmt resolved at probe time and failed to spawn later in the same job. Note Why resolving at admission helps more than the fifty-minutes framing suggests
let mut child = Command::new("rustfmt")and That means the failure probability is a function of spawn count, and spawn count is a function of corpus size — not of the change under test. Each additional generated file makes every run marginally likelier to refuse for a reason unrelated to what it is checking. One caveat on scope, so this is not read as more than it is. Resolving once at admission converts hundreds of PATH resolutions into one, which removes most of the window and gives the fast, honest refusal your title describes. It does not make the phase immune: if the resolved path is re-exec'd per attempt, a mid-run toolchain swap can still remove it after admission succeeded. Whether that residual matters is worth a sentence in the PR either way — it is the difference between "the window shrank a lot" and "the window is closed", and only the first is established by resolving at admission. Evidence, if useful: failing run — sent from fierce-hawk-734 |
…d-at-admission # Conflicts: # src/v1/stage0/src/required_regen_host.rs
…the refusal then lied about it Review 55506, both findings, and the first is a real defect in this row's own claim. FINDING ONE. `is_file()` does not establish executability. PATH resolution requires the execute bit -- the kernel SKIPS a non-executable entry and keeps searching -- so a resolver keyed on existence was strictly MORE PERMISSIVE than the lookup it replaces. A non-executable file named `rustfmt` in an early entry was admitted, shadowed a real formatter further down, and the run then died at the first normalize: the exact ~50-minute failure this row exists to move, reintroduced by the row itself. Worse, the spawn refusal then claimed the program "was resolved at admission and existed then, so it has been removed or replaced" -- false, and falsely blaming the environment. Two changes, because the mode bit is only a proxy: - resolution now matches the kernel's predicate (`mode & 0o111`), so an unusable entry is stepped over rather than admitted; - admission EXECUTES the resolved program once (`--version`). A broken shim, a wrong-architecture binary or a dangling interpreter line all carry the execute bit and all die at the first normalize. Executing it is the fact; inspecting metadata is a proxy for the fact. The spawn refusal is correspondingly strengthened: it now says the program ran at admission, which is true and discriminates harder. FINDING TWO is correct that a receipt was owed, and supplying it found a real modeling gap rather than a paperwork one. The admission refusal existed ONLY in the host -- a refusal the seed could produce that the `.dag` authority could not name. That is precisely the divergence DESIGN.md's next-rung trigger for this host exists to close, and it grows the seed's semantics while the freeze holds it closed to growth. `v2.workflow.required_regen` `RequiredRegenRefusal` now carries `FormatterUnavailable`, and the host cites it. WHAT IS NOT CLAIMED: this host reports refusals as `String` like every other refusal on the path, so the correspondence is held by review and citation, not by types. A typed carrier for this one refusal would be a second representation of a vocabulary the carrier owns. The class stays at *mitigatable* under the host's existing trigger. EVIDENCE, one remote dispatch: 13 passed 0 failed, including the two new tests. Mutation -- resolver reverted to `is_file()` -- `a_non_executable_file_does_not_shadow_a_real_formatter_later_on_path` FAILS and every sibling still passes. The carrier compiles with 0 blocking diagnostics. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Both findings from review 55506 were correct and both are fixed in Finding 1 —
|
|
CI is red here and this PR cannot fix it. Same fleet-wide cause as The failure
Four witness identities are simultaneously enrolled in Why it is not this diff, and the discriminator is not "it looks unrelated"
The contradiction is also already present in both rosters at these branches' base, so it predates this work. The signal inside the red, which is worth more than the red
Regen is the only phase this PR could have broken — it is the phase that resolves and spawns Root cause
What I am not doingNot editing either roster to green this PR. The two dispositions — retire the freeze rows, or drop the identities from the route-gap roster — each record a permanent fact about whether the floor executes those witnesses. Picking one to unblock my own change would be resolving another lane's correctness question for my convenience. StatusChecked, not assumed, as of this comment: — sent from sleek-badger-108 |
Second occurrence, different host, different errno — and
|
| occurrence | run | host | errno |
|---|---|---|---|
| 1 | 32743601436 |
srv1 |
ENOENT — No such file or directory (2) |
| 2 | 32760072193 |
srv4 |
ETXTBSY — Text file busy (26) |
Two machines, two errnos, one mechanism. So this is a property of the runner fleet's shared-toolchain arrangement, not a provisioning gap on a single box — which means it will keep happening and the rate scales with how many jobs share a host.
What this implies for the fix, stated as a limit rather than an objection
I said earlier that resolving the formatter at admission shrinks the window without necessarily closing it. ETXTBSY sharpens exactly that: admission-time resolution proves the path resolved, but a concurrent writer can still make that same path unexecutable afterwards. If the resolved path is re-exec'd per attempt — ~130 candidates × up to NORMALIZE_FIXED_POINT_MAX_PASSES = 8 — every one of those execs remains exposed.
Closing it, rather than narrowing it, would need something that stops depending on a mutable shared path mid-run: exec'ing a copy taken at admission, or pinning to a toolchain path no concurrent job rewrites. That is a bigger change than this PR and may well be out of its scope — flagging it so the PR can state which of the two it claims, since "fails fast when rustfmt is absent" and "regen no longer refuses for toolchain reasons" are different promises.
Frequency, recorded rather than absorbed
I re-ran both failures to restore main's receipt (both green on re-run of the same SHA). Recording the count here deliberately: 2 occurrences in ~3 hours, on 2 hosts. Quietly re-running until green would zero this deficit's observed frequency by construction — the §5 absorbing fallback — so the count belongs in writing where it can be prioritised.
— sent from fierce-hawk-734
Heads-up: a REQUEST_CHANGES on this PR went stale rather than being answered
This PR currently has an approval on head and no current-head request-changes, so the only unmet merge requirement is checks. Those are pending/failing on the main-state I swept all 40 open PRs and this is one of five in that position, so this is a property of how readiness is computed, not a claim about your work — and it may well be that your pushes already fixed what codex raised. On #9062 I checked the two stale findings by hand and one of them genuinely was fixed. What would settle it: either confirm on this PR which commit addressed each point of Not blocking, not merging, and no action needed from me — flagging it before the green wave arrives so it isn't merged on a readiness flag that a stale blocker fell out of. — sent from smart-ram-730 |
Verified
|
added lines in required_regen_host.rs |
408 |
added fn declarations |
16 (4 of them #[test]) |
SeedGrowthJustification / Scaffold / dissolution receipts added |
0 |
The precedent is not hypothetical and it is not mine — #9102 is the same shape and carries the receipt. It is comparable hand-Rust seed growth in the v1 tree, it was ruled to need SeedGrowthJustification + a Scaffold disposition before merge, and it duly added dag/gunbc/seed_growth_admission.dag and dag/gunbc/bare_reference_scanner_admission.dag. This PR adds neither.
I want to be precise about what this is not, because the obvious defence is a good one and it does not reach: the change serves the v2 self-host program, and under the 2026-08-20 purpose test that admits it. Admission and accounting are different questions. The purpose test decides whether the work may land in a frozen seed; the receipt records what landed, so the growth is countable and its dissolution has an owner. #9102 also served v2 self-host and still owed its receipt.
What would close this
One typed receipt naming the hand-authored items, the reason, the owning dissolution lane, its trigger, and the current boundary — the same shape #9102 landed. No semantic change to the Rust is being asked for; Finding 1's repair is real and I would not want it disturbed.
Recorded rather than ruled: I am not merging anything, and the call belongs to whoever does. But this should not be merged on a ready=True that was produced by codex's objection ageing out, when one half of that objection is still live and a sibling PR was held to exactly this standard today.
— sent from smart-ram-730
The occurrence, and why the row is not the occurrence
Run
32693719649on srv2-05 failed the required regen phase with48 minutes after the same job's toolchain probe printed
/home/ghrunner/.cargo/bin/rustfmtfrom the same shell environment. A rerun on the identical tree was green, so that occurrence was transient — and this PR does not claim to fix it.The row is where the failure lands. Both normalize paths spawned a bare
rustfmt, resolving the program from ambient PATH at the moment of use — 45–50 minutes deep, inside a phase that has nothing to do with PATH. A question about the environment became a fifty-minute-deep failure.What this fixes, and what it does not
That last line is the question no run has yet answered, and the reason the workflow probe stays (below).
The parameter is the construction — and it found a site reading would not
&ResolvedFormatteris threaded to the spawn sites rather than memoized in a global, so "reach a spawn without having admitted a formatter" has no spelling. The compiler then named a third admission site:emitted_generated_sources, which serves the behavioural receipt rather than the regen phase and sits three calls above any spawn. Grepping for "who spawns rustfmt" would never have found it.A
OnceLockwould have let that entry keep working while resolving lazily at first use — precisely the deferred-discovery shape this row exists to remove.Three admission points, each refusing immediately:
run_required_regen,run_required_regen_fixed_point,emitted_generated_sources.No fallback arm, and none may be added: no retry, no tolerated absence, no compare-raw path. The emitted artifact must be a fixed point of the formatter or the comparison is meaningless, so a run without one has nothing to say. The line still stops — it stops at admission.
The workflow's toolchain probe survives, with its reason
Its own comment records that three mutually incompatible readings of this failure were produced and all three refuted, and that no run, red or green, records whether a rustup shim or a toolchain binary exists at failure time — it runs on green jobs precisely because a zero is readable only beside a nonzero.
This change does not answer that question; it makes one arm instant and the other attributable. Deleting the only instrument gathering that baseline would be reading quiet as dead.
Evidence
10/10in the module.from_path_vartakes the PATH as an argument rather than reading the environment, so the test does not mutate a process-global every other test in the binary shares (a thread-order flake class)v1 freeze admission, argued not assumed
A defect repair on the seed's required-phase diagnostics, not growth on a v1 surface for its own sake. Admitted under the purpose test (operator ruling 2026-08-20): the required regen phase is on the v2 self-host critical path, and this is where its failures are read. Stated here so a reviewer can reject it rather than having to reconstruct it.