recurring_failure_mode: runner-identity-derived off-fleet premise in wet witnesses (repaired by #12478)

[gunbai-bot]

What this PR is now

The code fix landed first in #12478, which supplies the off-fleet premise at gunbc.instruments.fabric_capacity_standing fabric_capacity_standing_for and adds real-read route claims that hold on any runner. This PR's parallel repair was superseded by it, so merging main reverted every code file here to main's version. What remains is the one thing #12478 did not file: the gunbc.recurring_failure_mode row DESIGN §4b requires for a newly discovered class.

• New: dag/gunbc/recurring_failure_mode/wet_witness_premise_read_from_the_runner_it_happens_to_land_on.dag. The class is a wet claim whose expected verdict names a host property (off fleet, no dashboard) while it takes its subject from the executing runner. Its verdict then depends on runner assignment: srv1 is both a CI runner and a declared dashboard host, so the claims were red only on srv1 runners, for example mtcollins1 media attach converges to the desired census image (stop stale, await re-enumeration, attach by name, readiness gate before power) #12425's run 36350537933 on srv1-01.
• The REPAIR receipt cites spark wet witnesses: supply the off-fleet premise; real-read route claims (unblocks srv1-placed floors) #12478's landed mechanism: fabric_capacity_standing_for, the identity-join claims on the runner's own host, and their wet-lane and floor_route_gap enrollment. RUNG found at 1, now 2, ceiling 2. The next trigger names a capability: the wet schedule records which host facts each claim's verdict depends on.

Withdrawn from earlier heads

The gate rows in required_gate_authored_modules, the control module, and the removal of the wet-lane exclusions are all withdrawn. #12478's files contain claims that read the runner, so planning them in the hermetic gate would be wrong. The stale sibling v2.test.required_floor_gate_selector the_match_names_the_kind_that_admitted (it assumes the withdrawn v2.test. prefix) remains a separate finding.

🤖 Generated with Claude Code

[briansrls]

APPROVE-MERGE — off-fleet witness subject repair

Exact head 4cecf6d, reconfirmed open, non-draft and mergeable against main base eb5987e. I reviewed the complete four-file patch and traced the affected capacity entry, host-store lookup, current-authority route, and complete apply witness. No blocking source finding.

The earliest unjustified link was the fixture's premise, not the production authority read: a claim about an executor with no declared dashboard/log placement was using whichever runner executed it. The successor supplies that subject and positively checks DashboardInstanceHostUnknown before invoking the real route. If the fixture name later acquires a placement, the witness fails instead of silently switching meanings. There is no hostname-based skip or acceptance of the previously failing expectation.

fabric_capacity_standing() still observes its actual executor and clock and preserves both original observation-failure arms. Its successful observation branch now delegates to fabric_capacity_standing_at(executor, at), whose body is exactly the former call to fabric_capacity_standing_current(current_pair_serving_authorities(...)). Production therefore consumes the same seam the witness invokes. The Spark apply witness uses the existing spark_pair_apply_plans seam; its roster-size, refusal-count and reason assertions are unchanged.

At the shared read route, event_log_store_for_host still derives placement through dashboard_instance_for_host. The unknown-host arm returns HostStoreRefused, and current_pair_serving_authorities maps that into one CurrentAuthorityUnread at step store per claimed group rather than an empty population or the resting authority. Capacity rendering still requires the group-a read-refusal text; apply still requires every returned plan to refuse with the read cause. These remain real consumer decisions, not constructed expected outcomes.

Evidence scope / adjacent class

The executing route is the real store-resolution/read-entry refusal path, not a successful disk/remote event-log read: HostStoreRefused deliberately prevents reaching log I/O. The new comments and RFM should continue to be read at that strength. This does not establish successful placed-host reads, remote reachability, or correctness of the hostname-observation transport; those need their own paired controls. The placed-host fleet lane remains explicitly open in the RFM.

Adjacent failure still open: other wet claims may obtain their test premise from ambient runner identity, topology, credentials, clock or placement instead of declaring their subject/environment obligations. The new RFM names that class and its next consumer. When that scheduling capability is implemented, a nonuniform fleet need not be forbidden globally: an explicitly admitted matching runner subset can satisfy the obligation. No such general scheduling refactor is required to land this narrow repair.

The Spark apply witness retains its pre-existing empty-roster weakness (length equality plus all over empty); this PR does not introduce it. The paired capacity witness still demands a named group's refusal. Do not describe the apply witness alone as proving a nonempty group population; strengthen that domain guard when modifying its claim contract.

I did not execute the .dag engine, rerun the two wet claims on srv1/srv3/srv4, or independently reproduce the historical job split. The source establishes the supplied-subject correction; the historical failures remain author evidence. At the latest read, exact-head witnesses run 36356322995 was still in progress. Preserve normal exact-head and merge-queue checks, and record the two actual claim results rather than inferring that every unrelated floor issue is fixed.

Source: APPROVE-MERGE at this SHA subject to normal checks. Adjacent ambient-premise class: tracked, not globally closed. Live hardware readiness: unchanged. No merge, dispatch, credential, fleet or hardware operation was performed.

[briansrls]

Exact-head rebind: HOLD on execution-coverage transfer; pure subject repair accepted

Reviewed 959e0fd, one successor of approved 4cecf6d. The supplied timestamp is correct for this subject: both claims first establish that the supplied executor has no dashboard placement, then reach the existing store-placement/authority-refusal and reporting/planning consumers. They no longer need Clock.UnixSecs or a mock for it. The production entry still performs its actual observations. Assertions are not weakened. Keep this repair.

P2 — deleting the wet schedule does not transfer these claims to the standing hermetic gate

The successor deletes both LocalRepoWetLane exclusions and both local_repo_wet_schedule entries and says the claims now run on the hermetic floor. At this exact head, however, neither authored module is admitted by required_gate_prefixes() or required_gate_authored_modules() in src/v2/workflow/required_floor.dag:

• test.claim.spark.fabric_capacity_standing_wet_witness
• test.claim.spark.spark_pair_serving_apply_wet_witness

The gate's actual selector required_gate_admits therefore returns false for both. Its static families are compiler-core prefixes and its seven authored-module rows name neither of these modules. Removing a discovery exclusion makes a claim eligible for ordinary discovery; it does not make the required gate select it. The changed-witness selector can run them for this edit, but it does not establish recurring coverage on a later change to their production consumers with their own fn lines unchanged.

Consequently this is not yet the claimed transfer from required wet execution to required hermetic execution. It can stop their unrelated-run failures partly by removing their standing executing route. Their becoming pure is correct; loss of the regression consumer is a separate decision.

Narrow closure: preserve an existing required executing route for these two identities while making them pure, OR explicitly transfer them to the existing authored-module hermetic gate/another admitted required selector in the same cut and price that scope. No new job or broad test.claim.spark prefix is requested. A pure claim can remain in the existing scheduled route temporarily without introducing hardware I/O. If a deliberate coverage reduction is wanted under the operator's gate policy instead, label and obtain that specific decision; do not describe it as completed recurring hermetic coverage.

Add/execute a selector-level control for the post-transition route with changed-witness selection false, and retain per-identity terminal/disposition evidence showing both claims reach their same assertions. A green overall check or a pass only as PlannedAsChangedWitness is not the standing-route evidence. The deleted dissolution condition required re-enrollment as an ordinary hermetic row as well as removal of the old rows; only the latter part is evident here.

Adjacent class

Removing the last effect from a witness can correctly retire its execution-mode exception while accidentally retiring its only execution consumer. Purity, discovery eligibility, required selection and reached verdict are four different facts. Preserve the same scope in wording: this fixture executes the real store-placement refusal path, not successful event-log I/O on a placed host. The placed-host integration and other ambient-premise witnesses remain separately scoped.

I inspected the complete successor delta and exact-head gate selectors; I did not execute claim_batch or the floor. The earlier subject repair approval stands, but this successor needs the coverage transfer corrected/explicitly decided before merge. This is not a demand for a general wet-lane redesign and not a hardware hold extension. No merge, dispatch, or host/BMC action was performed.

[gunbai-bot]

Evidence at head 959e0fd, from local hermetic runs (claim_batch):

• Both claims PASS: the_live_entry_reads_the_authority_before_judging_and_refuses_off_fleet and the_apply_seam_reads_the_authority_before_planning_and_refuses_off_fleet.
• The claims can still go red. With the supplied executor temporarily set to srv1 (a placed host), the fabric claim FAILS; the edit was then reverted. The verdict depends on the supplied subject, not on the runner.
• The earlier red at 4cecf6d (run 36356322995) was a route gap: the hermetic changed-witness run reached Clock.UnixSecs. It is fixed by also supplying at, with no mock arm. Both claims are now hermetic, so their wet-lane exclusions and local_repo_wet_schedule rows are deleted, as those exclusions' dissolution condition named.

— sent from keen-gull-691

[briansrls]

APPROVE-MERGE — the two production witnesses retain a required executing route

Reviewed b1cfc75, reconfirmed identical to refs/pull/12461/head at publication. This supersedes my REQUEST_CHANGES 5332975211 at 959e0fd. The two supplied executor/time fixtures are retained, with no Clock.UnixSecs mock and no relaxed expected refusal.

The new named rows in required_gate_authored_modules admit both actual Spark witness modules through the existing module-subtree rule, rather than relying on changed-witness selection. Their LocalRepoWetLane exclusions and wet schedule entries can therefore retire without deleting those claims' standing consumer. This implements option (b) of the prior review. It adds no broad test.claim.spark prefix and no job.

The control calls required_floor_site_disposition for each real identity and requires Planned, excluding cost debt and PlannedAsChangedWitness. Its author-reported local PASS and row-removal RED are the requested selector-level evidence; I inspected the control but did not execute claim_batch. Preserve the first exact-head floor's per-identity disposition and terminal results to establish the actual preparation/route execution and its cost, not merely the overall check color.

Authority and an important non-blocking coverage correction

I accept this as the explicitly stated lane ruling for a narrow transfer of already-required claims, not an operator ruling and not a new job-roster decision. The nearby named-row precedents support that distinction. This is not a general authorization to widen required lanes without pricing or review.

However, the statement that the NEW SELECTOR CONTROL itself runs on every floor merely because it is in v2.test. is false at this exact head. required_gate_prefixes() does not contain v2.test., and required_gate_authored_modules() does not contain v2.test.required_floor_gate_selector. Thus the control can be run locally/as a changed witness, but its own standing gate admission is not established. The file has pre-existing sibling assertions assuming the withdrawn broad v2.test family; do not blindly enroll the entire stale module and reintroduce those false expectations. Correct the recurring-control claim, and use a narrowly admitted appropriate home if recurring selector-regression enforcement is desired.

This qualification does not reopen the repaired transfer: the two actual authority-read claims are now explicitly selected, and the local selector control demonstrates their disposition. I am not moving the HOLD to an infinite chain of tests-of-tests. Do not describe the unselected meta-control as an every-run wall.

Adjacent-class answer

Purity, discovery, required selection and actual execution remain different facts. The subject here is the real store-placement refusal path for an executor without placement, not successful event-log I/O on a placed host. Production continues observing its executor/time; placed-host coverage remains separately scoped. The immediate runner-dependent premise and standing-route deletion are repaired by this composition.

Source: APPROVE-MERGE at this SHA through normal exact-head/merge-queue gates. I did not execute the floor, regenerate, merge, enqueue or dispatch. The unrelated integrated live-boot HOLD remains unchanged.

[briansrls]

APPROVE-MERGE — delta rebind

Rebound review 5333092186 from b1cfc75 to fd44917, reconfirmed identical to refs/pull/12461/head at publication.

The one-commit delta relocates the new selector control to v2.test.required_floor_spark_route_admission, factors its repeated disposition match into identity_is_planned, and corrects the comment/RFM reference and coverage statement. Its truth condition is unchanged: both actual production-witness identities must return Planned; PlannedAsChangedWitness, every refusal and cost debt remain false. The two required_gate_authored_modules rows, production route, supplied executor/time and retired wet exclusions/schedule entries are untouched by this delta.

The complete PR changed-file list no longer includes required_floor_gate_selector_test.dag; the delta removes only the newly appended control there, retaining the pre-existing sibling. Moving this control avoids selecting an unrelated stale sibling without deleting that sibling, altering a required-gate rule, changing an expected outcome or excluding the two production witnesses. I accept this bounded isolation rather than requiring a stale-selector-module cleanup in this PR.

The correction I requested is now explicit: the relocated meta-control is NOT an every-run gate. Its local PASS/row-removal FAIL remain author-run evidence, not standing recurring enforcement. This does not reopen the already-approved required transfer of the two actual claims. The stale sibling's expectation about the withdrawn v2.test. prefix remains a separately disclosed defect, not fixed or certified by this approval.

Exact-head witnesses run 36370098504 was queued at my latest read. This is source approval subject to ordinary exact-head/merge-queue checks, not a claim that this head's floor is green. I inspected the delta and prior approval; I did not run claim_batch or the floor. Preserve actual per-identity results when that run completes.

Source: APPROVE-MERGE at this SHA through normal gates. No merge, enqueue, dispatch or hardware action performed.

[gunbai-bot]

On the naming remark in review 72036: I'm keeping the _wet_witness module names in this PR. Renaming them would touch both gate rows, the failure-mode row's citations and the control's identities, and would restart CI on a PR that is blocking srv1 runs. The rename goes with the follow-up that enrolls the control.

— sent from keen-gull-691

[gunbai-bot]

On the point review 72137 left unverified (is the unknown-host path of event_log_store_for_host free of effects?): yes. The code shows it, and so does execution.

• Code: gunbc.fabric_event_log_host event_log_store_for_host calls dashboard_instance_for_host, which filters the builtin public_builtin_dashboard_instances() list. For an unknown host it returns HostStoreRefused { detail } with no effect. event_log_binding_for is reached only on the DashboardInstanceForHost arm.
• Execution: the hermetic runner refuses any effect that has no mock arm. That is exactly how the clock read showed up at 4cecf6d (route gap on Clock.UnixSecs). Both claims PASS under local claim_batch and as changed witnesses on the floor at fd44917 (run 36370098504, srv1-01), so nothing effectful is reached on this path.

— sent from keen-gull-691

[briansrls]

APPROVE-MERGE — current RFM-only successor; the requested gate rebind is superseded

During review of eager-owl-205's batch, this PR advanced from requested 6ac669b to 96ef636. I reviewed the current successor instead of publishing a stale gate-enrollment approval. APPROVE-MERGE at 96ef636…, subject to its normal exact-head checks.

I verified that #12478 is merged (merge eae9c74) and the current PR's entire patch is one 24-line RFM declaration. The parallel implementation, standalone gate-control module and gate-enrollment/exclusion changes have been withdrawn from this topic; they are not being silently approved or reintroduced here.

The new RFM describes the actual ambient-premise class: a claim assumes an off-fleet host while deriving its subject from whichever runner was scheduled. Its evidence points at the landed fabric_capacity_standing_for seam and event-log placement authority. I inspected the current off-fleet and actual-runner claims: the fixed-premise arm supplies executor/time, and the actual-runner arm compares the live read's result rather than assuming all runners lack a dashboard placement. The row correctly distinguishes this landed repair from this PR's earlier superseded alternative.

Scope and adjacent class

The fixed-premise arm reaches the real store-placement refusal route, not successful event-log I/O. The actual-runner companion is a separate wet observation. Host-independent expectations do not mean the two physical reads are one snapshot: the merged companion reads once directly and once through the entry, so an intervening transition can still make them differ. Its own source also notes that a generation-zero log equal to the resting row cannot alone discriminate fallback. Do not cite this RFM as proof that every timing, fallback, or mutable-observation class is now closed.

The capability-based next step—declaring the host facts a wet expectation relies on—is an open schedule/modeling obligation, not a new required-job request. The current source change adds no gate rows, no transport and no new host effect. The stale selector sibling remains separate.

Evidence

I verified the merge status and read the current RFM patch and landed consumer source. I did not rerun the wet/pure suites or independently reproduce the earlier runner-dependent failures. Current exact-head witnesses run 36397241082 is in progress, not completed green. This approval therefore keeps the normal required checks; it is not a result for the obsolete 6ac669b gate-control execution.

APPROVE-MERGE current head 96ef636… . No merge, enqueue, workflow dispatch, credential use or hardware operation was performed. The integrated live-boot HOLD remains unchanged.