Repository navigation
Falsifier alert describes the CURRENT failure: body rewritten per window, class transition bypasses the throttle - #7826
Merged
Merged
Conversation
added 3 commits
August 5, 2026 01:36
…s First red Found by executing readState against the REAL #7737 body rather than a fixture: its body has no state marker, so first_red parsed to null and the rewrite would have labelled the current run 'First red' — asserting something false in the very body this change exists to keep honest. The legacy rendered 'First red: ...' line is the only record of when an incident started, so it is now recovered from it. If neither the marker nor the legacy line is present on an existing issue, the first red is reported as not recorded; it is never back-filled with the current run. A genuinely new issue still records this run, which is correct. Verified by execution against #7737's actual body (recovers run 30809682025) and by mutation: deleting the recovery arm reds 2 controls.
royal-raven's #7818 makes an enrolled-but-unreached component keep its planned identity, so the affected-set cold control now resolves to present/unverified with a Skipped outcome instead of collapsing to state=absent. This arm labelled every present-but-unverified control 'RED', which would then report a control that never ran as a failing one — a state-space conflation with different remedies (restore the enrollment vs fix the control). Both arms still say selection runs UNVERIFIED, so fail-closed is unchanged; only the label distinguishes them. Also gives 'not_concluded' a phrase instead of rendering the raw tag. Mutation control: forcing didNotRun=false reds the discriminating assertion.
…e body lied still-bat-561 verified the alert MECHANISM is not broken — #7737's 2026-08-04T12:31 heartbeat comment does name the current failing set, at the promised at-most-daily cadence. The previous wording said recovering the real failing set meant downloading a run log and grepping it, which overstated the defect: the set was in the issue, in a comment. What is actually wrong is narrower and worth stating exactly, since a receipt that overstates its own incident is the same class of defect as the stale body: the title and body — what a reader meets first — describe a resolved failure, the current one sits in a comment up to 20h old below the fold, and the class transition announced nothing because BudgetExceeded -> WitnessRed was absorbed into an unchanged Still red. No behavioural change; this corrects the justification only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
Issue #7737 has been open since 2026-08-03 with a body describing a
BudgetExceededkill onwitness_committed_is_fixed_point(5004ms against a 5000ms budget). That failurestopped occurring at 2026-08-03T17:53. For the seven windows since, the falsifier has
been failing
mode=WitnessRedon entirely different witnesses.To be precise about what is and is not broken (correction from still-bat-561, verified
against the issue): the alerting mechanism works. The 2026-08-04T12:31 heartbeat comment
does name the current failing set, at the at-most-daily cadence the workflow promises. This
is not missed alerting, and an earlier draft of this description said the current
witnesses appear nowhere in the issue — that was wrong.
What actually fails:
failure. The current one is only in a comment that may be up to 20h old, below the fold.
BudgetExceeded→WitnessRedis a differentincident with a different remedy, and it was absorbed into an unchanged "Still red".
The alert was not missing and not wrong when written — it was true-at-creation and
unmaintained, which is worse, because an open issue reads as an accurate live description
of the current failure.
Root cause, located
falsifier-alert.ymlcomputed atitleand abodyon every red window but, when an issuealready existed, applied neither. The only write on the sustained-red path was a
heartbeat comment throttled to >20h. So the body froze at first red permanently.
The same throttle also swallowed the load-bearing signal: a red that changes class is a
different incident from a red that persists (different remedies — re-basis a dated ceiling
vs. fix a witness vs. repair a runner), but a class transition went through the identical
20h rate limit and was suppressed as "still red".
Change
First red:is preserved verbatim as asnapshot;
Current:plus the failing set are re-derived each window. Body edits do notnotify, so this costs no noise and is deliberately not rate-limited.
stays throttled; a transition comments immediately naming previous → current.
@mentionis minted from a string in this file —no declared roster maps a falsifier lane to an owner, so the alert routes to nobody
rather than guessing, and says so. Carries a 🟡 dissolve-on for the grounding carrier.
The signature derives from identities the receipt already emits — component
index,label,and
outcometag — never from rendered prose. BecauseBudgetExceededmaps toRefusedandWitnessRed/Inframap toFailed, Class A vs Class B is already distinguishable withno seed change. An unreadable receipt gets its own
receipt-unavailable:<cause>signature andis never folded into a known failing set.
Prior state rides in an HTML-comment marker in the issue body, so the issue remains the single
authority for its own state and no external store is introduced. A body with no marker (#7737,
opened by the previous version) parses to an explicit
null— which compares unequal to everyreal signature, so the first red window after this lands rewrites #7737's stale body rather
than silently keeping it.
One defect this found in itself
Checking that claim against the real #7737 body (not a fixture) surfaced a bug in the first
version of this change. #7737 has no state marker, so
first_redparsed tonulland therewrite would have stamped the current run as
First red:— asserting something false in thevery body this PR exists to keep honest. The true first red (run
30809682025, 2026-08-03) wassitting in the legacy rendered line all along.
Fixed: the legacy
First red: ...line is now recovered when no marker exists. If neither ispresent on an existing issue, the first red reads
not recorded (issue predates state tracking)— never back-filled with the current run. A genuinely new issue still records the current run,
which is correct. Verified by executing
readStateagainst #7737's actual body (recovers30809682025), plus a mutation dropping the recovery arm, which reds 2 controls.Execution evidence
Not a typecheck-and-ship. The pure decision logic was extracted from the shipped workflow
bytes (not a re-typed copy) and executed against receipt-shaped fixtures and the real #7737 issue body — 22
assertions, all green. The renderer assertions drive the real
componentReport, not a copy:Green alone proves nothing, so each control was verified to red when the behavior is
wrong, by mutating the shipped source:
outcometag (class transition invisible)First red:)REDlabelSource restored and re-verified
ALL GREENafter each. Also checked: YAML block-scalarindentation uniform (no tabs), and the full extracted script passes
node --check.A latent conflation #7818 would have activated
royal-raven's #7818 makes an enrolled-but-unreached component keep its planned identity, so the
affected-set cold control now resolves to
present/verified:falsewith a Skipped outcomeinstead of collapsing to
state: absent. This file'spresent-but-unverified arm labelled everysuch control RED — so once #7818 lands, a control that merely never ran would have been
reported as a failing one. Different remedies (restore the enrollment vs. fix the control).
Neither change causes that alone; it appears only when #7818 lands under this file's unchanged
branch. Fixed here:
Skipped/not_concludedreads DID NOT RUN, a genuinely failed controlstill reads RED, and both still state that selection runs UNVERIFIED — fail-closed is
unchanged, only the label distinguishes them.
Honest bounds
bytes, but this repository has no harness that runs workflow JS, and I did not want to
plant an unenrolled test that rots (nor to gate the alerting path on a self-check, which
would turn a probe failure into an alert outage — the alerter is the safety mechanism).
The mutation receipt above is authoring-time evidence, not an enrolled control. A follow-up
could run the probe as a step after the alert step in this same workflow, so a break is
visible without suppressing alerts; that needs a checkout this job does not currently do,
so it is deliberately out of scope here.
InfravsWitnessRedis still indistinguishable at row grain — both map toFailed{error, output}infloor_component_outcome_of_failure_mode, discarding the mode.So the signature separates Class A from Class B but not a real regression from a broken
runner. royal-raven's F1-transport: row-grain failure_mode edge beside ObservationOutcome #7827 adds a
failure_modefield additively; this PR deliberately doesnot depend on that unmerged stack, since this is the change that unfreezes falsifier is RED — 9 of 9 components not Done #7737. Follow-up
extends the signature to
index:label:outcome:failure_mode, which upgrades a class transitionto the real three-way and makes an
Infra→WitnessRedflip bypass the throttle instead ofreading as unchanged red. Schema stays
v1(additive field; a bump would blind every deployedalerter, which validates schema by strict equality and refuses on mismatch).