Skip to content

Falsifier alert describes the CURRENT failure: body rewritten per window, class transition bypasses the throttle - #7826

Merged
briansrls merged 4 commits into
mainfrom
session/zesty-fox-436-alert
Aug 5, 2026
Merged

briansrls merged 4 commits into
mainfrom
session/zesty-fox-436-alert

Conversation

@gunbai-bot

@gunbai-bot gunbai-bot Bot commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

What this fixes

Issue #7737 has been open since 2026-08-03 with a body describing a BudgetExceeded kill on
witness_committed_is_fixed_point (5004ms against a 5000ms budget). That failure
stopped occurring at 2026-08-03T17:53. For the seven windows since, the falsifier has
been failing mode=WitnessRed on entirely different witnesses.

To be precise about what is and is not broken (correction from still-bat-561, verified
against the issue): the alerting mechanism works. The 2026-08-04T12:31 heartbeat comment
does name the current failing set, at the at-most-daily cadence the workflow promises. This
is not missed alerting, and an earlier draft of this description said the current
witnesses appear nowhere in the issue — that was wrong.

What actually fails:

  1. The title and body — the two things a reader meets first — describe a resolved
    failure. The current one is only in a comment that may be up to 20h old, below the fold.
  2. The class transition announced nothing. BudgetExceeded → WitnessRed is a different
    incident with a different remedy, and it was absorbed into an unchanged "Still red".

The alert was not missing and not wrong when written — it was true-at-creation and
unmaintained
, which is worse, because an open issue reads as an accurate live description
of the current failure.

Root cause, located

falsifier-alert.yml computed a title and a body on every red window but, when an issue
already existed, applied neither. The only write on the sustained-red path was a
heartbeat comment throttled to >20h. So the body froze at first red permanently.

The same throttle also swallowed the load-bearing signal: a red that changes class is a
different incident from a red that persists (different remedies — re-basis a dated ceiling
vs. fix a witness vs. repair a runner), but a class transition went through the identical
20h rate limit and was suppressed as "still red".

Change

  1. The body is rewritten every red window. First red: is preserved verbatim as a
    snapshot; Current: plus the failing set are re-derived each window. Body edits do not
    notify, so this costs no noise and is deliberately not rate-limited.
  2. The computed title is applied on update, not just on create.
  3. A failing-set / failure-class signature bypasses the comment throttle. Unchanged red
    stays throttled; a transition comments immediately naming previous → current.
  4. Owner routing is a typed refusal. No @mention is minted from a string in this file —
    no declared roster maps a falsifier lane to an owner, so the alert routes to nobody
    rather than guessing, and says so. Carries a 🟡 dissolve-on for the grounding carrier.

The signature derives from identities the receipt already emits — component index, label,
and outcome tag — never from rendered prose. Because BudgetExceeded maps to Refused and
WitnessRed/Infra map to Failed, Class A vs Class B is already distinguishable with
no seed change. An unreadable receipt gets its own receipt-unavailable:<cause> signature and
is never folded into a known failing set.

Prior state rides in an HTML-comment marker in the issue body, so the issue remains the single
authority for its own state and no external store is introduced. A body with no marker (#7737,
opened by the previous version) parses to an explicit null — which compares unequal to every
real signature, so the first red window after this lands rewrites #7737's stale body rather
than silently keeping it.

One defect this found in itself

Checking that claim against the real #7737 body (not a fixture) surfaced a bug in the first
version of this change. #7737 has no state marker, so first_red parsed to null and the
rewrite would have stamped the current run as First red: — asserting something false in the
very body this PR exists to keep honest. The true first red (run 30809682025, 2026-08-03) was
sitting in the legacy rendered line all along.

Fixed: the legacy First red: ... line is now recovered when no marker exists. If neither is
present on an existing issue, the first red reads not recorded (issue predates state tracking)
— never back-filled with the current run. A genuinely new issue still records the current run,
which is correct. Verified by executing readState against #7737's actual body (recovers
30809682025), plus a mutation dropping the recovery arm, which reds 2 controls.

Execution evidence

Not a typecheck-and-ship. The pure decision logic was extracted from the shipped workflow
bytes
(not a re-typed copy) and executed against receipt-shaped fixtures and the real #7737 issue body — 22
assertions, all green. The renderer assertions drive the real componentReport, not a copy:

green DISCRIMINATING: BudgetExceeded->WitnessRed changes signature
green DISCRIMINATING: added failing component changes signature
green DISCRIMINATING: missing receipt != clean run
green DISCRIMINATING: unknown prior != current => transition fires
green signature is order-independent
green unparseable marker fails to unknown, not to a fabricated value
green DISCRIMINATING: real #7737 first red is RECOVERED, not fabricated
green DISCRIMINATING: skipped control labelled DID NOT RUN, not RED
... (22 total)  ALL GREEN

Green alone proves nothing, so each control was verified to red when the behavior is
wrong
, by mutating the shipped source:

mutation probe result
signature drops the outcome tag (class transition invisible) 3 RED
unreadable receipt folded into the clean state 2 RED
unparseable marker fabricates a value instead of unknown 1 RED
legacy first-red recovery removed (would fabricate First red:) 2 RED
skipped control forced back to the RED label 1 RED

Source restored and re-verified ALL GREEN after each. Also checked: YAML block-scalar
indentation uniform (no tabs), and the full extracted script passes node --check.

A latent conflation #7818 would have activated

royal-raven's #7818 makes an enrolled-but-unreached component keep its planned identity, so the
affected-set cold control now resolves to present/verified:false with a Skipped outcome
instead of collapsing to state: absent. This file's present-but-unverified arm labelled every
such control RED — so once #7818 lands, a control that merely never ran would have been
reported as a failing one. Different remedies (restore the enrollment vs. fix the control).

Neither change causes that alone; it appears only when #7818 lands under this file's unchanged
branch. Fixed here: Skipped/not_concluded reads DID NOT RUN, a genuinely failed control
still reads RED, and both still state that selection runs UNVERIFIED — fail-closed is
unchanged, only the label distinguishes them.

Honest bounds

  • No persistent regression control ships with this. The probe above executes the real
    bytes, but this repository has no harness that runs workflow JS, and I did not want to
    plant an unenrolled test that rots (nor to gate the alerting path on a self-check, which
    would turn a probe failure into an alert outage — the alerter is the safety mechanism).
    The mutation receipt above is authoring-time evidence, not an enrolled control. A follow-up
    could run the probe as a step after the alert step in this same workflow, so a break is
    visible without suppressing alerts; that needs a checkout this job does not currently do,
    so it is deliberately out of scope here.
  • Infra vs WitnessRed is still indistinguishable at row grain — both map to
    Failed{error, output} in floor_component_outcome_of_failure_mode, discarding the mode.
    So the signature separates Class A from Class B but not a real regression from a broken
    runner. royal-raven's F1-transport: row-grain failure_mode edge beside ObservationOutcome #7827 adds a failure_mode field additively; this PR deliberately does
    not depend on that unmerged stack, since this is the change that unfreezes falsifier is RED — 9 of 9 components not Done #7737. Follow-up
    extends the signature to index:label:outcome:failure_mode, which upgrades a class transition
    to the real three-way and makes an Infra→WitnessRed flip bypass the throttle instead of
    reading as unchanged red. Schema stays v1 (additive field; a bump would blind every deployed
    alerter, which validates schema by strict equality and refuses on mismatch).
  • This does not change the receipt read path, which is correct as-is.
  • Disjoint from Falsifier lane attribution: one enrollment authority, run standing that cannot green #7815 and Falsifier totality receipt: unreached components keep their planned identity #7818 — no merge-order dependency in either direction.

@gunbai-bot gunbai-bot Bot changed the title F1: Falsifier totality receipt + oncall router Falsifier alert describes the CURRENT failure: body rewritten per window, class transition bypasses the throttle Aug 5, 2026
@gunbai-bot
gunbai-bot Bot marked this pull request as ready for review August 5, 2026 01:26
gunbc-ci-auto-heal added 3 commits August 5, 2026 01:36
…s First red

Found by executing readState against the REAL #7737 body rather than a fixture:
its body has no state marker, so first_red parsed to null and the rewrite would
have labelled the current run 'First red' — asserting something false in the very
body this change exists to keep honest.

The legacy rendered 'First red: ...' line is the only record of when an incident
started, so it is now recovered from it. If neither the marker nor the legacy line
is present on an existing issue, the first red is reported as not recorded; it is
never back-filled with the current run. A genuinely new issue still records this
run, which is correct.

Verified by execution against #7737's actual body (recovers run 30809682025) and
by mutation: deleting the recovery arm reds 2 controls.
royal-raven's #7818 makes an enrolled-but-unreached component keep its planned
identity, so the affected-set cold control now resolves to present/unverified with
a Skipped outcome instead of collapsing to state=absent. This arm labelled every
present-but-unverified control 'RED', which would then report a control that never
ran as a failing one — a state-space conflation with different remedies (restore
the enrollment vs fix the control).

Both arms still say selection runs UNVERIFIED, so fail-closed is unchanged; only
the label distinguishes them. Also gives 'not_concluded' a phrase instead of
rendering the raw tag.

Mutation control: forcing didNotRun=false reds the discriminating assertion.
…e body lied

still-bat-561 verified the alert MECHANISM is not broken — #7737's 2026-08-04T12:31
heartbeat comment does name the current failing set, at the promised at-most-daily
cadence. The previous wording said recovering the real failing set meant downloading
a run log and grepping it, which overstated the defect: the set was in the issue,
in a comment.

What is actually wrong is narrower and worth stating exactly, since a receipt that
overstates its own incident is the same class of defect as the stale body: the title
and body — what a reader meets first — describe a resolved failure, the current one
sits in a comment up to 20h old below the fold, and the class transition announced
nothing because BudgetExceeded -> WitnessRed was absorbed into an unchanged Still red.

No behavioural change; this corrects the justification only.
@briansrls
briansrls merged commit 9bd0dda into main Aug 5, 2026
5 checks passed
@briansrls
briansrls deleted the session/zesty-fox-436-alert branch August 5, 2026 05:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant