Skip to content

feat(OMN-18988): re-emit a lab-pass receipt for any sha the converged image contains whose only failures were binding-class - #3915

Closed
jonahgabriel wants to merge 6 commits into
devfrom
jonah/omn-18976-reemit-v2
Closed

jonahgabriel wants to merge 6 commits into
devfrom
jonah/omn-18976-reemit-v2

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Sep 21, 2026 •

Copy link
Copy Markdown
Collaborator

Ticket: OMN-18988, a child of OMN-18976. Option (1) per the team lead's ruling, then widened by the team lead's scope extension (the section The widening below). Stacked on the part-one PR — until that merges this diff also shows its commits; the new work is the last five commits.

What part one left

#3910 stopped a queued merge being recorded as a terminal FAIL. That left the sha with no receipt at all, which is strictly better and still not an answer: the delivery gate keeps refusing it, and its own verify run is over. Only a later run can speak for it.

The window, and why it is not ancestry

A verify run reads the lane twice: at the revision it started on, and at the one it converged to. Every merge that landed strictly between those two was queued behind this one — its own verify run found the lane still on the earlier revision and exited without a receipt. That set, and only that set, is what a convergence may answer for.

The bound is the whole point. It is emphatically not every ancestor of the converged revision: the lane has run for weeks and contains thousands of shas it never individually converged on, and emitting for those would manufacture receipts for merges nothing observed. A queue is not a history.

Every unresolved case yields an empty window — no previous revision, no converged revision, a lane that never moved, an unreadable comparison. Emitting nothing leaves the gate refusing exactly as it does today; a wrong PASS is the one outcome this must not introduce while removing a wrong FAIL.

The evidence is re-keyed, not re-taken

The converged run's probes were read against the image the lane came to run, and that image contains each of these shas — so they are precisely the reads a lab pass for that sha would have made, at the only moment it could. Re-probing here would read a newer lane state and attribute it to an older sha, which is the weaker claim and the easier one to get wrong.

A failing convergence re-emits a failure. Every sha in the window inherits what the run found. A re-emission that quietly upgraded to PASS would be the worst possible outcome of this ticket, and there is a test for exactly that direction.

Provenance, and two refusals

Each re-emitted receipt carries converged_via, so a receipt written by a sha's own run and one written by a later convergence stay tellable apart. Both are honest; they are not the same evidence. The field is written only when non-empty, on the same terms as node_inventory, so RECEIPT_VERSION is unchanged and every receipt already in flight parses byte-identically — there is a test asserting an ordinary receipt's wire shape is unchanged.

The re-key refuses two things outright: onto its own sha, which would race two artifacts of one name from one job; and from a receipt that is itself a re-emission, because provenance that chains is provenance nobody can read.

Shape

A matrix job, because the number of queued shas is discovered at run time and a uses: step cannot be looped. fromJSON over the verify job's output, and an empty array skips the job — the common case, since the lane usually converges on the very next merge.

Five gates refused the first draft, each fixed at cause

Worth listing, because every one of them was right:

gate what it caught
lab-pass receipt isolation paths not derived from a per-run variable
CI evidence policy an if-no-files-found: error uploader undeclared, and then its pre-upload assertion missing if: always()
rebuild-trigger ceiling a job with no timeout-minutes
runner routing audit a hardcoded ubuntu-latest instead of the fork-aware selector
CI workflow resilience raw setup-uv instead of the shared CI Python environment

The matrix leg keys its receipt directory by sha as well as run and attempt, because two legs sharing one directory would upload each other's receipt.

Verification

tests/ci/ and tests/unit/ci/ are green whole -- 4453 passed, 7 skipped -- with mypy --strict and ruff clean on both changed modules. The window and re-key have nineteen tests, all boundaries: two-sha queue in, converged sha excluded, outside-the-window excluded, empty window, both missing endpoints, unresolvable comparison, lane that never moved, dedup. The failing-convergence direction is among them: a FAIL convergence re-emits a FAIL. The widening adds twenty-five: the eligibility decision over every failure shape (binding-only, one health check among binding ones, deployed_revision FAIL rather than INDETERMINATE, already-passing, unreadable), and surface resolution over each of the three surfaces winning, a readable non-containing surface not vetoing a later one, and all three silent yielding no candidate.

The last two commits also re-point the two OMN-18144 tests the part-one rename moved, which splits 1 and 2 were correctly red on. Re-pointed, not relaxed: each now asserts the not-a-lane-failure property directly rather than via whichever enum member carries it.

Split 12 is red on the committed lane census aging past its 3-day threshold. Unrelated, pre-existing on dev, left alone.

The semantic, stated plainly

A re-emitted PASS for S means: the converged image containing S passed the lane's
checks at X.
It does not mean the lane was observed applying S by itself, and it
never can — S was queued behind another deploy and no run watched it land alone. That
is what a lab pass for a queued-behind merge can ever mean, and converged_via is on the
receipt so a reader gets that sentence rather than reconstructing it from timestamps.

The widening

Part one and the first draft of this PR answered for one class only: a sha that was
queued behind another deploy and therefore emitted no receipt at all. The trigger is
now wider, and the widening is what makes this the path that unblocks a cut.

A receipt can also exist and say FAIL for a reason that is not about the software. The
two checks deployed_revision and probe_generation_bound are binding-class: they
answer whether the run could bind its own observation to the sha under test, not whether
the lane was healthy. A run that could not read the lane's revision records a binding
failure and the sha is then refused delivery forever, on evidence that was never about
the code. That receipt is now eligible.

Three refusals bound it, and each is a separate test:

  • deployed_revision must be INDETERMINATE, never FAIL. A FAIL is a measured
    mismatch, the lane demonstrably running something else; an INDETERMINATE is an
    unreadable one. Treating the two alike would re-emit over a real mismatch.
  • Every failing check must be binding-class. One failing health check and the receipt
    is refused by name. A receipt whose health checks genuinely failed is never re-emitted,
    under any combination.
  • A passing receipt is left alone, and a re-emission never chains from another
    re-emission.

The binding is re-established, never inherited

The converged run may itself have failed to bind — that is the case this widening exists
for — so its answer is not evidence for anybody else. Each candidate's binding is resolved
fresh, in the verify job that already holds the lane, across three surfaces tried in
order: the container's revision label, the deploy agent's /health loaded_code_sha,
and the lane's /ready version.

The first surface that establishes containment wins. A surface that is readable but
does not contain
the candidate does not veto a later one: it is one lane fact
disagreeing, not a verdict, and a single stale label must not suppress a correct answer
the agent already has. If all three are silent the candidate is omitted and named in
the log
rather than answered for — no receipt, the gate keeps refusing, and the refusal
is visible where a reader of that run will find it. Silence is not a pass.

The surface that won is written to the receipt as converged_via, so a reader gets
which lane fact bound the sha rather than a bare assertion that something did.

Replaces #3912

Same reason as its parent: a version-trio conflict with dev that cannot be resolved
locally under the shared-clone guard (OMN-18974). Rebuilt on current dev, no version bump.

Evidence-Ticket: OMN-18988
Evidence-Source: OCC#10656

…lab pass

The verify job correctly declines to hold the single lab verify runner open for
a deploy-agent queue it cannot outlast. It recorded that decision as
INDETERMINATE, which maps onto a lab-pass check with ok false, and the receipt
verdict rule is PASS iff every check passed, so a queued merge was emitted as a
terminal FAIL and rule 24(b) refused a sha whose only fault was being second in
line.

The job already said so in its own evidence: this asserts nothing about the
lane. A check that asserts nothing about the lane and a check the lane failed
are not the same claim, and only one of them should refuse a delivery.

Measured on the .201 dev lane over two consecutive merges, each probed while
the lane still carried the PREVIOUS sha: artifact 10622202672 for 22a0ca1 with
the lane at 08db894, and artifact 10623436027 for 4277d6e with the lane at
22a0ca1. Both FAIL on this one check with every other check green. Under a
steady merge rate the queue never drains inside one verify window, so this was
the normal outcome for a busy period rather than an edge case.

The queued case becomes its own convergence outcome and emits NO receipt. That
is not an invention: EnumLabPassResult is deliberately two-valued and its own
contract already says an in-flight pass emits no receipt rather than a PENDING
one, precisely so the gate's absent branch and its not-yet-passing branch are
the same branch. A merge whose turn has not come is in flight.

Nothing opens. The sha is still refused, because absent still fails closed.
What changes is that the refusal stays recoverable by a later convergence
instead of being contradicted forever by an artifact asserting the lane
misbehaved.

The three existing outcomes are untouched and a test pins each mapping, because
collapsing any two of them is how this defect arrived. The OMN-18144 refusal
itself is unchanged: it still fires on the same queue depths, still refuses
nothing on an empty or unreadable queue, and still costs the runner zero wait.
Its own test is re-pointed rather than relaxed, and now asserts the property it
was always for, that a refused run is never a finding about the lane.

Also pinned, having been a docstring claim until now: the delivery gate sorts an
exact-name artifact query newest-first and reads only the newest. That is
load-bearing for any later receipt being able to supersede an earlier answer,
and nothing tested it.

The emit step is gated on an output the guard publishes rather than on a
mapping re-derived in a run block, and an EMPTY value still emits, because a
guard that died before writing anything has not earned the suppression.

Not in this change: re-emission for a sha that converges late. That needs a
design decision between two shapes with different blast radii, and it is raised
rather than guessed.

Ticket: OMN-18976
No version bump, deliberately. This changes CI wiring, a guard and tests, with
no package behaviour to release, and the three files a bump touches are the
only ones that ever conflict with dev here. Two branch rebuilds were spent on
that conflict tonight before dropping the bump removed it.
…erged one

Part one stopped a queued merge being recorded as a terminal FAIL. It left the
sha with no receipt at all, which is strictly better and still not an answer:
the delivery gate keeps refusing it, and its own verify run is over.

A verify run reads the lane twice, at the revision it started on and the one it
converged to. Every merge that landed strictly between those two was queued
behind this one, had its own verify run find the lane still on the earlier
revision, and exited without a receipt. That set, and only that set, is what a
convergence may answer for.

The bound is the point. It is not every ancestor of the converged revision: the
lane has run for weeks and contains thousands of shas it never individually
converged on, and emitting for those would manufacture receipts for merges
nothing observed. A queue is not a history. Every unresolved case yields an
empty window, which re-emits nothing and leaves the gate refusing exactly as it
does today, because a wrong PASS is the one outcome this must not introduce
while removing a wrong FAIL.

The evidence is re-keyed rather than re-taken, and that is deliberate. The
converged run's probes were read against the image the lane came to run, and
that image contains each of these shas, so they are the reads a lab pass for
that sha would have made at the only moment it could. Re-probing later would
read a newer lane state and attribute it to an older sha, which is the weaker
claim. A failing convergence therefore re-emits a failure: every sha in the
window inherits what the run found, and a re-emission that quietly upgraded to
PASS would be the worst possible outcome of this ticket.

Each carries converged_via, so a receipt written by a sha's own run and one
written by a later convergence stay tellable apart. Both are honest and they
are not the same evidence. The field is written only when non-empty, on the
same terms as node_inventory, so the receipt version is unchanged and every
receipt already in flight parses byte-identically.

Two refusals in the re-key itself: onto its own sha, which would race two
artifacts of one name from one job, and from a receipt that is itself a
re-emission, because provenance that chains is provenance nobody can read.

It is a matrix job because the number of queued shas is discovered at run time
and a uses: step cannot be looped. An empty array skips the job, which is the
common case.

Five CI gates refused the first draft and each was fixed at cause rather than
worked around: the per-run receipt directory, the evidence-policy declaration
with its pre-upload assertion under always(), the job ceiling, the fork-aware
runner selector and the shared CI Python environment. The matrix leg keys its
directory by sha as well as run and attempt, because two legs sharing one would
upload each other's receipt.

Ticket: OMN-18976
THE SEMANTIC, PLAINLY. A re-emitted PASS for S means: the converged image
containing S passed the lane checks at X. It does not mean the lane was
observed applying S by itself, and it never can, because S was queued behind
another deploy and no run watched it land alone. That is what a lab pass for a
queued-behind merge can ever mean, and converged_via is on the receipt so a
reader gets that sentence rather than having to reconstruct it.

Rebuilt on current dev with no version bump. The three files a bump touches are
the only ones that ever conflict here, and two branch rebuilds were spent on
that conflict before dropping it.

222 tests pass on the rebuilt base.
@jonahgabriel
jonahgabriel enabled auto-merge (squash) September 21, 2026 10:44
@jonahgabriel jonahgabriel changed the title feat(OMN-18976): answer for the merges that were queued behind a converged one feat(OMN-18988): answer for the merges that were queued behind a converged one Sep 21, 2026
…tter

The advisory-job gate refused the new matrix job, correctly and for a reason
that is this ticket's own argument turned on me: a check that runs and cannot
fail reads as coverage.

Its verdict was that reemit-queued-receipts is in neither required_status_checks
nor the external-context tuple, so it can fail nothing. That is true and it is
the right classification, because the job is not a verification at all. It runs
AFTER the merge and after the lane converged, and it writes evidence rather
than judging any.

Enforcing it would be wrong rather than merely awkward: it would gate a pull
request on a rebuild that has not happened when that pull request is reviewed.
The sibling verify job in this same workflow is post-merge for exactly that
reason.

The output IS gating, which is the part worth being clear about. The delivery
gate refuses a sha whose receipt is absent or non-PASS, so a failure here
withholds a receipt and the sha stays refused. The consequence lands on the
delivery attempt, not on this job.

Ticket: OMN-18988
The annotation was eight lines above the job key, inside an explanatory
comment block. The gate accepts it on the job line itself or on the single
comment line DIRECTLY above, and nowhere else, for a reason worth keeping: an
annotation separated from what it excuses drifts onto the wrong key the next
time someone edits between them.

So the marker is now the last comment line before the key, with the reasoning
left above it where a reader wants it. Verified by running the gate against
this tree rather than by re-reading the rule: PASS, no unannotated advisory
setting and no unenforced verification job.

Ticket: OMN-18988
…tself

The re-emission trigger widens from the absence of a receipt to the CLASS of
its failure. This commit is the decision core; the wiring follows.

Why it widens. The 430ff34 receipt failed on deployed_revision INDETERMINATE
with commands_ahead ZERO, because the agent answered 404 for the run own
correlation id and the lane budget never started, plus probe_generation_bound,
because the probe read a different container generation than convergence
verified. The lane had converged and was running the sha. Neither says anything
about the lane, and frozen as FAIL both are inherited by every merge behind
them, so waiting can never unblock a cut.

is_binding_only_failure draws the line this feature lives or dies on: a receipt
whose health checks genuinely failed is NEVER re-emitted, because turning a
real finding into a PASS is strictly worse than the wrong FAIL being removed.
One non-binding failure poisons the whole receipt; there is no partial
re-emission.

The discriminator worth naming: deployed_revision counts only with outcome
INDETERMINATE. The same check with outcome FAIL is the lane having had its
whole budget and not converged, which is a statement about the lane. Keying on
the check name alone would collapse those two, and that is the easiest way to
get this wrong.

resolve_lane_binding re-establishes the binding rather than inheriting it. The
converged run could not bind its own observation, which is why its receipt
failed, so copying that binding would copy the defect. Three surfaces are read
in descending directness: the container revision label, the agent recorded
loaded_code_sha, the runtime ready version. The first to establish containment
wins and names itself in converged_via.

A surface that merely ANSWERS does not veto the others, deliberately: a
container label can lag a restart while the agent has already recorded the new
code, so a readable-but-non-containing surface must not make the common case
unbindable. All three silent is a refusal, and so is all three answering
without containment. Nothing bound means nothing claimed.

18 tests, every one a boundary between the two classes or between bound and
unbound. mypy and ruff clean.

Ticket: OMN-18988
onexbot-occ-writer Bot pushed a commit to OmniNode-ai/onex_change_control that referenced this pull request Sep 21, 2026
…fra#3915

OCC companion by node_pr_lifecycle_fix_effect (OMN-13317 F1 / OMN-13990 / OMN-14285). Product PR head 25870acd0f3a4c49ba72dd82b6f6bca6cb622f5b.
jonahgabriel pushed a commit to OmniNode-ai/onex_change_control that referenced this pull request Sep 21, 2026
#10656)

* evidence(OMN-18988): author OCC companion for OmniNode-ai/omnibase_infra#3915

OCC companion by node_pr_lifecycle_fix_effect (OMN-13317 F1 / OMN-13990 / OMN-14285). Product PR head 25870acd0f3a4c49ba72dd82b6f6bca6cb622f5b.

* evidence(OMN-18988): self-bind OCC#10656 + rebind contract_sha256

* evidence(OMN-18921): mint the owed OCC self-bind for PR #10656

---------

Co-authored-by: omnimarket-bot <bot@omninode.ai>
Co-authored-by: onexbot-occ-writer[bot] <onexbot-occ-writer[bot]@users.noreply.github.com>
…ing-class

A receipt that FAILED because the run could not bind its observation to the
sha is recording something about the RUN, not about the lane. Freezing it as
terminal refuses delivery of a sha forever on evidence that was never about
the software. The trigger now widens from "queued, so no receipt" to "no
receipt, or a receipt whose failing checks are all binding-class and whose
deployed_revision is INDETERMINATE rather than FAIL".

A receipt whose health checks genuinely failed is NEVER re-emitted: the
eligibility decision refuses it by name, and a FAIL deployed_revision is a
measured mismatch rather than an unbindable read, so it is refused too.

The binding is RE-ESTABLISHED rather than inherited, in the verify job that
already holds the lane, across three surfaces in order -- the container's
revision label, the deploy agent's /health loaded_code_sha, and the lane's
/ready version. The first to establish containment wins; a surface that is
readable but does not contain the candidate does not veto a later one, and
all three silent means the candidate is omitted and named in the log rather
than answered for. The surface that won is recorded on the receipt as
converged_via, so a re-emitted receipt is tellable apart from one the sha's
own run wrote, and a reader can see WHICH lane fact bound it.

The evidence-policy entry is re-keyed onto matrix.candidate.sha, which is
where the uploader actually writes.
@jonahgabriel jonahgabriel changed the title feat(OMN-18988): answer for the merges that were queued behind a converged one feat(OMN-18988): re-emit a lab-pass receipt for any sha the converged image contains whose only failures were binding-class Sep 21, 2026
@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Hostile Reviewer — DEGRADED (informational)

Critical findings: 0
Major findings: 0
Total findings: 0
Models succeeded: glm-review

Note: Fewer than 2 reviewer models succeeded. Degraded results are informational (OMN-8468/OMN-8524) and do not block merge. Error: cli_review exit 2 (fewer than 2 models succeeded — partial/total outage)


Semantics (OMN-17492 — the model finds, thread resolution gates)

Surface Meaning Blocks merge?
Review threads Per-finding, posted by the reviewer No (informational)
Hostile Review Thread Gate Deterministic: unresolved hostile-reviewer threads exist Fails until resolved (not yet a required context)
degraded verdict Fewer than 2 models succeeded (infra) No

Powered by omniintelligence.review_pairing.cli_review — multi-model adversarial review: qwen3-review, qwen3-review-b, glm-review (OMN-8468/OMN-8524/OMN-17492)

@jonahgabriel

Copy link
Copy Markdown
Collaborator Author

Superseded by #3921. Part one landed as a squash (e5af728ee, #3914), which re-minted the commits this branch was stacked on, so it conflicted with dev on check_dev_lane_staleness.py and could not be merged or rebased in place -- the shared-clone guard refuses both inside a worktree (OMN-18974). #3921 is the same change re-applied onto current dev, carrying OMN-18990's queue-lag staleness arm in the same module untouched. No work is lost and nothing is relaxed.

auto-merge was automatically disabled September 21, 2026 12:08

Pull request was closed

jonahgabriel added a commit that referenced this pull request Sep 21, 2026
… image contains whose only failures were binding-class (#3922)

* feat(OMN-18988): answer for any sha the converged image contains whose only failures were binding-class

Part one (OMN-18976, #3914) stopped a queued merge being recorded as a
terminal FAIL, which left the sha with no receipt at all: strictly better,
still not an answer. A later run has to speak for it, and this is that run.

THE WINDOW, and why it is not ancestry. A verify run reads the lane twice,
at the revision it started on and at the one it converged to. Every merge
strictly between those two was queued behind this one. That set, and only
that set, is what a convergence may answer for -- emphatically not every
ancestor of the converged revision, because the lane contains thousands of
shas it never individually converged on. A queue is not a history. Every
unresolved case yields an empty window, because a wrong PASS is the one
outcome this must not introduce while removing a wrong FAIL.

THE WIDENING. A receipt can also exist and say FAIL for a reason that is
not about the software. deployed_revision and probe_generation_bound are
binding-class: they answer whether the run could bind its own observation
to the sha, not whether the lane was healthy. Such a receipt is now
eligible, under three refusals, each with its own test: deployed_revision
must be INDETERMINATE and never FAIL, because a FAIL is a measured
mismatch and an INDETERMINATE an unreadable one; every failing check must
be binding-class, so one failing health check refuses the whole receipt;
and a passing receipt is left alone.

THE BINDING IS RE-ESTABLISHED, never inherited. The converged run may
itself have failed to bind -- that is the case this exists for -- so its
answer is not evidence for anybody else. Each candidate is resolved fresh,
in the verify job that already holds the lane, across the container
revision label, the agent's /health loaded_code_sha and the lane's /ready
version, in that order. First to establish containment wins. A surface
readable but not containing the candidate does not veto a later one: that
is one lane fact disagreeing, not a verdict. All three silent means the
candidate is omitted and named in the log, and no receipt is written --
silence is not a pass. The winning surface is recorded as converged_via,
so a reader gets which lane fact bound the sha.

The evidence is re-keyed, not re-taken: the converged run's probes were
read against the image that contains each of these shas, so they are the
reads a lab pass for that sha would have made at the only moment it could.
A failing convergence re-emits a failure.

Rebuilt on dev after #3914 squashed; carries OMN-18990's queue-lag staleness
arm untouched. Replaces #3915.

* test(OMN-18988): state the LAN-bound runner guard per job, not per file

OMN-17888 guards one property: a job that needs the tailnet fleet must not
resolve its placement from the shared trusted-CI seam, which org scope
retargeted to hosted compute on 2026-08-26. It asserted that property as a
substring search of the whole workflow file, which was exact while the file
held a single job with a selector.

This branch adds a second. The re-emitter downloads two artifacts, re-keys
one and uploads it. It touches no lane, so the shared hosted seam is the
correct selector for it, and the file-wide search read that as the incident
it was written for.

The publisher assertions now read that job own runs-on, and a companion
test walks every job whose placement resolves to self-hosted and refuses the
shared seam on each. Three jobs are in that census today, the publisher and
both verify legs, and the census is asserted non-empty so the guard cannot
pass vacuously. Reintroducing the OMN-17888 defect on the publisher is caught
by name; the hosted re-emitter is not.

Claude-Session: https://claude.ai/code/session_011WaS5jrEdJDstEEtiJuRvW
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant