Skip to content

fix(OMN-16964): audit delegation chain topics - #3551

Closed
Patel230 wants to merge 215 commits into
devfrom
lakshman/omn-16964-ledger-audit-topics
Closed

Patel230 wants to merge 215 commits into
devfrom
lakshman/omn-16964-ledger-audit-topics

Conversation

@Patel230

@Patel230 Patel230 commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fix the delegation-chain audit projection so the scheduled AC1 canary can observe every required ledger hop instead of deterministically reporting an empty chain.

The chain writer already queries public.event_ledger for four exact topics, but node_ledger_projection_compute did not subscribe to those topics. This PR closes that topology gap.

Changes

  • Subscribe the ledger projection compute node to:
    • onex.cmd.omniclaude.delegate-skill.v1
    • onex.cmd.omniclaude.delegation-routing-request.v1
    • onex.evt.omniclaude.routing-decision.v1
    • onex.evt.omniclaude.delegate-skill-completed.v1
  • Add one exact topic_match route for each topic.
  • Add a regression test proving every writer topology hop is both subscribed and uniquely routed by the audit projection.
  • Update the projection contract count/version and regenerate affected topic enums.
  • Document the three cross-repo publishers in the subscribe-wiring health allowlist.

Test plan

  • Local narrow suite: 90 passed
  • Lab .201 broader targeted suite: 25 passed
  • Full pre-commit suite: passed
  • Pre-push check-only hooks: passed

The live scheduled AC1 canary must remain unbound until this change is merged, deployed to the dev lane, and a scheduled run produces the complete four-hop chain.

Runtime startup gate

  • N/A — this PR does not touch auto_wiring/, service_kernel.py, handler __init__ signatures, or kernel-level registration.

Type safety checklist

  • No new metadata["key"] or metadata.get("key") string literal access on Pydantic model fields
  • No new metadata: dict[str, Any] fields without TypedDict or # ONEX_EXCLUDE: comment
  • No new bare except Exception
  • No metadata-dict keys added
  • No service added to docker-compose.infra.yml

Strict-gate ordering

  • No — not a gate PR, skip this section.

Related issues

  • Linear: OMN-16964

Evidence-Source: OCC#9588
Evidence-Ticket: OMN-16964

jonahgabriel and others added 30 commits September 10, 2026 11:42
…lare a binding (#3397)

Follow-up to omnibase_infra#3390, which squash-merged as 8e3cde8 while
this commit was still in the governed pre-push. The rule change landed there;
this is the fixture that exercises its other branch.

OMN-16493 (closer run 34443923859, 2026-09-10T06:08:34Z) is not OMN-17478's
shape. Its lane authored two structural self-binds carrying binds_ac, each
with a derived ::pr-live-state twin, so four non-probative records land in
the coverage denominator against six verified. It clears by two. A fixture
built only on OMN-17478's shape, where every provenance item is unbound and
the denominator is empty, never exercises that branch. The test asserts the
denominator is exactly 4, not only the decision: a change that released this
by emptying the corpus rather than scoping it would pass a decision-only
test and would have reopened the padding loophole.

The second test states the carry-forward rule as an assertion: removing the
four bindings empties the denominator, so declaring a criterion on a
structural self-bind only ever moves records INTO it. Never put binds_ac on
a self-bind; autobind is right to omit it, and a merged contract cannot be
corrected because the append-only gate is the authority.
…ary link 5 at it by name (#3381)

* feat(OMN-16964): create the ledger_chain relation and point link 5 at it by name

OMN-16025 link 5 reads "complete ledger chain + replay green through an HONEST
tier-2 verifier (SKIP != PASS)". The READER for that link merged in #3072 and
called ledger_chain. That relation exists in no database on any lane, so every
run since link 2 was closed has reported ledger_replay_not_configured at 4 of
5 links (34355201941, 34356615155).

Two changes, both on the consumer side of the leg.

1. Migration 105 creates ledger_chain in the omnibase_infra database, beside
   its two nearest neighbours 088 overseer_tick_ledger and 089
   verification_receipt_ledger, which are verification ledgers in that same
   flat set. Five columns are a contract with the merged reader and are
   commented as such. verifier_verdict carries a CHECK pinning it to pass /
   fail / skip, so a column that cannot represent SKIP can never force a
   writer to lie. There is deliberately no payload column. The migration also
   grants chain_canary_reader column-scoped SELECT on exactly the five columns
   the reader selects, guarded on role existence with a NOTICE, matching the
   contract already documented against CHAIN_CANARY_READER_PASSWORD in
   docker-compose.infra.yml. No credential is minted or rotated.

2. The leg's source becomes a variable NAME. #3072 shipped a raw-DSN
   ledger_source field, which is the exposure OMN-18060 removed from the
   sibling projection leg: onex skill serialises every arg into the node
   payload, onto the bus and into the event log, and the flag lands in argv.
   ledger_source_env replaces it outright with no shim, because no CLI row
   ever set the old field and so no caller can break. The model refuses a
   value that parses as a connection string, and the handler refuses a DSN
   that arrived on any other flag, reporting a new REFUSED status that renders
   as a non-passing link 5 rather than as an error.

The lane declaration is read through a new load_lane_ledger_readback, which
shares one core with its projection sibling so a refusal cannot be lost from
one copy. Its lane set and its block key are separate constants: widening one
leg's scope must be a deliberate edit to that leg.

expected_ledger_hops now gets the same comma-splitting as the terminal topic
fields, because the workflow passes it as one scalar flag; without it the
declared hops would arrive as one string matching nothing and a complete chain
would report CHAIN_INCOMPLETE.

Effect on the canary today: the relation is real and empty, so link 5 moves
from "no source configured" to CHAIN_INCOMPLETE naming the hops it did not
find. That is a changed, falsifiable status reached through a real connection,
and it stays red until the writer lands.

Ordinal 105, not 104: 104 is BURNED. OMN-17923 retired
104_create_validator_ro_role.sql and its record requires the migration to come
back as a new number, never as 104. This is the first new forward migration
since that retirement, which surfaced a conflation in the retirement gate: its
byte-identity test fingerprinted the WHOLE forward directory, so it asserted
not only "the retirement removed 104 and nothing in 001..103 changed" -- the
invariant it owns and its docstring states -- but also "no forward migration
may ever land again", which it never claimed and which its sibling test
deliberately allows for by bounding itself to ordinal <= 104. The assertion is
now scoped to ordinals at or below the retired one, via a new optional include
predicate on compute_migration_fingerprint. Same algorithm, same stamped
digest, same file count: the retirement proof is unchanged and now stays true
as the stream grows. The artifact is restamped, 88 -> 89 files, which the
unscoped sibling test enforces.

docker-compose.infra.yml's CHAIN_CANARY_READER_PASSWORD comment named
"migration 104" as the file granting that role. It is corrected: 104 is a
burned ordinal, 105 issues the link-5 grant, and the link-2 grant it described
is issued by no migration at all (OMN-18115).

RED first. 6 of 7 canary-side tests and 3 of 4 lane-loader tests failed before
the implementation. 1072 tests green after across the db and canary suites,
mypy --strict clean.

* fix(OMN-16964): schema-qualify every relation reference in migration 105

The Application Database Domain Enforcement gate (OMN-15361, ci.yml step
"Enforce schema qualification in changed SQL") rejected the migration. It
inspects only the DIFF, so the unqualified references in the older files of
this set are grandfathered rather than permitted, and a new file does not get
to copy them.

Qualified: the CREATE TABLE target, the GRANT target, and all three COMMENT
targets. The table lands in `public` either way — that is the search_path
default the runner uses — so this changes nothing about what is created and
everything about whether the statement says so explicitly.

A note in the header records why, so the next reader does not simplify it back.

The stream fingerprint is restamped, since the migration's bytes changed.

* fix(OMN-16964): correct a false claim about where the link-2 grant is issued

An earlier commit on this branch asserted, in docker-compose.infra.yml and
again in migration 105's header, that no file issues chain_canary_reader's
link-2 grant and that it is live on the .201 dev lane only because it was
applied out of band. That is wrong, and it was my error.

The grant IS issued from this repository, on every migration run, by
scripts/run-forward-migrations.sh in its login-only role grant seam
(OMN-18060, #3352, 0e9106d, merged 2026-09-09T08:46Z):

    "chain_canary_reader:public.delegation_workflow_state:correlation_id,state"

reasserted through reassert_login_only_role_grants, with
tests/unit/infra/test_login_only_role_grants_omn18060.py pinning the entry.
The lane's forward-migration one-shot logs it verbatim today.

How I got it wrong: I searched docker/migrations/forward/*.sql and src/, found
nothing, and concluded the grant was unissued. I never searched the migration
RUNNER, which is where it lives. Absence from the two places I looked is not
absence from the repository — the exact shape of error the standing rule about
proving a zero with a positive control exists to prevent, applied here to a
grep rather than to a sweep.

Both comment sites now attribute the grant to the seam, by script name and by
the test that pins it, and say plainly that it is a runner seam rather than a
.sql file so the next reader does not repeat the inference. Migration 105
keeps issuing the link-5 grant as a .sql migration, because that one
accompanies the CREATE TABLE that needs it.

Comments only; no SQL statement, no grant and no behaviour changes. The stream
fingerprint is restamped because 105's bytes changed.

Reported by the team lead against lane omn18115-grant-by-migration's read-only
measurement. OMN-18115, which I filed on the same false premise, is corrected
separately.

* fix(OMN-16964): use pinned ledger ownership proof
…e serves (#3398)

`check_health_dimensions` reads a TOP-LEVEL `dimensions` object keyed by
dimension name. Nothing in this repository has ever produced that shape.

What the runtime actually serves, read live off the `.201` dev lane
(compose project `omnibase-infra`, runtime `0.38.22`) on 2026-09-10:

    details.runtime_health.dimensions -> [ {name, status, detail}, ... ]

six entries, every one `HEALTHY`. That is exactly what
`omnibase_infra.runtime.health.runtime_health_block.build_runtime_health_block`
builds and what `omnibase_infra.services.health_checker` mounts under
`details[RUNTIME_HEALTH_DETAIL_KEY]`, where that key is `"runtime_health"`.

So the check returns

    GET .../health -> 200 but carries no 'dimensions' object; an absent
    dimension set is not a healthy dimension set

against a lane reporting six healthy dimensions -- a fail-closed verdict
whose stated reason is false.

WHY NOBODY SAW IT. The probe never reached a lane. Until the parent commit
it addressed the lane as `localhost` from inside the runner container, so
every call ended in `Errno 111` and the parser was never entered. The three
existing cases passed because they asserted against the invented shape rather
than a capture -- the check proved its own fixture. That is why the new cases
read a payload captured off the live lane and a second case guards the
capture itself, so a hand-edit into the invented shape cannot make the first
one vacuous.

The two superseded cases are deleted rather than kept alongside: this repo
does not carry compatibility for a shape with no producer.

Malformed entries now count as UNHEALTHY rather than being skipped. An entry
the reader cannot judge is precisely the case a silent `continue` would turn
into a green run, and the fail-closed table is parametrized over all five
ways the block can be absent: no `details`, no `runtime_health`, an empty
list, a non-list, and a dimension carrying no status.
…3399)

* feat(OMN-18116): originate the causal edge and carry it on the wire

The runtime discarded the one fact a chain replay needs. Every event published
as a consequence of consuming another was built at a single site that already
held the consumed envelope, and threw the parent identity away. A chain read
back was a set of hops sharing a correlation id, with no statement of what
caused what, so a replay_green verdict could only ever be a claim.

Origination is one site: the dispatch-result applier reads the consumed
envelope from the context it is already bound to and stamps parent_envelope_id
on every event of that dispatch. No handler participates -- the canonical
signature is handle(request: ModelX) -> ModelY and never sees an envelope. No
consumed envelope means the hop is a chain head, which is a checkable
statement, not a gap. A self-edge is refused rather than published, because it
would let a broken chain re-derive as a valid one-hop chain.

The wire header half is not optional decoration. event_ledger fills its
per-hop identity column from the message_id HEADER, and that header was a
uuid4 default nobody set from the envelope. Measured read-only on the dev
lane over a two-hour window: of 737 ledger rows whose decoded body carried an
envelope_id, the column matched the body on 0 and differed on 737. An edge
recorded only in the body would be compared against a column holding an
unrelated transport id and would fail every hop. So identity and edge are
bound from the envelope, in one helper, at all three envelope publish seams --
the Kafka broadcast path, the in-memory bus and the typed publish adapter.
A rule that holds on one transport is a rule a verifier cannot rely on.

The edge decodes through a new optional-UUID parser rather than the existing
one. That one mints a fresh UUID when a header is absent or malformed, which
is right for an identity every message must have and catastrophic for an
edge: it would invent a parent for a chain head. Absent means absent.

No migration. The projection dumps the whole header model into the
onex_headers JSONB, so the edge is readable today at
onex_headers ->> 'parent_message_id'. A dedicated column is a convenience for
the verifier and is deliberately left to land beside the ledger_chain work,
whose PR is already re-scoping the forward stream's ordinal pin.

Core pin moves to 0.47.8, which carries the envelope field, with the version
matrix regenerated from pyproject rather than hand-edited.

* fix(OMN-18116): move the released-core seam fixture with the pin

The seam test freezes the exact released omnibase-core version infra consumes,
so a pin bump that does not move the fixture is a red gate by design. The
governed pre-push full suite on the lab host caught it, which is the gate
doing its job rather than a surprise.

Version string only. Every symbol and field type the fixture pins is unchanged
by 0.47.8, whose entire diff is one additive optional field on the event
envelope.

* fix(OMN-18116): prove postgres readiness with a real query, not pg_isready

Three consecutive governed full-suite runs on the lab host errored in this
module -- a varying subset each time, 4 then 3 then 3 -- while the other 27421
tests passed. A set that changes run to run on identical bytes is a clock or a
race, not a defect in the code under test, and the traceback names both.

The postgres image's entrypoint runs initdb against a TEMPORARY server on a
local socket, stops it, then starts the real one. pg_isready answers
affirmatively for that temporary server, so the probe could pass while the
server the next statement targets did not yet exist. Measured on the lab host:
CREATE ROLE role_omnibase_infra returning psql exit 2 immediately after a
passing readiness probe, surfacing as an ERROR in setup under check=True
rather than the skip this module intends for an unavailable environment.

Readiness is now a real SELECT 1 over TCP as the same user and host the tests
themselves use. It can only succeed against a server accepting the connections
the tests make, which closes the window rather than widening the wait.

The module-scoped container bring-up is also charged to the first test's clock,
and pulling and starting an image on a loaded host exceeds the suite-wide 60s
cap. That module now carries its own timeout. Neither change touches an
assertion or a skip condition: an absent Docker daemon still skips, a server
that never comes up still skips, and every statement under test still runs
under check=True.

Unrelated to the causal edge, and on this branch only because it is what the
governed pre-push gate blocked on. Never overridden.

* chore(OMN-18116): restamp the runner image identity for the core pin bump

The shared CI env digest is computed over the dependency set, so moving the
omnibase-core pin to 0.47.8 moves it. The recorded value is regenerated by
scripts/ci/runner_image_identity.py rather than hand-edited; the smoke gate
that caught the drift is the reason the file is machine-generated.
…#3395)

* fix(OMN-18083): the rebuild trigger's lane model learns projection_readback

Every merge to dev that reaches the overlay validation has failed to
rebuild the runtime since 12:42:50Z — five consecutive red runs,
including #3356's merge, which is the one that removed dev's two
hostile-review defects.

ModelCiBusLane sets extra="forbid" and did not know projection_readback,
which omnimarket declared on the dev lane (OMN-18060, omnimarket#2420).
The overlay stopped validating outright.

extra="forbid" is RETAINED. Widening it would silently accept a typo that
routes a publisher to a default it never declared — a worse defect than
this one, and the hole the model's own docstring exists to defend.

The nested block is a typed model, not dict[str, str]: a free dict would
accept dsn_value, i.e. a real DSN in a checked-in file, which the
overlay's contract forbids ("a NAME, never a value").

Kept local to the script rather than importing ModelLaneProjectionReadback
from src/. Layering permits the import (20 scripts do it), but this script
is loaded BY FILE PATH in hermetic tests, so it must not depend on the
package being importable.

THE TEST READS THE LIVE OVERLAY, NOT A FIXTURE. A fixture pins what this
repo BELIEVES omnimarket declares and would have passed through both
outages — this is the second time a consumer model has gone red on a key
the overlay legitimately declared, and the first is written into that
model's own docstring. It skips only when the sibling checkout is absent,
because a missing clone is not evidence of drift.

Proven on this exact source, both directions:
  remove the field    -> RED, 2 failed (live-overlay guard + shape test)
  relax extra=forbid  -> RED, DID NOT RAISE (the AC2 test)
Two different tests catch the two mutations, so a future fix cannot buy
green by removing the guard.

Also fixes a defect I introduced mid-change: adding a nested typed model
meant ModelCiBusLane's own postponed annotation needed resolving under
file-path loading. Without it, test_pr_merged_rebuild_trigger.py went
from 33 passed to 5 passed + 28 errors — while my own file, both
mutations, mypy and ruff all read clean. Baselined with git stash to
prove it was mine. Hence _CI_BUS_TYPES_NAMESPACE and both model_rebuild
calls.

Full tests/unit/scripts/: 1757 passed, 2 skipped, 0 failures.

* test(OMN-18083): fail closed when live overlay is unavailable

* test(OMN-18083): keep live overlay guard in binding workflow

---------

Co-authored-by: jonahgabriel <jonah@omninode.ai>
…n observe (#3402)

The NOT_CONVERGED finding read, verbatim:

    The command was published; the lane did not apply it.

The guard observes neither half of that. What CI publishes is a
redeploy-start command (`scripts/trigger_rebuild_on_merge.py`,
`TOPIC = "onex.cmd.omnimarket.redeploy-start.v1"`), and that is all the job's
`published` output attests. The deploy agent consumes a DIFFERENT topic,
`onex.cmd.deploy.rebuild-requested.v1`, which only node_redeploy's deploy
effect emits. An orchestrator and that effect sit between the two.

MEASURED 2026-09-10 by the lane that owns the agent, reading the control topic
offsets 95..125 against the agent's own job store: pull request 3383 merged
07:37:40Z, and the first real rebuild-requested command appeared at 11:08:23Z
-- 3h31m after this guard's window had closed. Publish-to-accept, once a
command existed, was 268ms, and a second job 962ms. Nothing was slow on the
agent and nothing was wrong with its clone. The effect is serial: it publishes
one command, then polls for completion until its own 660000 ms timeout, and it
spent 07:06Z to 11:08Z working through five stale correlations queued ahead of
the real merges.

So the sentence asserted the one thing that was false, in the first line a
lane reads when diagnosing a red convergence job. It sent one lane toward the
agent's shared clone and another toward the wall-clock bound. The cause was in
neither. That is the same defect class as the two probe defects this ticket
already fixed -- a claim the code cannot support -- and it is the most
expensive of the three, because it misdirects the diagnosis rather than just
failing.

The verdict now states what the run attests, names the topic split and the
effect's serial behaviour, and gives an ordered checklist that starts at the
hop BEFORE the agent. The pre-existing DLQ and consumer-group guidance is kept,
not replaced: a dead-lettered downstream event is still a real cause of this
shape (OMN-17888).

The workflow comment deriving the 25-minute bound from the same 660000 ms is
corrected for the same reason: that number bounds ONE correlation's execution
and says nothing about the queue ahead of it, so the bound covers a lane whose
effect queue is empty and cannot cover a backlogged one.

The bound is deliberately NOT widened. No wall clock short of hours would have
caught the measured case, and a longer wait holds the single omnibase-deploy
runner slot longer, which serializes the next merge's guard behind this one --
already visible on 2026-09-10, when one merge's window did not open until 26
minutes after its own merge because it queued behind its predecessor. The
right fix is for the guard to read queue position rather than wall clock, and
the queue it needs to read is the effect's. That is a design change, not a
constant, and it is recorded rather than smuggled in here.
…g it (#3404)

With the address and the dimensions path fixed, the first receipt to reach a
CONVERGED lane exposed the last defect, and it is the one that made a passing
compose-dev receipt impossible.

Measured on run 34478680748, from the job log, to the millisecond:

    13:12:28.5442420Z  ok: dev lane converged onto c187442 after 0h22m
    13:12:28.7181102Z  uv run python scripts/ci/lab_pass_receipt.py probe-lane
    13:12:28.8260884Z  GET .../8085/ready -> 0 Errno 111 Connection refused

174 milliseconds. `check_dev_lane_staleness.py` returns the moment the RUNNING
container carries the expected `org.opencontainers.image.revision` label, and
that label is set when the container is CREATED, not when the runtime has bound
its port. So the instant convergence succeeds is the instant the lane is least
able to answer, and the three HTTP checks fired straight into a compose
recreate.

THE PERVERSE CONSEQUENCE, which is why this is not cosmetic. Compare that
receipt with the one from the run before it (`ba15dd33`), where the lane had NOT
converged and was therefore serving an older revision quite happily:

    ba15dd3   deployed_revision FAIL   ready_main ok    ready_effects ok
    c187442   deployed_revision ok     ready_main FAIL  ready_effects FAIL

The HTTP checks pass only when the revision check fails, and fail exactly when
it succeeds. A compose-dev receipt could not be a PASS by construction. That is
the fail-always shape this module's own docstring names, one layer deeper than
the two defects already fixed under this ticket.

THE BUDGET IS NOT A CONSTANT. The compose `x-healthcheck-defaults`
`start_period` is 10s and measures something else entirely; deriving a settle
budget from it would repeat the false-derivation defect corrected in the
sibling commit. What the caller passes is whatever remains before the emitting
job's own ceiling, minus a reserved tail for the reads, the emit and the
upload. So the single omnibase-deploy runner slot stays bounded exactly as it
is today, a 22-minute convergence leaves ~6 minutes of settle, and a
29-minute one clamps to zero rather than eating the time the receipt needs to
be WRITTEN -- an unwritten receipt being the one outcome worse than a failing
one.

BOTH readiness endpoints are waited on, not just main. Measured on the lane
2026-09-10: at 12:36:29Z `omninode-runtime` read "Up 3 minutes (health:
starting)" while `omninode-runtime-effects` was still "Created"; at 12:37:29Z
main was healthy and effects had been up 44 seconds. Waiting on main alone
would clear the race for `ready_main` and leave it for `ready_effects`.

IT IS A WAIT, NEVER A VERDICT. The checks still run afterwards and still fail on
a lane that never comes up. Every check carries what the probe waited for,
passing ones included, because a green read taken with no settle budget is a
different fact from a green read taken after the lane reported itself up, and
the receipt must not make them look identical.

Five tests, red at the parent: the signature carries a budget; a lane that
answers on the third poll produces passing checks; a lane that never answers
still fails every check; a zero budget is recorded in the evidence rather than
hidden; and the workflow actually passes a budget. The budget tests run on a
virtual clock, because patching only `time.sleep` would leave the production
loop spinning against a real deadline and quietly turn a unit test into a
thirty-second one.
…ready has (#3403)

* fix(OMN-15504): give the dev lane the poll margin every other lane already has

Two independent defects, both surfaced by the delegation chain going dead on
the .201 compose dev lane at 2026-09-10T11:56:43Z.

1. THE DEV LANE WAS THE ONLY LANE ON THE LIBRARY DEFAULT
--------------------------------------------------------
aiokafka evicts a consumer whose handler has not returned to the poll loop
within max_poll_interval_ms. The eviction is not the visible failure: the
handler finishes a moment later, its OffsetCommit is refused with
UnknownMemberIdError, the committed offset never advances, and the same records
are redelivered on every rejoin. It does not self-heal, and a container restart
does not clear it, because the committed offset lives in the broker.

judge (docker-compose.judge.yml:177), stability-test
(docker-compose.stability-test.yml:213,298,393) and lakshman
(docker-compose.lakshman.yml:228) have all carried 1800000 for some time. The
dev lane was never brought along, so it ran the 300000 default -- and
node_delegate_skill_orchestrator's worst-case handler duration is its contract's
wait_timeout_seconds of 300, which landed on that deadline exactly. The group
livelocked through four identical ~300 s join-then-evict cycles with its
committed offset frozen at 317 and lag static at 31.

Scoped to the dev-lane overlay rather than the shared base
(docker-compose.infra.yml x-runtime-env), because the prod overlay merges that
same base: setting it there would change the prod lane's consumer behaviour,
which is not this change's to make. Applied to the three services that host
auto-wired consumers -- omninode-runtime, runtime-effects, runtime-worker --
which is the placement stability-test already uses.

This is the SECOND of two halves and deliberately not the first. omnimarket
#2449 makes the handler bound itself, so the fix holds on any lane including
one still on the library default. Raising the deadline here restores the MARGIN
between the two, so a handler that is merely slow degrades into a late terminal
instead of a permanent livelock.

Gated by tests/unit/scripts/test_omn15504_dev_lane_consumer_poll_interval.py,
which derives the declaration from the overlay and compares against the
stability lane rather than a literal -- a pinned number would pass just as
happily if every lane drifted together. Both tests fail on pre-change content.

2. A DOWNSTREAM STALL WAS REPORTED AS AN INGRESS FAULT
-------------------------------------------------------
The chain canary's terminal_missing verdict rendered a budget-exhausted ingress
as "ingress also returned ok=false (dispatch_timeout: ...)". That reads as an
accusation and was taken as one: the investigation of this very outage spent its
first pass on the ingress while the delegate-skill consumer sat livelocked one
hop downstream. An ingress that consumes its ENTIRE budget and then reports a
dispatch timeout has accepted the request and published it -- its own access log
and the command topic both show it at offsets 345 and 346 -- and is observing
the same silence the readback did. It is the component REPORTING the defect and
was the only one the verdict named.

The discriminator is evidence the probe already collects: elapsed_ms against
budget_ms. A budget-exhausted answer now attributes the silence to the chain and
points at the command topic's consumer. An ingress that refuses INSIDE its
budget is a genuinely different finding and keeps its own distinct wording, so
this corrects an attribution rather than dropping one.

CORRECTED PREMISE, recorded rather than quietly dropped: the brief for this work
stated that link 1 is judged from the /skill response's ok flag. It is not.
_build_link_verdicts already passes link 1 on reachability alone, which is why
the failing runs scored 1/5 rather than 0/5 -- the one link that passed WAS
link 1. The misattribution was entirely in the prose verdict beside the table,
and only that is changed here. A fix aimed at the link logic would have changed
correct code. test_link_one_still_passes_when_the_ingress_answered pins that,
and it passes both before and after.

Tests: tests/unit/nodes/node_chain_canary_effect/ 180 passed, 1 skipped; the two
new attribution tests fail on pre-change content and the link-1 test passes on
both. mypy clean on both touched packages; ruff format/check clean.

Ticket: OMN-15504
Evidence-Ticket: OMN-15504

* chore(OMN-15504): vendor the owed node_projection_delegation 0038 migration

NOT this ticket's change, carried here because it blocks every commit in this
repo and nobody else has it in flight.

scripts/sync-node-migrations.sh --check runs with always_run: true, so it fires
on every infra commit regardless of paths, and it compares against the CANONICAL
omnimarket clone's live state. That clone advanced past this branch's first
commit, and node_projection_delegation/0038_delegation_events_writer_identity.sql
-- merged on omnimarket dev as OMN-18140 (omnimarket#2446, e8b8f496) -- had never
been vendored into this repo. Verified before carrying it rather than assumed:
the file is present on omnimarket origin/dev, and no open omnibase_infra PR
carries the vendoring (all 40 open PRs enumerated by file list).

So the vendoring is genuinely owed, mechanically produced by the repo's own sync
script, and exactly one file. It is a separate commit so it stays legible in this
PR's history and can be read without the OMN-15504 diff around it.

Stated plainly because it has blast radius beyond this ticket: vendoring the
migration is how it reaches the infra migration runner, so it will apply on the
next dev-lane rebuild. That is the lane OMN-18140 was always going to reach, and
the dev lane is the fully mutable test platform, but it is not silent and should
not be reported as if it were.

Ticket: OMN-15504

The vendoring also requires the migration's declaration row in
docker/migrations/forward/_ledger/application-migrations.tsv, which the sync
script does not author. The row's classification is copied from this node's own
immediately preceding migration (0037): same stream, same owner, same `tenant`
domain -- 0038 adds columns to delegation_events, which is a tenant-domain
projection resolving the tenant_projection binding, exactly as 0037 does. The
checksum column is a plain sha256 of the file, verified against 0037's existing
row before writing 0038's rather than inferred from the column name.
validate_application_migration_manifest.py then passes: 175 active, 0 blocked.

* fix(OMN-15504): record the canary attribution change in its node contract

The contract-sync gate is right: the handler changed and the contract did not.
node_version 1.3.1, and the metadata description now states what the verdict
discriminates on and, equally, what did NOT change -- link 1 already passed on
reachability alone, so only the prose verdict beside the link table was wrong.
Recording the unchanged half matters: the next reader of this contract should
not go looking for a link-logic change that was never made.

No bypass token: a gate that fires because the contract is genuinely stale is
a gate doing its job.

Ticket: OMN-15504

* chore(OMN-15504): move the migration-declaration count pin 174 -> 175

The count pin belongs to the vendored 0038 migration in the previous
commit, not to this ticket. Recorded in the test beside the existing
rationale so the next reader can tell an OMN-15504 change from an
OMN-18140 one that arrived through the vendor-sync hook.

Caught by the governed pre-push full-suite escalation on h201.2, not
locally: 1 failed, 27509 passed. Fixed rather than narrowed.

Ticket: OMN-15504
…t] (#3405)

Co-authored-by: omninode-bot <bot@omninode.ai>
…esh build scope (#3408)

OMN-18114 added the carrier to both lane overlays, to the lane census manifest,
and to both `*_LANE_ONLY_RUNTIME_SERVICES` arrays in deploy-runtime.sh -- and to
neither refresh script. That last omission is the whole difference between the
carrier being declared and the carrier running.

`resolve_lane_runtime_services` appends the lane-only array only when no build
override is in play:

    # A scoped-build override is an explicit operator instruction to touch ONLY
    # the named services; silently appending to it would defeat the point.
    if [[ -n "${RUNTIME_BUILD_SERVICES_OVERRIDE:-}" ]]; then
        return 0
    fi

Both governed refresh scripts ALWAYS export that override, so for the two lanes
this repo refreshes the lane-only array is unreachable and each refresh script's
own REFRESH_BUILD_SERVICES is the entire restart set. The carrier was absent from
both.

Measured live, read-only, on the lab host 2026-09-10: the stability-test lane's
own dry-run printed a ten-service build scope with the carrier absent, while the
lane census reported `container_absent` for it at severity critical. The lane is
the surface the `stability-proven` premise of every live prod-promotion grant is
resolved from, so it was running degraded against its own declared topology.

The gate, red-first at the parent commit
----------------------------------------
tests/unit/scripts/test_omn18114_lane_carrier_build_scope.py derives the
obligation two independent ways and asserts both, so neither derivation alone
has to be right:

  1. every `*_LANE_ONLY_RUNTIME_SERVICES` member appears in that lane's
     REFRESH_BUILD_SERVICES -- the restart set and the build scope must agree;
  2. every service the LANE OVERLAY promotes into the `runtime`/`full` compose
     profile appears there too -- derived from compose, so it holds even if the
     deploy-runtime.sh arrays are themselves wrong.

At the parent commit both fail on both lanes, naming exactly
`tenant-projection-writer`. Four failures, two derivations, one service per lane.

Why the sibling gate missed it
------------------------------
test_omn17448_dev_lane_writer_build_scope.py derives its expected set as overlay
services MINUS base services, on the reasoning that an overlay key also present
in the base is an override rather than a new service. True for every service it
was written against, false for this one: the carrier IS declared in the base,
gated behind the inert `profile-carrier-optin` compose profile, and what makes it
real on these two lanes is the overlay promoting it into `runtime`. Subtracting
the base subtracted precisely the service that needed covering. Derivation 2
above closes that hole at its root.

This gate's array reader is also line-based and comment-stripping rather than the
sibling's negated-character-class body, which stops at the first round bracket in
an array's prose and silently drops every entry after it. That is a measured
failure, not a hypothetical -- the dev refresh array carries a NOTE asking future
editors to keep brackets out of its comments. A reader that cannot be truncated
needs no such promise, and a regression test injects a bracketed aside to prove
it.

Also here
---------
- A pre-commit hook runs this gate and the OMN-17448 one on any edit to the
  arrays, the lane overlays, the base compose, or the gates themselves, so the
  two halves cannot drift apart within a single commit. CI already covers the
  file through the full-suite job.
- test_stability_lane_rollback_omn18061.py pinned the build set size to a literal
  10. The count was never the property under test -- rollback-set equals
  deployed-set is -- and a literal there means every legitimate widening arrives
  with an unrelated red test and an invitation to bump the number. Now shrink-only
  at that floor, which still catches the regression worth catching.

Verification
------------
- The new gate is red at the parent on 4 assertions and green here.
- Resolved by actual bash evaluation rather than a regex read, the stability
  scope is 11 services and the dev scope 17, each ending in the carrier.
- 57 passed across the new gate plus the OMN-17448, OMN-18012, census-parity and
  OMN-18061 rollback siblings.
- `bash -n` and `shellcheck -S error` clean on both refresh scripts.
- ruff format, ruff check and `mypy --strict` clean on the new test.
… AC1-3) (#3407)

* fix(keycloak): set fullScopeAllowed=false on all 9 clients with scope mappings for role-bearing service accounts (OMN-17848 AC1-3)

* fix(keycloak): die on non-200 scope-mappings GET; add failure test (OMN-17848)
… and names the bar (#3409)

Implements AC3 and the closer half of AC4 from the ruling recorded at
docs/tracking/ROLLING_WORK_LEDGER.md:6148 (orchestrator, under the
deterministic-truth doctrine; not an operator ruling). omnimarket#2460
carries the proof class. This carries the JOIN, because deciding whether a
criterion is state-shaped needs the ticket's acceptance text and
node_dod_verify never sees it.

It moves in BOTH directions, deliberately.

LOOSER: a ticket whose every parsed criterion asserts live state, each
discharged by a verified readback, no longer dies on
behavior_proving_count <= 0. That is the case the ruling exists to admit.
Bounded to ALL criteria: one behaviour-shaped criterion, or a body whose
criteria cannot be read, and the behaviour proof is still required. A ticket
is judged by its strictest criterion, never on average.

STRICTER: a behaviour-shaped criterion whose only verified checks are
readbacks now HOLDS. Binding used to key on status alone and never on proof
class, so a readback silently discharged a criterion asserting what the code
does. That hole is invisible to every test that has no behaviour proof at
all, so it has its own test with a real proof beside it.

Deliberately untouched: merge-state, surrogate and indeterminate checks bind
exactly as before. Narrowing those is a different argument nobody has made,
and making it here would retroactively un-close tickets that flipped on that
basis.

AC3: the gap_no_behavior_proof comment now names the bar -- which command
shapes prove behaviour, which prefixes are transparent, that a live probe
through ssh/aws/kubectl/curl/a shell loop is not behaviour-proving, and that
an asserted readback is the admissible route for live state. A hold that
states a verdict without a remedy teaches nobody anything, which is what
three lanes spent a night rediscovering by measurement.

HONEST LIMIT, in the code and not only in the PR: criterion shape is read
from prose and is a heuristic. State-shaped requires a positive state marker
AND no behaviour marker; everything else falls to behaviour and holds. The
asymmetry is chosen -- reading a criterion as behaviour-shaped costs a
comment, reading one as state-shaped RELEASES a flip on a readback -- so the
vocabulary earns its way in one phrase at a time and the veto wins ties.

node_version 1.11.0 -> 1.12.0, so the gap-comment fingerprint moves with the
rule it describes.
…gent's DEV scope (#3411)

The gateway forwarder was explicitly out of OMN-18108's scope because it is a
SEPARATE compose project (`omninode-gateway`) with a mounted config, and its
only deploy path was a hand-run `scripts/deploy-gateway.sh`. Every agent-path
DEV deploy therefore advanced the runtime family and left the forwarder on
whatever image it already carried, indefinitely, with `restart: unless-stopped`
keeping it running and healthy so nothing reported it.

Design: the agent CALLS the sanctioned script rather than reimplementing
compose handling for a second project. A gateway deploy is build -> resolve
digest -> retain the previous image as a durable rollback tag -> sync two
root-owned host files -> rewrite the image pin in the lane env file -> write
the rollback registry -> reload the systemd unit -> verify the running digest.
Six of those eight steps have no analogue in `_compose_build` / `_compose_up`,
and the half that would drift silently is the rollback record. The script also
resolves its own repo root from `$0`, so invoking the copy inside the
deploy-source clone the GIT phase has just reset is what binds the gateway
image to the deployed sha -- nothing is passed and nothing can disagree.

Scope vs compose argument are deliberately different lists.
`services_for_scope(RUNTIME|FULL, lane=DEV)` now carries `gateway-forwarder`,
because a DEV deploy is RESPONSIBLE for it. Every site that builds an argument
list for the `omnibase-infra` compose project subtracts it via
`without_gateway_services`: that project does not declare the name, so passing
it would abort the runtime phase on `no such service` -- the same fatal shape
the OMN-18108 comment warns about for its own list. A command naming ONLY
gateway services is branched explicitly, since an empty residual list means
"the whole scope" everywhere else.

Fail-closed precondition: before any build effect the step proves the lane env
file exists and declares both operator-supplied map paths, and that each
resolves to a readable file. Only PATHS are read; the contents of those files
are never opened, logged or carried. Every `GATEWAY_`-prefixed variable except
the three location overrides is stripped from the script's environment, so a
value the agent happens to be carrying can never be the one a map resolves to.
That is the "never from env" half of AC1, enforced rather than assumed.

Prod, stability-test and judge are unchanged, asserted twice over: on the
resolved service lists and on the actual argv the executor issues, because an
equal list and an equal invocation are different claims.

Tests: 42 new in `test_gateway_lane_scope_omn18134.py`, including positive
controls that fail if the gateway service name is absent from the compose model
the module scopes. Red-then-green measured by reverting only the two
behavioural changes with the symbols left importable: 18 failed / 24 passed,
then 42 passed. The 24 that stay green under the revert are the positive
controls and the unchanged-lane assertions, which is correct -- those assert
that nothing moved. Full deploy-agent unit suite 525 passed.

The mutating shell-out is stubbed by default in the deploy-agent unit conftest,
with a `gateway_lane` marker opt-out, following the existing
`promotion_guard` arrangement: several tests drive `rebuild_scope` while
stubbing only the compose methods by hand, and leaving the new step live for
those would let a unit test run a real deploy on whatever machine the suite is
on.

Closes AC1 (agent-path build/recreate from the deployed sha; maps read from the
lane config store and never from env), AC2 and AC3. AC4 is a lab readback on
the next agent-driven dev deploy and is reported as a residual -- this lane
holds no grant for a gateway redeploy and performed none.

Refs: OMN-18134, OMN-18108, OMN-15521, OMN-15743, OMN-18120
…ts own loader demands (#3412)

#3392 credentialed the gateway forwarder's dev-lane legs and did not
credential the healthcheck that watches them. `onex-gateway-canary-probe`
calls the SAME `load_gateway_forwarder_runtime_config` the forwarder does,
and that loader fails closed -- correctly, by design -- as soon as any bus
leg names a `sasl_credential_ref`. The probe inherited a hard refusal it was
never given the argument to satisfy, so every 15s tick died in configuration
load before dialing a broker:

  ValueError: gateway bus leg 'local_bus' names sasl_credential_ref=... but
  no lane-credential map was supplied; the gateway refuses to start rather
  than open a client that cannot authenticate

Measured on .201 2026-09-10T18:17Z against the deployed fix, with the
forwarder and its healthcheck read in the same window: the process was
authenticated and delivering (`Authenticated as onex-dev-lane via
SCRAM-SHA-256`, zero `Connection at redpanda:9092 closed`, dev
`onex.evt.omniclaude.tool-executed.v1` 599687 -> 599711, and
`omninode_internal.work_events` n_tup_ins 71208 -> 71237 with n_tup_upd
unchanged at 94811, i.e. real INSERTs rather than the no-op upsert loop),
while `docker inspect` reported the container `unhealthy` with FailingStreak
5 and the traceback above as its last probe output.

That is the one state a healthcheck must never invent: it declared the
traffic path dead while the path was carrying traffic. It is the exact
inverse of the 2026-08-04 outage the probe was built for -- ready-file
present, forwarding 0% for four days -- and it misleads in the same way,
because both make the health signal independent of the thing it names.

WHY THE EXISTING TEST DID NOT CATCH IT. `test_compose_passes_the_dev_lane_
credential_fail_closed` asserts the flag is `in compose` -- a substring match
over the whole file. The flag WAS in the file, on the forwarder's `command:`.
The healthcheck's `test:` array was never looked at, so a file-granularity
assertion reported green on an invocation-granularity defect. The three tests
added here are per-argv: the healthcheck carries the flag; the process and
the probe name the SAME map path (two argv lists naming two maps would make
the probe a check on a configuration nothing runs); and the probe's parser
actually accepts it, since a flag argparse rejects would turn the healthcheck
into a usage error on every tick. The first also asserts the healthcheck
still invokes the probe at all, so it cannot pass vacuously if the
healthcheck is later rewritten.

WHY THE ARGUMENT IS OPTIONAL AND NOT REQUIRED, unlike `--broker-ref-map`: a
deployment whose legs are all PLAINTEXT names no ref and needs no map. The
loader already refuses by name when a ref is present and the map is absent,
so making it required here would break that deployment in order to restate a
refusal the loader owns.

NOT A ROTATION (Operating Rule 22): no credential is minted, rotated,
re-issued or revoked, and no value appears in the diff, the tests or any
error message. Only the PATH of the operator-supplied map is named.

Red at parent: 3 failed / 16 passed. Green at head: 35 passed across this
file and the runtime loader suite; ruff, mypy --strict and pre-commit
--all-files clean.
…n for (#3413)

Measured against OMN-17771's real acceptance section, not a paraphrase: the
first cut of the marker set recognised 2 of its 5 criteria, so
_every_criterion_is_state_shaped returned False and the ticket the ruling
names as its worked example would still have been held by the behaviour
conjunct. Shipping that would have been a rule that reads well and does not
do the thing it was ruled for.

Five phrases added, each lifted from one of those criteria rather than
invented, which is the standing rule for this set:

  "Zero `client_id=x` ... occurrences"   -> zero ... occurrences
  "`beta-layout` checkers green"         -> checkers green
  "a client that exists on the plane"    -> exists on the
  "Merged to `main` and read back"       -> merged to `x
  "returns HTTP 200 with a form"         -> HTTP <status>

The veto is unchanged and still wins ties, so the widening cannot become a
general loosening. Four controls assert exactly that: a criterion carrying a
widened state phrase AND behaviour language still reads behaviour-shaped.

The worked example is now pinned end to end -- all five of OMN-17771's real
criteria, asserted individually and through the aggregate. That assertion is
what would have caught the first cut.

node_version 1.12.0 -> 1.12.1.
…oven load-bearing (#3414)

Review finding on #3413, accepted rather than argued: the vocabulary test
asserted the whole OMN-17771 acceptance body classifies as state-shaped, so a
marker that never fires, or one whose semantics are wrong, was invisible --
a neighbouring marker carried the assertion and the aggregate stayed green.

That was not hypothetical on this body. Measured by deleting one alternation
line at a time from the live pattern and re-running:

  delete the HTTP marker    -> the end-to-end test stays GREEN
  delete the merged-to marker -> the end-to-end test stays GREEN

because AC2 is already matched by the older `returns` marker and AC5 by
`read back`. Two of the five phrases this PR added contributed nothing that
any test could see, which is precisely what the reviewer said.

Three things close it, per marker rather than per body:

1. Isolation. Each added phrase gets the shortest criterion carrying it and
   nothing else, lifted from the OMN-17771 criterion the phrase was taken
   from. Where the real criterion also carries a pre-existing marker the
   isolated form drops that word -- "gives HTTP 200" rather than "returns
   HTTP 200" -- because leaving it in is what made the row vacuous.

2. Mutation controls, which are the half isolation alone cannot do. Each
   test deletes that marker's own alternation line from the vocabulary and
   asserts the same criterion, held byte-identical, stops classifying. A
   misspelled, over-escaped or shadowed marker cannot pass its control. The
   helper refuses to run unless the fragment names exactly one line and that
   line is an alternation branch, so a control can never be built on a
   mangled pattern and quietly prove nothing. Each control also asserts the
   other four markers still fire, so an empty or inverted pattern cannot
   satisfy every row at once.

3. Near-miss and veto coverage. Each marker must refuse the neighbouring
   string it must not accept, which catches looseness rather than silence,
   and the behaviour veto is now asserted against every added marker
   individually rather than against four sampled combinations.

The end-to-end blob test and the existing veto controls are unchanged and
still present; this adds to them rather than replacing them.

Attribution: a concurrent lane in the same worktree had independently staged
the isolation and near-miss tests for this same finding. That work is kept
here rather than discarded, with its HTTP row repaired -- it read "the URL
returns HTTP 200" and so passed with the HTTP marker deleted, reproducing
the reported defect inside the fix for it.

No src change; vocabulary and node_version are untouched.
Remove the governed impacted-test selector from the pre-push stage. It was
the only hook here that invoked a test runner, so pre-push becomes policy,
type and lint only: mypy, architecture-layer validation and deploy-scope DoD
parity, three hooks, 16.2 s measured on a representative change against
180 s for the whole-tree pass.

Hosted CI is the enforced merge gate and has been the test surface in
practice since the trusted-CI routing seam went hosted on 2026-08-27. The
local leg bought earlier feedback rather than a verification guarantee, and
it was the measured load on the lab: 67 heavy omnibase_infra pre-push runs in
the five days to 2026-09-09, each budgeted at 5.0 runnable cores, heavy
escalations taking 2h43m to 3h49m. Feedback moves later, not away.

Plan of record: knowledge-base-internal#328, phase 1. Operator ruling
2026-09-10: retire the test leg one repo at a time. Repo 1 of 3.

The selector script is retained, not deleted. The gate-runner compose file
and the runner routing policy both reference it and roughly ten test modules
pin its content, so its header is restated as manual-invocation only. The
CI-side selector is byte-unchanged.

Closes a coverage gap the retirement would otherwise have opened silently.
Pre-push ran -m "not integration"; CI runs -m "not slow and not chaos and
not kafka and not performance and not live_github_api". Collecting tests/unit
under each filter and diffing gives 39 tests, 25 performance and 14 slow,
across four files, whose only execution path was the leg being removed. Four
of them carried a second exclusion, a skipif on the generic CI variable that
every Actions runner sets, so scheduling alone would not have run them.
microbenchmarks-nightly.yml is their new path and the skip is now an explicit
ONEX_RUN_MICROBENCHMARKS opt-in that only that workflow sets. All 39 pass in
isolation; two read as failures under a load average of 46 on the launching
Mac, which is the shared-runner timing sensitivity their exclusion reasons
already describe.

tests/ci/test_prepush_test_leg_retired_omn18162.py is the enforcement, not a
note: it asserts no hook reaching pre-push invokes a test runner, that the
other three survive by id and in order, that the selector script stays and
says it is manual-only, that no env knob restores the leg, that the nightly
exists and selects both markers, and that the microbenchmark skip is never
keyed on the generic CI variable again. It carries a positive control, which
caught a real fail-open hole in its own scan.

Also folds in one pre-existing ruff auto-fix on dev, an f-string with no
placeholder in scripts/tests/test_seed_keycloak_clients.py, unrelated to this
change but corrected by the format hook.
onex-gateway-forwarder.service is Type=oneshot with RemainAfterExit=yes.
systemd refuses a reload job on a unit that is not active, so the script's
bare 'sudo systemctl reload' was only a valid deploy step while the unit was
already running. On .201 the unit went to failed at 2026-09-10T18:16:19Z, and
every --execute after that died at its LAST step, after the build, the host
file sync and the gateway.env digest pin had all already been applied. The
lane was left pinned to a digest nothing was running.

The reload step now reads the unit's activation state: reload when active
(unchanged, ExecReload force-recreates), reset-failed then start when failed,
start when inactive. The registry rollback_command carries the same sequence,
because a rollback is most often reached FROM a failed unit.

The systemctl fake in the test harness now models real systemd activation
state instead of exiting 0 unconditionally, which is why the old bare reload
passed every test and still failed on the host.
… gate, date the doctrine (#3417)

Applies four orchestrator decisions to the OMN-18162 retirement, which merged
as e50b0b8 before the decisions arrived.

Corrects a wrong claim in that PR. It said 39 tests/unit tests had no CI
execution path. nightly-tests.yml already ran `-m "slow or chaos"` over
tests/, so the 14 slow-marked ones always had one. The real gap was the 25
performance-marked tests, which no workflow selected positively.

So microbenchmarks-nightly.yml is deleted and replaced by a performance-tests
job inside the existing nightly-tests.yml. One nightly surface, one summary,
one failure notification, rather than a second workflow duplicating all three.
The job is pinned to [self-hosted, omnibase-ci] rather than reading the
trusted-CI seam: these assert wall-clock thresholds, the seam currently
resolves to a hosted runner, and a seam flip must not silently relocate them.
It runs serially, since running benchmarks under xdist measures contention.
The slow/chaos job keeps its own placement and parallelism, untouched.

Drops the env gate entirely. ONEX_RUN_MICROBENCHMARKS is gone and the four
skipif decorators with it. Selection is by marker alone: the performance
marker keeps these out of pull-request CI exactly as before, and the nightly
job selects it positively. No ambient switch and no dual path; rollback is a
git revert of the squash.

Three tests move, and only three. test_get_p99_latency_under_threshold,
test_registration_throughput_regression and test_secondary_index_speedup get
an individual performance marker. Marking their class instead was tried and
reverted: it would have pulled six tests that do run in pull-request CI out of
it. Proven by collecting the module under CI's own filter before and after,
where the difference is exactly those three, all of which were already dead in
CI behind the removed skip.

Doctrine becomes the dated pointer rather than the rewrite. CLAUDE.md gains
one section saying the leg was retired on 2026-09-10, what remains, how to
roll back, and that the full rewrite waits for the phase-0 report because
coupling the rules to a reversible change means a revert silently reverts
them. The pre-commit config block carries the same pending note.

The enforcement test follows: it now asserts the nightly job exists, selects
the performance marker, stays pinned to the fleet, and is a dependency of the
summary job, and that no skipif or environment read returns to the
microbenchmark module.
… path (#3418)

* test(OMN-15744): golden and error chain tests for the MSK SNI bastion path

Nothing watched the lab -> SNI bastion -> MSK routing path, which is why a
2 -> 6 broker scale-out on 2026-08-06 left four of six brokers mis-routed for
a month. Bootstrap and metadata are served by any broker, so every liveness
surface stayed green; only leader-routed requests failed, with
NotLeaderForPartition, which reads as transient noise. It was found by hand
during an unrelated diagnosis. These tests replace that luck.

Three tests in tests/integration/bastion:

  golden — every live broker answers AS ITSELF through the map. Discriminates
    on the broker's own error code: a leader-routed offsets request returns 0
    from the partition leader and 6 from any other broker, so the answer
    reports the responder's identity. Carries its own positive control.

  error — an SNI the map does not carry must fail at connect, not silently
    reach a real broker. The original defect was precisely that an unknown
    name got a working session from the wrong broker. A mapped SNI must
    succeed in the same run, so a down bastion cannot read as a pass.

  resize detector — every broker the MSK control plane reports must be
    routable, with an unmapped name required to be unroutable so the claim is
    not vacuous. This is the test that turns red on the next resize.

A TLS-only test was considered and rejected on measurement, not taste: all six
broker SNIs present ONE wildcard certificate (sha256[:16] e4224daa249fdb90),
so a handshake test cannot tell which broker answered and would have been
green throughout the outage.

"Produce to a topic led by each of the six brokers" was the first design and is
not achievable: the sanctioned topic path governs replication factor only and
nothing here can influence leader placement, so such a topic would land
wherever the cluster chose. The error-code discriminator covers all six
brokers deterministically instead, broker 4 included.

Scheduled hourly by msk-bastion-canary.yml on the self-hosted omnibase-deploy
runner, the only runner with reach to the bastion. Read-only by design: no
produce and no topic creation against tenant data on a schedule.

tests/ci/test_msk_bastion_canary_workflow.py guards the canary against being
defanged — dropped schedule, wrong runner, continue-on-error, legs no longer
invoked, or a produce creeping in. The bastion marker is excluded from the
nightly sweep, which runs where the bastion is unreachable and would otherwise
go permanently red.

* chore(OMN-15744): regenerate runner image identity lock

Adding the `bastion` pytest marker to pyproject.toml changed a component the
runner image identity binds, so tests/ci/test_runner_image_identity.py went
red. Regenerated with scripts/ci/runner_image_identity.py --mode generate.
Identity v7 d2e218e9a49625aad7918e6592d1f11c.
…t] (#3419)

Co-authored-by: omninode-bot <bot@omninode.ai>
…an attested write (#3422)

* feat(OMN-18159): the runtime kernel's projection adapter can perform an attested write

delegation_events carries a writer attestation -- writer_identity stamped by
Postgres as CURRENT_USER, written_at as NOW(), both restated on the DO UPDATE
arm -- and the staging green bar reads it to answer which database principal
wrote a row.

Four implementations of that write exist across this workspace. Three could
express it. ProjectionDatabaseOperations, the adapter this kernel injects into
every in-process projection handler, could not: every column was a bound
parameter, only conflict keys were held off the update arm, there was no
RETURNING clause and the method returned bool. So a handler running here could
not stamp the attestation at all, and omnimarket's delegation handler refuses
this adapter BY NAME rather than writing a row whose attestation column would
be NULL -- because a NULL there reads exactly like "nobody wrote this", which
is indistinguishable from "an unscoped principal wrote this". This removes
that refusal.

The DECISIONS come from omnibase_core's shared upsert plan: which column goes
on which arm, which SQL expression is admissible, which identifier is safe.
That plan is shared with omnimarket's three sync adapters precisely so the
closed set of expressions reaching a statement UNCAST cannot exist in four
divergent copies. The rendered SQL is deliberately NOT taken from it: this
adapter quotes every identifier and schema-qualifies the table, which no other
consumer does, so rendering here keeps the statement byte-for-byte what it has
always been. A test pins that byte-for-byte claim rather than asserting it.

THE BUG THIS NEARLY SHIPPED WITH, now pinned by its own test. Each domain
subclass guards writes -- the internal one rejects a canonical tenant_id, the
tenant one resolves attribution and scope, the catalog one enforces declared
access. Those guards hung off `upsert`, so an `upsert_returning` added beside
it inherited the BASE implementation and silently skipped every one of them.
Both entry points now route through one `_prepare_write` hook that each
subclass overrides, so a domain rule added later reaches both by construction.

A caller-supplied tenant override is refused rather than honoured. Scope here
is derived from the verified projection authority, or from the row's own
recorded attribution when none is bound. Accepting an override would
reintroduce exactly the attribution-from-the-caller path those guards exist to
close.

There is now ONE composed statement rather than two. A separate bool-only path
would be a second place for arm placement to drift, which is the failure this
ticket is unwinding; a plain upsert is the same statement with no RETURNING.

Pins omnibase-core to 0.47.9 (both declaration sites), which is the release
carrying the shared plan.

* chore(OMN-18159): rebind the runner image identity to the bumped core pin

The runner image records a shared_env_digest over the dependency set it bakes,
and the omnibase-core 0.47.8 to 0.47.9 bump changes that set. The build-smoke
gate refused the stale binding and named its own remedy; this is that
regeneration, not a relaxation.

The recomputed shared_env_digest matches the value CI independently computed
(4186fdd0a7819e8d3f4763eb), which is the check that the two sides agree about
what the image would contain rather than merely agreeing that a file changed.

* chore(OMN-18159): carry the core pin bump through its two dependent seams

A pin move in this repo is not a one-line change, and CI said so twice.

The fallback version matrix is derived from pyproject.toml and had drifted;
regenerated with scripts/update_version_matrix.py, which is what the repo's
own dependency-bump checklist prescribes and what the failing test names. The
matrix is never hand-edited -- pyproject.toml stays the single source.

The OMN-14628 released-core seam fixture freezes the exact symbols and field
TYPES infra consumes from the published core, precisely so a pin move fails
loudly rather than silently changing what infra compiles against. Re-frozen at
0.47.9. That it then passed on a version bump ALONE is the useful result: it
says this core release added a module without touching any symbol or type
infra depends on, which is the claim the seam exists to check rather than
assume. Its header comment had also gone stale a bump earlier, naming 0.47.7
while the frozen version said 0.47.8; re-pointed so the file stops
disagreeing with itself.
…ants (#3425)

* feat(OMN-18159): vendor 0039 and declare the aggregate views' read grants

The omnimarket half of Phase 1b(ii) re-groups the four delegation aggregate
SQL views on tenant_id, so a reader that derives its scope from a tenant
filter can finally read them. Until now they aggregated the whole table and
depended on the reader arriving with a row-level-security session scope, which
the runtime kernel's read seam cannot do -- it saw no rows and would have
published zeros that read as a quiet period rather than as a fault.

This is the infra half, and it lands first because the node-migration vendor
parity gate requires it: the source PR is refused until the vendored
counterpart exists here.

Three declarations, each demanded by a gate rather than guessed at:

The migration is vendored byte-identical and declared in the application
migration ledger with its hash, so a clean redeploy creates the grouped views.

The tenant_projection_writer principal gains SELECT on the four views. That
grant is DERIVED from the node contract's own db_io declaration by
scripts/generate_application_database_table_grants.py rather than hand-written
-- the first version of this change hand-edited the three instance files and
was both wrong about the schema and, worse, carried an indented rationale
comment that the generator silently destroys. That failure mode is documented
in the test that caught it, which records a previous author doing the same
thing; I did it anyway, which is its own lesson about documentation making a
hazard feel handled.

The views join the logical-to-physical schema bridge. They are tenant-domain
by contract and physically bare in public, exactly like the delegation_events
table they read and the savings read views beside them -- a view cannot live
in a schema its base table does not. Without that entry the derived grant
names tenant.<view> while the migration grants the bare name, and the
grant-delivery gate correctly reports the privilege as undelivered: a
capability the runtime advertises and the database refuses. Undelivered count
returns to its pre-existing ratchet bound of 1, which is the unrelated
residual the gate already records.

* fix(OMN-18159): declare the four delegation aggregate views in the legacy-migration manifest

The grant derivation reads the PINNED omnimarket contracts root, not
omnimarket dev. The four aggregate views' db_io declaration lives in the
omnimarket source PR, which cannot merge until the runtime finds the
matching topology grant -- and the pin cannot advance until that PR
merges. LEGACY_MIGRATION_TABLE_DECLARATIONS exists for exactly this
ordering, per its own docstring, so the four views are declared there
against the vendored migration that creates them.

The entries become redundant once the pin advances past the omnimarket
PR; removing them then is inert either way, because the derivation is
idempotent per relation.

* fix(OMN-18159): parenthesize the EXTRACT operand so the SQL domain gate can parse it

The changed-SQL schema-qualification gate matches `\b(?:from|join)\s+`
and reads the next identifier as a relation target, so the bare
`EXTRACT(EPOCH FROM created_at)` is parsed as a FROM clause naming a
table called created_at. Every other migration in this tree that extracts
an epoch already parenthesizes the operand -- 0001_create_gateway_link_health
and 079_create_capsule_effectiveness_view both do -- and that form is
byte-for-byte the same SQL to Postgres. The sibling delegation migrations
carrying the bare form predate the gate and are not re-read, because it
only checks SQL a change touches.

Vendored copy and omnimarket source stay byte-identical; the ledger
declaration hash is recomputed.
…entity (#3427)

The canary was landed knowingly-red because the omnibase-deploy runner had no
AWS identity at all and every leg died in boto3 before reaching the bastion.
It now assumes msk-bastion-canary-control-plane through GitHub OIDC
(omninode_infra aws/cluster-dev/msk_bastion_canary_iam.tf), which carries
exactly kafka:ListClustersV2 and kafka:ListNodes on omninode-dev-msk.

Data-plane credentials are still absent, deliberately. This workflow's own
header says the .201 gateway's existing IAM Roles Anywhere profile is the
data-plane identity and that minting a second one would be wrong, so the role
carries no kafka-cluster grant. The header now states the resulting expected
state plainly instead of the old blanket 'no identity' one: the resize
detector and the error chain go green, the golden leg stays red until that
profile is wired into the runner, and the job is red overall. That is a
narrower, still-real red.

The verify step uses boto3 rather than the AWS CLI because the deploy runner
image has no aws binary -- checked on the live container, not assumed. A
verify step failing on a missing CLI would be indistinguishable from a failed
assume.

Action pinned to the same configure-aws-credentials SHA omninode_infra uses,
account id from vars.AWS_ACCOUNT_ID rather than a literal, matching that
repo's convention. No continue-on-error anywhere: tests/ci/
test_msk_bastion_canary_workflow.py forbids it and would go red.

Local: uv run pytest tests/ci/test_msk_bastion_canary_workflow.py -q -> 7 passed.
…s stay inside RLS (#3428)

* fix(OMN-18159): realign the delegation aggregate views' owner, which 0039 dropped

0039 had to DROP and CREATE each of the four aggregate views, because
PostgreSQL refuses a CREATE OR REPLACE that changes the column list and every
one of them gained tenant_id. A DROP discards the view's OWNER as well as its
privileges. 0039 restored the privileges at its foot and did not restore the
owner, and the owner is not a cosmetic property: a view reads its base tables
with the privileges of the VIEW'S OWNER.

The role that reaches 0039 is not the role that created those views. 0032,
0033 and 0034 each issue RESET ROLE, so everything applied after them runs as
the migration runner's own identity, which on a lane and in this repo's
fixtures is a SUPERUSER. delegation_events carries FORCE ROW LEVEL SECURITY
precisely so that even its owner is filtered, but a superuser bypasses
row-level security unconditionally. After 0039 a read through any of the four
views returned EVERY tenant's rows whatever app.tenant_id held -- the
unscoped-serving leak OMN-18159 exists to remove, reintroduced by the change
that removes it.

Measured on a clean postgres:16-alpine, applying the node's directory under
SET ROLE <migrator>: without 0039 all four views are owned by <migrator> with
a NULL relacl; with 0039 all four are owned by postgres while
delegation_events stays owned by <migrator>. It also denied the migrator's own
reads, which is how it surfaced -- three tests failing with 'permission denied
for view projection_delegation_summary'.

0040 realigns each view's owner to the base table's, idempotently and
fail-closed. Additive rather than an edit to 0039: 0039 is applied on the .201
dev lane with a recorded content_sha256 and is declared in this manifest, so
check_migration_append_only.py refuses a modification and the forward runner
would raise a conflicting-checksum error. Additive is also what makes the
repair reach a lane that already applied 0039.

The omnimarket-side test and its positive control land with the source PR:
withholding 0040 fails the assertion, applying it passes.

* fix(OMN-18159): keep the fence detector's trigger phrase out of 0040's exception text

The forward runner refuses any node migration that enables forced row-level
security without a fence-manifest entry. It strips `--` comments before
matching, so prose is safe, but 0040's RAISE EXCEPTION text spelled the
trigger phrase inside a STRING LITERAL, which survives the strip. The runner
read a migration that only reads pg_class and issues ALTER VIEW ... OWNER TO
as one that turns on forced row-level security, and refused it:

    FATAL: node:node_projection_delegation:0040_... enables [it] but is not in
    the operator fence manifest ... NOTHING was applied by this migration.

This is the omni_home rule 15 class -- a gate that parses text firing on prose
that merely mentions its trigger -- and the fix is the one that rule gives:
name the property, never the literal. A fence entry would have been the wrong
answer twice over, because the migration does not enable anything.

Verified by running the Sanitized Legacy RDS Fixture Proof locally, which is
where CI caught it: FAIL before (fresh real migration pass 1), PASS after. The
detector's own rule, run against the file, now returns no match. The CI job
only prints the first 240 lines of the failing log and the FATAL sits past
that, so the local run was the only way to see it.

* fix(OMN-18159): make aggregate views invoker scoped
…t] (#3429)

Co-authored-by: omninode-bot <bot@omninode.ai>
* fix(OMN-18032): make bulk throttle operation-aware

* fix(OMN-18032): keep queue observations non-blocking

* test(OMN-18032): lock operation throttle policy

* fix(OMN-18032): normalize observation receipts

* fix(OMN-18032): derive receipt queue policy

* fix(OMN-18032): keep automerge arming queue gated

---------

Co-authored-by: jonahgabriel <jonah@omninode.ai>
onexbot-occ-writer Bot and others added 23 commits September 15, 2026 17:06
…t] (#3594)

Co-authored-by: omninode-bot <bot@omninode.ai>
)

* ci(OMN-18205): re-pin the two orphaned omniclaude reusable references

The Pin Reachability gate (OMN-15538) fails on every new pull request in this
repository. Two `uses:` pins name omniclaude `b441ff9d979e248ac20c51a00c135a3ce273cef2`,
which is omniclaude's "promote dev to main (#2017)" commit. The object still
exists, but tonight's omniclaude ancestry reconciliation left it reachable from
neither `dev` nor `main` -- both are now `2173a846`. An unreachable pin is the
exact time bomb that gate exists to catch: it resolves right up until it does
not.

Measured rather than assumed: a grep for that sha across omnibase_infra,
omnibase_core, omnimarket, onex_change_control, omniweb, omnidash and omnimemory
finds it in this repository only, in these two files. Older open pull requests
here still show the gate green because their runs predate the rewrite.

Both pins move to `2173a846258c05b77858c454176f22ff1e41a3aa`, the current head
of omniclaude `dev` and `main` -- reachable from a protected branch, which is
what the gate asks for.

The content change this picks up is in the same direction as the pin repair,
not incidental to it. Between the two commits, both reusables received the
OMN-18205 correction that the untrusted test is the FORK test and only the fork
test. The clause that also sent every same-repo pull request based on `dev` to
the public runner class pinned every private repository's required checks to
GitHub-hosted runners, which the 2026-09-14 operator ruling forbids. Pinning
forward adopts that fix here; a genuine fork pull request still resolves to the
public variable, so the OMN-16683 boundary is unchanged.

* docs(OMN-18413): re-pin the required-checks.yaml doc reference to the same tip

required-checks.yaml's deploy-gate row still named
fcbc85f5642f07e15fe1d6b1c168c96883d16ac3, the other omniclaude sha orphaned by
the same release-sync main move that stranded the two workflow-file pins in
the prior commit. That field is documentation-only (not consumed by the
context->job resolver per this file's own header), so it does not block the
Pin Reachability gate, but it described a dead object and is fixed in the
same change set for correctness.

Verified: check_pin_reachability.py reports all 20 cross-repo pins OK
(previously red on the two workflow-file pins alone).

* fix(OMN-18413): update the pinned-sha regression test for the re-pin

tests/ci/test_ci_workflow_resilience.py::test_reject_skip_token_gate_uses_no_checkout_reusable
hardcoded the old omniclaude sha (b441ff9d9) as its expected value, so the
re-pin in the prior commit broke it: assert 'OmniNode-ai/...f22ff1e41a3aa' ==
'OmniNode-ai/...35a3ce273cef2'. Confirmed via `git diff <old>..<new> --
.github/workflows/reject-deploy-gate-skip.yml` on omniclaude that no checkout
step was reintroduced between the two commits -- the invariant this constant
pins (the reusable performs no checkout of untrusted input) still holds at the
new tip, so this is a re-pin of the constant, not a relaxed assertion.

uv run pytest tests/ci/test_ci_workflow_resilience.py -q: 44 passed.
…hosted runner (#3596)

The per-run runner router (OMN-18031) decides between the lab fleet and
GitHub-hosted from live capacity. Six of its steps can move the answer toward
hosted and none of them read repository visibility, so the 2026-09-14 operator
ruling -- a private repository never runs CI on a GitHub-hosted runner -- was
unenforced in the one component whose whole job is choosing a runner class.

Both repositories whose seam shadow already names the fleet are private
(omniweb, omninode_infra, read live). Neither wires a route job today, so this
is a fan-out hazard rather than a live incident: the moment one does, the
saturation branch places its jobs on ubuntu-latest, where the billing
annotation means they do not execute at all.

The never-widen guard cannot catch it. That guard asks whether the answer
gained a fleet label the seam did not carry; a private-to-hosted downgrade
NARROWS, so it passes. This is the opposite invariant and needs its own check.

What lands:

- Visibility is read live, at decision time, by probe_repo_visibility. No list
  of private repositories is declared anywhere, because a declared list goes
  stale silently the first time a repository changes visibility. internal is
  private here: not publicly readable, billed the same.
- apply_visibility_rule runs on the ANSWER, after the never-widen degrade. That
  order is load-bearing: the degrade emits hosted, so running the rule earlier
  would leave the one path that ignores the elimination able to violate the
  ruling.
- A capacity downgrade is REVERSED (the fleet being busy is not a reason to
  break the ruling; the job waits). A trust reason is REFUSED, not reversed --
  reversing fork isolation would put untrusted code on the fleet, and reversing
  the hosted allowlist would put a registry push on the fleet it is isolated
  from. Blocked, no labels, exit 3, and the run stops.
- Unknown visibility gets the no-downgrade half and never the refusal half.
  Every public repository's seam reads hosted today, so refusing on a transient
  metadata read failure would convert it into a fleet-wide CI outage.
- visibility is a required keyword with no default: a defaulted one fails OPEN.

Both route call sites are wired, the reusable and the probe's inline fallback,
because a fallback that silently drops the rule is worse than no fallback. The
job token is enough -- repository metadata is readable under the metadata scope
every Actions token carries -- so this adds no credential and no scope.

Proven rather than asserted: removing the one call to apply_visibility_rule
turns 13 of the new tests red, and the never-widen cross-product sweep now
carries the visibility dimension (4536 combinations) with its non-vacuity
control intact.

Nothing here writes an Actions variable, flips a seam, or changes placement for
any public repository.
* ops(OMN-18411): cap the .201 omnibase-ci runner fleet at 60 (from 88)

A CI burst drove one-minute load to 100.9 on the 32-core .201 host, with
two runner containers pinned at 200% CPU. Idle runners cost nothing, so
the registered runner count sets the worst-case concurrent-job ceiling
on a fixed-core box -- 88 registered runners is what let a burst reach
that load.

Declares expected_count/burst_count as 60 in config/runner_fleet.yaml and
removes the omninode-runner-61..88 service and creds-volume definitions
from docker/docker-compose.runners.yml, so a future deploy-runners.sh run
does not resurrect them. The live .201 fleet was already brought down to
60 out-of-band (stopped container + gh api deregister, serial, busy-checked
immediately before each removal, no full-fleet recreate) -- this PR lands
the matching repo state so the next sanctioned deploy stays consistent.

Out of scope, untouched: omninode-deploy-runner, the two customer-plane
runners, and every .201 compose lane.

Ledger: docs/tracking/ROLLING_WORK_LEDGER.md:8380 (OPERATOR-CONSENT row).

* ci(OMN-18411): retrigger CI — the run predating the OCC autobind's Evidence-Source insertion could not be rerun in place
… own workflow commit (#3601)

A caller's `uses: ...@<sha>` pin governs which workflow FILE loads and nothing
else. Both of this repo's self-checkouts inside runtime-rebuild-trigger-reusable.yml
were written `ref: dev`, so under omnimarket's pin the YAML came from one commit
and the scripts it invokes came from whatever dev held at run time.

That skew is not theoretical. OMN-18387 made --projection-url a required argument
of scripts/ci/lab_pass_receipt.py probe-lane and updated its caller atomically in
this repo; omnimarket's pinned copy of the YAML kept invoking the new script
without it (run 35019423922, exit 2, refusing to emit an invalid receipt), which
produced a FAIL compose-dev lab-pass receipt.

verify-sibling-converged now checks out at `inputs.infra_ref ||
github.job_workflow_sha` -- the only expression meaning the commit this workflow
file was loaded from. trigger-rebuild's checkout deliberately stays on dev and
carries the greppable escape-hatch annotation with the reason: that clone's HEAD
is read as the published --primary-ref, so pinning it would rebuild the lane at a
stale infra revision. The residual it leaves is named in the file rather than
left implicit.

dependency-cascade.yml's dep-provenance checkout had no ref at all, the same
class, and is pinned the same way.
…hop 0 publishes under the chain's own identity (#3600)

* fix(OMN-18389): the chain canary waits for the writer it grades, and hop 0 publishes under the chain's own identity

Two residuals of chain-canary link 5, both surfaced once OMN-18398 made
`public.ledger_chain` non-empty on the .201 dev lane.

1. Read-before-write. The canary's link-5 leg read `ledger_chain` once,
   immediately, while `node_delegation_chain_ledger_effect` is a separate
   process dispatched BY the same terminal event that settles for up to
   20 x 250 ms before it writes. Run 35016830667's own correlation had five
   rows persisted at 21:16:56, one second AFTER the read at 21:16:55; on the
   20:42 scheduled run the gap was 55 seconds. The leg now polls, bounded by
   the readback window link 4 already holds open, keyed on the EXPECTED HOP
   COUNT and not on a fixed sleep: it stops the instant every declared hop is
   present, and an incomplete verdict now states how long it waited.

2. Identity replaced between hop 0 and hop 1. RuntimePatternBBroker
   publishes onex.cmd.omnimarket.delegate-skill.v1 with
   publish(topic, key, value, None), and both concrete buses mint a fresh
   ModelEventHeaders when handed none, whose correlation_id and message_id
   each default to uuid4(). event_ledger populates both columns from the
   HEADER, never from the envelope body, so a correlation-scoped read for the
   delegation's own identity found four hops of five and the canary reported
   ledger_chain_incomplete on a chain that had run end to end. This is the
   OMN-16931 shape one seam further out. The broker's two publishes now bind
   their headers to the envelope through header_identity_fields_from_envelope,
   the same helper publish_envelope uses; absence is preserved, so hop 0 still
   publishes with no parent_message_id and remains a checkable chain HEAD.

Red-first, against parent aa8c86c:
  tests/unit/runtime/test_chain_head_identity_omn18389.py  2 failed, 1 passed
  tests/unit/nodes/.../test_handler_chain_canary_ledger.py 2 failed, 29 passed
At this head both files are green (34 passed). The passing rows in each red
run are the positive controls: the envelope BODY already carried the right
correlation id, which is what makes this a body/header split rather than a
lost identity.

* test(OMN-18389): the head-hop identity seam lives in tests/integration

It drives the REAL subscribe/publish boundary of a live broker service
across EventBusInmemory -- broker.start() -> _handle_command_message ->
_publish_worker_command -- rather than calling a function, which is the
shape test_tenant_dimension_seam_omn16831.py already uses for the tenant
dimension on the same seam. No external service is required: the in-memory
bus mints the same default headers Kafka does, which is the behaviour under
test. Also satisfies the Integration Test Coverage gate, which the unit
placement did not.

* chore(OMN-18389): record the link-5 bounded wait in node_chain_canary_effect's contract

contract_version 1.4.1 -> 1.4.2, node_version likewise. No input model, no
output model and no verdict member changes: the leg that used to read
ledger_chain once now polls for it, keyed on the expected hop count and
bounded by the readback window link 4 already holds open. The
LEDGER_CHAIN_INCOMPLETE detail now states how many reads were made over what
window. Closes the contract-sync gate, which correctly flagged a handler
change with no contract update.
…key (#3603)

* ops(OMN-18415): give the omnibase-ci runner fleet the local-LLM HMAC key

The Hostile Review Gate's reviewer HMAC-signs every local model call and fails
closed per model when LOCAL_LLM_SHARED_SECRET is absent. It was on no runner in
the 60-runner omnibase-ci fleet, so every real adversarial review failed; a
separate shell defect then laundered that into a non-blocking degraded verdict,
so the gate reported reviews it had never performed.

Declares the key once on the &runner-env anchor (a mapping, so the merge key
reaches all 60 fleet services), fail-closed with :? rather than a soft default:
an empty value still satisfies the transport's is-it-set check and produces a
wrong signature, which is the same silent class this removes. The deploy runner
and the two customer-plane runners define their own environment blocks, do not
merge this anchor, and stay credential-free.

No value is committed. The host reads it from docker/.env in the compose project
directory, which resolves identically for the runner-monitor auto-bounce cron
and for deploy-runners.sh, and which is absent from SYNC_PATHS so a repo-side
rsync can neither publish nor blank it.

* ops(OMN-18415): add the serial, busy-checked rolling mode the env change needs

Container env is frozen at creation, so the declaration in the previous commit
reaches a live runner only through a recreate. The default deploy path recreates
every service in one compose call and takes the fleet down at once; --soft never
recreates and cannot carry an env change at all. The procedure that works, one
service at a time skipping any runner executing someone else's job, was a
hand-typed loop, which is the shape that kills a live job when a busy check is
subtly fail-open.

--rolling recreates one named service per compose call, never sweeps orphans,
and touches a runner only when GitHub's busy flag and the container process list
BOTH say idle. An API error, an unreadable process list, or a disagreement is
UNKNOWN and skips. Idleness is re-checked immediately before the stop, because a
job can start between selection and recreate. A runner that does not come back
halts the roll rather than cascading; busy runners are retried in later passes
and named in the summary if they stay busy.

The roll asks GitHub for no registration handle: a recreate restores the
runner's cached registration from its per-runner named volume.

* ops(OMN-18415): add the canary limit to the rolling fleet deploy

A fleet roll starts with one runner, proven, before the other 59 are touched. --limit=N stops after N successful recreates and says so; the default of 0 means the whole fleet, so omitting it can never silently roll a subset and report a completed roll.
…_ONLY_KEYS (#3602)

* fix(OMN-10856): classify ONEX_IMAGE_DIGEST/ONEX_DEPLOYMENT_SHA as K8S_ONLY_KEYS

The three onex-dev runtime Deployments bind ONEX_IMAGE_DIGEST and
ONEX_DEPLOYMENT_SHA as kustomize-replacement placeholders (omninode_infra#1487),
which the reverse-direction env-parity walk flagged as unclassified k8s-only
keys since nothing in this repo's compose lanes binds them. Compose lanes
identify a build by BUILD_SOURCE/EXPECTED_BUILD_SOURCE (release vs workspace),
not by a pinned OCI digest or deploy-time git sha, so there is no compose
counterpart to bind — a workspace build has no image digest until
BUILD_SOURCE=release publishes one, and compose has no kustomize-replacement
analogue. Classified both keys in K8S_ONLY_KEYS with the value-backed
justification the gate requires.

Verified: test_k8s_bound_keys_are_bound_in_compose passes on the parent of
omninode_infra#1487 (cae11c25) and fails on it (51579dad) confirming the
introducing commit; passes again with this classification against current dev.

* fix(OMN-10856): clarify blank build identity placeholders
…client family, so an MSK IAM lane can be asked about consumers (#3606)

* fix(OMN-18418): the delegate lane probe uses the runtime's own Kafka client family

The fail-closed gate in front of every dispatched delegation resolved
consumer-group liveness through a synchronous confluent admin client, while
every producer, consumer and admin client the runtime wires is aiokafka via
build_aiokafka_auth_kwargs. Token-callback mechanisms are not expressible as
librdkafka config entries, so build_confluent_auth_config refuses AWS_MSK_IAM
by design -- and that refusal arrived at the gate as UNKNOWN, which fails
closed, so onex-dev refused every in-cluster 'onex delegate --bus kafka'
before publishing with a message about the lane rather than the client.

One client resolution path, not a second set of credentials: the probe now
resolves its client from the same ModelKafkaEventBusConfig and the same auth
builder the runtime uses, so the MSK token callback has one implementation.
Fail-closed semantics are unchanged -- unreachable broker, partial listing and
unanswerable question all still raise rather than returning an empty tuple.

* test(OMN-18418): prove the lane probe against a real auth-required broker

The unit tests pin what live_consumer_groups CONSTRUCTS. A fake admin client
cannot prove the other half: that the new client gets the right ANSWER off a
live cluster. That half is where a client-family swap goes wrong quietly --
the group state is not on aiokafka's listing at all, so the probe narrows by
name and then describes, and the broker spells the state 'Stable' where the
confluent enum spelled it 'STABLE'.

Runs in the existing customer-path boundary job against its auth-required
Redpanda, with a real consumer joined to a real group: a negative control on a
topic nothing is bound to, the positive, and a discriminating control that the
same group id stops being reported once its only member leaves. Residual
stated in the module: the harness is SCRAM, not AWS_MSK_IAM.
…instead of its own (#3605)

* fix(OMN-18414): the evidence closer reads the declared Gate: grammar instead of its own

One `Gate:` key in a Linear description was read by two parsers whose
accepted vocabularies did not overlap at all. The authoring authority is
omniclaude's ticket-creation admission guard, which mandates four forms.
This consumer accepted exactly one form that was not among them. So every
ticket the guard admitted arrived here as an unreadable declaration and was
held, and the form this consumer required would have been refused at
creation.

Four Done-candidates were held on nothing else at the 2026-09-15T21:34Z
dry-run tick: OMN-18403, OMN-18387, OMN-18368 and OMN-18365.

The grammar is now declared once, in
`src/omnibase_infra/contracts/gate_binding_grammar.json`, and both consumers
read it. This module spells no form pattern of its own; a test reads its
source and fails if one reappears.

Exactly one of the five forms is a proof pointer. A `workflow_run` binding
still resolves to its newest completed run's conclusion and still holds on
anything but success, unchanged from OMN-16106. The four traceability forms
reach the stage after the probe, which is the absence of a proof pointer
rather than a pass: every other conjunct of the sweep still applies. The
contract's `$comment` carries the live evidence for why resolving those four
to a probe is refuted rather than merely unimplemented.

Text matching no declared form is still held, and the hold now names the
forms that would have been readable.

* test(OMN-18414): integration coverage for the gate-binding conjunct

Drives `_gate_probe_verdict` end to end over every declared form, with only
the `gh` boundary stubbed: contract, resolver, handler wiring and the read
are exercised together rather than the pure decision alone.

Carries the cross-repo half from this side too. The transcription
omniclaude's admission guard reads must be byte-equal to the declaration
here, and omniclaude pins the same equality from its side. Two assertions
rather than one is deliberate: the failure this ticket closes was a grammar
that drifted while each repository's own suite stayed green. It skips with a
stated reason when no omniclaude clone is reachable, never silently.

* docs(OMN-18414): record the gate-binding grammar change in the node contract

The contract-sync gate is right: the handler's rule changed, so the node's
own contract has to say so. Adds the OMN-18414 entry, bumps node_version to
1.15.0 and the contract version to 1.10.0, and moves the gap-comment
fingerprint's pinned version with it — the fingerprint carries the version
precisely so a gap comment cannot describe a rule the closer no longer
applies.

* fix(OMN-18414): declare the gate_binding module in the test-selection adjacency map

A new top-level module under src has to declare which modules' tests a change
to it pulls in, or the governed selector cannot narrow safely and the loader
test refuses the tree. Its only consumer here is the evidence autoclose
sweep, so a change runs the node tests. The contract file it reads is also
transcribed into omniclaude, whose suite pins the byte-equality from the
other side.
…clared by its contract (#3604)

* feat(OMN-18412): the CI runner routing decision is a COMPUTE node, declared by its contract

Operator direction, 2026-09-15: contract-driven, OmniNode primitives, no
one-offs. The per-run routing decision was a 900-line script with its
thresholds in a config file. It is now node_ci_runner_route_compute: a
contract, a declarative node, and one stateless definition-B handler taking a
typed request and returning a typed decision.

WHERE IT LIVES AND WHY. omnibase_infra owns the CI routing surface already --
the fleet inventory, the hosted-workflow list, the routing audit, the fleet
canary and the summary gate are all here, and the repository holds 125 nodes
including the compute archetypes. omnimarket owns the runtime-deployment
domain, which is a different question from where a CI job executes.

WHAT THE CONTRACT DECLARES: the label sets, every threshold, the reasons a
hosted verdict may be reversed for a repository that may not run hosted, and
the private-repository refusal. Parsed into a typed policy whose fields are all
required, so a contract missing one fails at load naming it rather than at the
first run that needed it. No threshold is read from an environment variable and
none is restated in the old config file -- two copies are two policies that
agree until someone edits one, and a test now fails if one reappears.

THE FLEET SIZE IS READ FROM THE FLEET INVENTORY, not restated. That is not
tidiness: the degraded floor was the literal 60, chosen as two thirds of the
88-runner fleet of the time, and the fleet was capped to 60 the same day this
was written. The literal had become equal to the whole fleet, so one runner
going offline would have refused the fleet on every run and reported it as an
ordinary return to GitHub-hosted. As a fraction of the declared inventory the
same two thirds tracks the cap.

The script keeps exactly what a boundary should: the probes, request assembly
and the CLI. It cannot drift from the decision because it no longer contains
one. Its node imports are function-local by necessity -- the self-hosted
lab-load probe imports this module on the fleet image's bare interpreter, which
has neither PyYAML nor pydantic.

An operator force input lands with it, subject to every safety rule rather than
a bypass of them: fleet cannot widen past the seam ceiling or take a fork pull
request, and hosted is still refused for a private repository. It exists so the
kill switch a static seam value provided survives per-run routing.

Not used: the sanctioned local node CLI. Measured today, it refuses to dispatch
unless the workspace venv carries every sibling package installed from git,
which on a hosted runner is minutes of install on the critical path of every
run for a decision that takes milliseconds. The same handler executes either
way; only the dispatch machinery differs. Recorded in the workflow beside the
invocation rather than left as an unexplained choice.

Proven, not asserted: 266 tests pass across the decision, the wiring, the
incident replay, the routing audit and the new contract-binding suite. Four
live dry runs through the node -- private plus saturation gives the fleet,
public with identical inputs gives hosted as the control, public with an idle
fleet and a fresh lab record gives the fleet, and the same run forced hosted
gives hosted. The contract suite asserts the declared reversal reasons are real
enum members and exclude every trust reason, which is the one defect nothing
else can see: a misspelling there silently stops reversing.

No Actions variable is written. Routing stays inert wherever a seam reads
hosted.

* test(OMN-18412): exercise the routing decision as CI invokes it

The unit suites cover the handler and the contract suite covers the node's
shape. Neither covers the seam that decides real placement: a subprocess
reading the committed contract and fleet inventory off disk, writing the four
lines a workflow consumes, and signalling a refusal through its exit status.

That seam has broken twice in ways every unit test survived. A JSON seam value
interpolated into a quoted shell argument lost its inner quotes and reached the
module as non-JSON, so every decision came back unparseable and looked exactly
like a correctly inert mechanism. And a refusal that exits non-zero is worthless
if the workflow's fail-closed floor overwrites its labels first.

Six cases, each against the real contract: the fleet path, saturation, the seam
surviving the shell with an unparseable-seam positive control so the assertion
discriminates, a private repository refused a hosted downgrade, a refusal that
exits 3 and still leaves both a record and the labels line the floor checks
for, and the operator override in both directions.

* fix(OMN-18412): an expression in a workflow_call output description fails every caller at startup

Measured on run 35037012235: ci.yml and the routing probe both concluded
failure with ZERO jobs, and the run title fell back to the file path instead of
the workflow name. That is the whole visible symptom. No check reports on a run
that never started, so the pull request showed 62 green checks, no red one, and
a merge state of BLOCKED with nothing to point at.

The cause was an illustrative snippet I put in the new runs_on output's
DESCRIPTION. Actions evaluates expressions inside a workflow_call outputs
block, where the needs context does not exist, so the reusable failed to
compile and took every calling workflow with it.

The description is now prose. A test asserts no workflow_call input or output
description in either routing workflow carries an expression, because this
failure is invisible to every check-based gate by construction.

* fix(OMN-18412): the route job must sync the environment, or it reports green while inert

Measured on run 35039101488. The route job installed two pinned packages --
the models need only pydantic and a YAML parser -- and then failed with
`No module named 'omnibase_core'`, fell through to its fail-closed floor, and
reported SUCCESS carrying `reason=probe_error:module_unavailable`. Both
consumer jobs scheduled hosted. Nothing was red. The mechanism was completely
inert and the only evidence was a reason string in an artifact.

The cause is not this node: importing anything under the package executes its
__init__, which eagerly pulls the database and broker surfaces, so no node can
be imported without the project's dependency set. Loading modules by file path
around that would be bespoke plumbing for one job; syncing the environment with
the repository's own shared setup action is what every other job here does.

Both call sites now use that action and run the module inside the synced
environment. The job's ceiling goes from 2 to 5 minutes -- the bound the wiring
suite already codified -- rather than wider, because this job is on the
critical path of every consuming run and a sync that stops fitting should fail
rather than silently cost every pipeline minutes.

A test asserts the setup step exists and precedes the decision at both call
sites. That is the only durable guard: the floor exists precisely so a broken
router still hands the run a usable runs-on, which is why nothing else can see
this failure.

* chore(OMN-18412): register the routing node's entry point

A contract-backed node is discoverable through the onex.nodes entry points,
and the entry-point checker holds the generated section to the set of node
directories on disk. Adding the node without its entry point left that test
red -- correctly: a node the runtime cannot resolve by name is a node only its
own importer can reach.

Generated with the repo's generator against this worktree rather than
hand-edited, so the section stays byte-identical to what the checker
regenerates.

* chore(OMN-18412): refresh the dispatch-parity baseline for the new contract

The committed baseline records the live dispatch corpus, so adding a contract
makes it stale by construction. Regenerated with the harness rather than
hand-edited.

The diff is the whole evidence that this node adds no dispatch surface: the
contract count moves 143 to 144 and nothing else changes. No new route, no new
subscribe topic, no new dispatcher -- which is what a pure compute node with no
event_bus block should look like, and would not be true if the node had
acquired one.
… the shared host (#3609)

Every lab-pass emitter in this repository and in the sibling reusable runs on
the same omnibase-deploy host and wrote its receipt and checks to a FIXED
absolute path under the shared temp root. mkdir -p on an existing directory is
a no-op, so a receipt.json from an earlier job survived into the next one; when
that job's emit did not overwrite it, the presence assertion found the stale
file, passed, and upload-artifact published it under THIS job's sha-keyed name.

Measured: omnimarket run 35035178406 uploaded artifact 10424013456 named
lab-pass-receipt-compose-dev-bc21f733... (an omnimarket commit) carrying a
payload sha of a7ea64f... (an omnibase_infra commit), with a deployed_revision
check citing check_dev_lane_staleness.py -- this workflow's emitter shape, not
the sibling one's sibling_revision / check_lane_sibling_revision.py.

The direction was safe: evaluate_gate cross-checks the artifact name against
the payload sha and refuses the disagreement. The cost is that the sibling sha
can then never receipt PASS, and rule 24(b) refuses it for staging.

Two changes, both fail-closed:

  * each emitting job claims RUNNER_TEMP/lab-pass-<run id>-<attempt>-<job> and
    removes it before use, so no file can survive from another job.
  * a new 'verify' subcommand asserts the receipt on disk carries this job's
    own sha and lane, run as a separate step before every upload. Kept out of
    'emit', which writes the file and so can only agree with itself.

config/ci_evidence_policy.yaml follows the three shared-host entries. The
boot-gate entry keeps its literal path deliberately: that job runs on an
ephemeral GitHub-hosted runner where no second job shares the filesystem.

Tests parse the workflow YAML and pin the properties rather than the spelling:
no lab-pass path under a shared temp root, every path derived from a per-run
variable, the uploaded directory is the one emit wrote into, and every emitter
verifies before uploading. Positive control: all eight fail against the
pre-change workflows.
…infra forward tree (#3608)

* fix(OMN-18079): vendor the overlay provider backfill into the infra forward tree

Companion to omnimarket#2579, and it lands FIRST: omnimarket's
node-migration-vendor-parity-gate refuses a node migration that has no
vendored counterpart in omnibase_infra@dev, so the mirror must exist here
before the source PR can merge.

Byte-identical mirror produced by scripts/sync-node-migrations.sh with
OMNIMARKET_SRC pointed at the source branch (the script's own first
resolution input). The forward-migration runner reads only
docker/migrations/forward/, so without this file the backfill would sit in
the node's source tree and never touch a deployed database -- the same
half-landed shape that let 0004 add a NULLABLE provider column and strand
every row already in delegation_routing_tenant_overlay.

The application-migrations.tsv row is what lets this land ahead of its
source: sync-node-migrations.sh --check treats a declared vendored file as
preserved history rather than drift. Same path omnibase_infra#3581 took for
0004 six hours earlier.

* fix(OMN-18079): allow exact paired migration source PR

* fix(OMN-18079): bind paired migration source to repo owner

* fix(OMN-18079): allow paired migration source without ref trailer
…nex-dev staging (#3611)

* feat(OMN-17057): a sibling merge delivers its own named revision to staging

An omnimarket merge could not cause a delivery at all. It rode along on
whatever the next omnibase_infra push happened to build, and that candidate
clones every sibling at branch dev at that instant -- so the sibling revision
in the image was never CHOSEN, could not be predicted, and no sibling merge
could prove delivery of its own sha. Measured on delivery run 35013603978: the
candidate vendored omnimarket 3038d1da, simply where that branch pointed 18
minutes after an unrelated commit merged here. Twelve omnimarket commits had
landed since by the time this was read.

The primitive already existed and CI was not reaching it. deploy_source_ref.py
has taken --ref plus --fallback-ref since OMN-17135, resolving the primary in
whichever repository carries it and taking the fallback everywhere else. The
candidate build passed the same branch name as both.

Trigger. runtime-rebuild-trigger-reusable.yml gains a terminal job that needs
the existing convergence job, so it fires only after the .201 dev lane was
proven to vendor the merged sibling sha and its compose-dev lab-pass receipt
was emitted. It sends one repository_dispatch carrying that repo and revision.
Because it lives in the reusable, every repository the runtime image vendors --
omnimarket, omnibase_core, omnibase_compat -- acquires the path with no
per-repo code. omniclaude is deliberately outside the family: it is not
vendored into the image, so there is nothing for it to pin.

Naming. build-workspace-candidate-runtime.yml gains pinned_repo and
pinned_revision, both empty by default so the push path is unchanged. When set,
DEPLOY_REF becomes the commit and DEPLOY_SIBLING_FALLBACK_REF the branch. The
pinned commit is fetched by sha into its clone, and after staging the vendored
sha in sibling-vcs-provenance.json is compared against the requested one -- the
same file the image bakes and the staging deploy later reads out of a running
pod (OMN-17926). The build manifest records pinned_sibling so a chosen revision
is distinguishable from a branch that drifted to the same commit.

Gating. lab-pass-gate keeps asserting a PASS receipt for this repo's sha, and
when a pin is present additionally reads the sibling repository's own
compose-dev receipt with a token scoped actions:read on that one repository.
dispatch-to-staging already needs lab-pass-gate under default success(), so a
missing, unreadable, malformed, name/payload-disagreeing or FAIL sibling
receipt leaves the announcement unfired.

Fail-closed throughout: a half-specified pin, a non-sha revision, a repository
the build does not stage, a commit its source does not contain, a vendored sha
that disagrees with the request, an absent sibling receipt and absent
announcement credentials are each a refusal. No force input, no skip input and
no if: override is added anywhere, and a test over the parsed inputs refuses
one later.

Concurrency. The delivery group gains a segment naming the pinned repo.
github.ref on a repository_dispatch resolves to the default branch, so without
it the next ordinary push would silently cancel a sibling's delivery.

The announcement payload carries pinned_sibling to omninode_infra, which
already resolves what each running image actually vendors; what it never had
was the other half of that comparison -- which revision was intended.

Tests parse the workflow graph and pin properties, not spellings. Positive
control: 12 of the 16 fail against the pre-change workflows, and the 4 that
pass are invariants that were already true.

* test(OMN-17057): bring the three reusable-workflow guards onto the new delivery job

Three existing guards asserted properties of a two-job workflow and failed on
the third. Each is extended by deciding what the new job needs, never by
excluding it.

  * the secrets guard now asserts the App pair alongside the SASL pair, and
    asserts the REQUIREDNESS of each: SASL required, the App pair optional at
    the interface so an existing caller keeps compiling, and refused at run
    time by the announcement job itself. Marking them required would break
    every caller in the commit that adds them; skipping on their absence would
    report a delivery that did not happen.

  * the self-checkout guard now maps the new job to None -- it runs gh and jq
    and nothing from the tree, so a checkout would be ceremony that later reads
    as a dependency. Listed rather than excluded, so adding a job still forces
    the decision the test exists to force, and a None job is asserted to
    reference no local action and no repository script.

  * the CI-python-environment guard now scopes the setup-action requirement to
    jobs that RUN Python. The pip-install prohibition stays universal. The
    discriminating half is the else branch: a job with no setup step is
    asserted to invoke no Python at all, so "forgot the setup action" cannot
    pass as "does not need it". Verified against the live graph -- all six
    pre-existing jobs across the three workflows classify as Python jobs with
    setup present, and only the new job classifies as neither.
* chore(deps): bump omnibase_core to 0.47.15

Automated dependency cascade from OmniNode-ai/omnibase_core release 0.47.15.

Triggered-by: https://github.com/OmniNode-ai/omnibase_core/actions/runs/35044074866

* chore(OMN-15639): carry the core 0.47.15 pin through the three surfaces that follow it

The cascade bump moved pyproject and uv.lock only. Three surfaces are derived
from that pin and go stale in the same commit:

- docker/runners/runner-image.lock.json: the runner-image smoke gate recomputes
  shared_env_digest from the resolved environment and refuses a stale record.
  Regenerated with scripts/ci/runner_image_identity.py --mode generate; the
  regenerated digest equals the value CI recomputed.
- version_compatibility._FALLBACK_MATRIX: regenerated by
  scripts/update_version_matrix.py, whose --check is the canonical drift gate.
- the core-release seam fixture: it freezes the released core version and
  compares it to the installed omnibase_core.__version__.

The OMN-17802 fallback boundary test moves with the floor so it keeps naming the
version immediately below the new pin.

Evidence-Ticket: OMN-15639
Evidence-Source: OCC#9743

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: jonahgabriel <jonah@omninode.ai>
…t] (#3607)

Co-authored-by: omninode-bot <bot@omninode.ai>
…e replay grades the tree it declares (#3610)

* fix(OMN-18419): the delegation chain records its causal edges, and the replay grades the tree it declares

chain-canary link 5 reported `ledger_replay_failed (4/5 links proven)` on a
chain that had run end to end. Two independent defects, one verdict.

1. The causal edge was dropped at two publish seams.

   `RuntimePatternBBroker._publish_worker_command` builds a fresh envelope for
   the route's command topic and published it as a chain HEAD every time. It
   publishes TWO hops of this chain: the head, from the HTTP ingress, and
   `onex.cmd.omnibase-infra.delegation-request.v1`, from
   `service_delegation_dispatch_port` -- which runs INSIDE the delegate-skill
   handler's dispatch, with the consumed envelope bound on the engine's
   contextvar. It now records that envelope. Absence is preserved: the ingress
   hop consumes nothing and still publishes with no parent, which is the
   checkable statement "head" the replay now checks rather than assumes.

   `MessageDispatchEngine` hands a dispatcher the JSON-safe materialization of
   the envelope and binds the typed one beside it "for transport identity (for
   example envelope_id)" -- its own words. The state_io wrapper read
   `envelope_id` as an ATTRIBUTE off that mapping, always None, so every outbox
   entry seeded its causation with the CORRELATION id via the fallback. Proven
   from the wire: the routing intent's deterministic id on run 35037024216
   re-derives as `uuid5(correlation, f"{correlation}:ModelRoutingIntent:0")`
   and not from the `delegation-request` envelope
   `f1cc9a51-ef02-40f4-b2e7-81e5b57942ed` that caused it. `_publish_outbox_batch`
   then dropped even that seed from the envelope it published. Both are fixed;
   the completion-bound sweep's deliberate self-seed still publishes no edge,
   because a correlation id is not an envelope id.

2. The chain is a TREE and the replay re-derived a LINE.

   `chain_replay._replay_one_hop` compared each hop's recorded parent against
   `observed[index - 1]`. `delegate-skill-completed` records the delegate-skill
   COMMAND as its parent -- it is a consequence of consuming that command, not
   of the routing decision that precedes it in time -- so a correct edge graded
   red. `chain_topology` is now an ordered list of hops that each declare their
   parent TOPIC (`ModelDeclaredChainHop`), and the replay grades the recorded
   edge against the envelope observed on the declared parent. Nothing about
   this canary is special-cased: a line is a tree whose every parent is the
   preceding hop. Tier 2 still reads the ORDER; tier 1 now reads the PARENT
   RELATION. A hop no declaration reaches replays RED with a stated reason
   rather than falling back to the positional guess -- the same rule tier 2
   already held.

The declaration also gains the `delegation-request.v1` hop, which was always
traversed and never declared, and `node_ledger_projection_compute` (contract
1.4.0) now records it -- that node is the only writer of `public.event_ledger`,
so without it the routing request's recorded parent named an envelope no
correlation-scoped read could resolve.

Evidence, read read-only on the .201 compose dev lane, correlation
41235987-425c-481b-b2e3-8970083ce512 (run 35037024216):

  delegate-skill.v1              6c9dc22f-...  parent=<null>  (head, correct)
  delegation-routing-request.v1  841f9c10-...  parent=<null>  (not a head)
  routing-decision.v1            223b9623-...  parent=841f9c10-...
  delegate-skill-completed.v1    3b1b8925-...  parent=6c9dc22f-...  (the branch)

RED-first, against parent 76c8658:
  tests/integration/runtime/test_chain_causal_edge_seam_omn18419.py
    3 failed, 1 passed -- the worker-command edge fails on
    `assert None == UUID(...)`; the ingress-head case is the positive control
    and passes on both sides.
  Replaying those four rows through the parent's own replay reproduces the live
  verdict exactly: hops 1 and 3 red, hops 0 and 2 green.
At this head both files are green (15 + 4 passed), the post-fix five-hop wire
shape grades 5/5 green, and the four rows as RECORDED still grade red -- the
fix does not paper over the evidence that produced the defect.

mypy --strict clean over 3044 files; `pre-commit run --all-files` green.

* test(OMN-18419): regenerate the dispatch-selection parity oracle for the fifth chain hop

Mechanical. Adding onex.cmd.omnibase-infra.delegation-request.v1 to
node_ledger_projection_compute's subscription changes that node's
content-hashed dispatcher set, so the committed Mode-A oracle drifts by
construction: 51 new probes, one new topic-scoped dispatcher, and the existing
ledger-projection dispatchers' topic sets. Verified structurally rather than by
line count -- no contract other than node_ledger_projection_compute appears in
the diff, and the header's corpus counts move by exactly the one topic.

  before: 101 distinct subscribe topics, 923 probes, 122 dispatchers
  after:  102 distinct subscribe topics, 974 probes, 123 dispatchers

Regenerated with the harness the test names in its own failure message, not by
hand. tests/integration/runtime/test_dispatch_selection_parity.py: 10 passed.

* test(OMN-18419): the refusal-path integration harness declares hops, not bare topics

Missed in the first commit: this file constructs HandlerDelegationChainLedger
with declared_chain= directly, so the declaration-shape change reached it too.
It failed CI split 2/15 on run 35043176426 with 'Expected mock to have been
awaited' -- the handler raised on the bare strings before reaching the persist
edge the test asserts on, which is the right failure and the wrong reason.

The harness declares a LINE, which is what its _observed builds; the tree cases
live beside the grading rule in tests/unit. What this file exercises is the
refusal path through the real wiring, which is indifferent to the topology's
shape.

tests/integration/nodes/node_delegation_chain_ledger_effect/: 4 passed.
…se the batched describe does not decode against MSK (#3613)

* fix(OMN-18418): describe lane-probe candidates one per request, not batched

Measured in-cluster on onex-dev 2026-09-16, after the client-family fix
reached the lane: the AWS_MSK_IAM refusal is gone and the probe now
authenticates and lists all 634 consumer groups, but the batched describe --
aiokafka puts every group sharing a coordinator into one DescribeGroupsRequest
and gathers the responses -- died with 'ValueError: Buffer underrun decoding
string' and took the broker connection with it, so the delegate gate refused
a healthy lane again under a new signature.

The same three groups bound to the delegate command topic, described ONE AT A
TIME against the same cluster in the same pod, returned cleanly: Stable with 1
member, Empty with 0, Stable with 1. Serial describes over a handful of
narrowed candidates cost nothing and are the shape proven to work on the lane
this gate guards. Not a downgrade: an error on any candidate still propagates
to UNKNOWN, and the shape is pinned by a test so the next edit cannot quietly
re-batch it.

* test(OMN-18418): cover lane probe serial describe integration

* test(OMN-18418): prove serial describe failure semantics

* fix(OMN-18418): bound serial consumer group describes
… the key (#3614)

Provisioning the signing key alone moved the reviewer's failure three seconds
later in the same call path and changed nothing a reader could see.
MixinLlmHttpTransport rejects a local LLM call before any HTTP is attempted
unless the destination falls inside LLM_ENDPOINT_CIDR_ALLOWLIST, and it refuses
to default that variable: a trust boundary nobody declared is not a boundary.
That variable was absent from every fleet runner too.

Measured, not inferred: reproducing the runner's exact environment (signing key
present, the rest of the LLM family absent) fails with "LLM_ENDPOINT_CIDR_ALLOWLIST
is required but not set" and exit 1, which is what the gate's suppressed stderr
was hiding. With both present, the same command reports the model succeeding.

Declared the same way as the key, on the shared anchor, fail-closed, reaching
only the 60 fleet services. It is not a secret, but it is still :? rather than a
literal default, because mirroring the library's own refusal is the point and a
host whose boundary differs must say so rather than inherit this lab's.
…ot the ambient env (#3617)

* fix(OMN-16871): resolve the delegate broker from the selected lane, not the ambient env

`onex delegate --bus kafka` resolved its broker ADDRESS from the ambient
`KAFKA_BOOTSTRAP_SERVERS`. On the launching Mac that variable names the .201
stability-test lane, so ad hoc delegations from a developer shell published
onto a governed proof lane; that lane's consumer-group list carried four
`local.omnibase_core.runtime_local_terminal_run_*` groups from CLI processes
on that Mac when the finding was re-verified.

Bus TYPE was already config-resolved (OMN-17304: env may bootstrap where
config is found, never what the transport is). This extends the same ruling
to the address: the env path is deleted rather than repointed.

- `--lane <id>` resolves the broker and the declared transport from
  omnimarket/config/ci_bus_lanes.yaml under the workspace root, through the
  shipped `load_lane_transport` reader. No second declaration, no second
  parser.
- With the resolved bus on kafka, exactly one of `--lane` or
  `--kafka-bootstrap` is required. Neither is a refusal naming the missing
  selection and listing the declared lanes; both together is a refusal.
- `build_backend_overrides` refuses a kafka bus with no address, so no
  argument combination yields a broker this process did not resolve.
- The deployed-lane locus probe is handed the RESOLVED address, and refuses
  without one, so it can no longer probe one broker while the publish goes to
  another.

Help text no longer documents the env var as the resolution path. SASL
credentials are unchanged and still arrive through `KAFKA_SASL_*`; what left
the environment is the address.

* test(OMN-16871): CLI-level integration proof that a shared-bus run must name its lane
…3618)

A runner busy through every retry pass is reported and left alone, which is correct. Without a way to return to it, clearing that residual meant a hand-typed compose call, i.e. the exact recipe this mode exists to replace. --only=NAME converges one named fleet service and fails closed on a name the compose file does not declare, because rolling nothing and printing a completed count reads exactly like a converged fleet.
)

* ci(OMN-18205): re-pin merge-hold gate to the seam-routed reusable

Re-pins this repo's merge-hold-gate job to omnimarket#2575's dev squash
commit, whose evaluate job routes through the trusted-CI seam instead
of an unconditional GitHub-hosted runner.

Evidence-Ticket: OMN-18205

* ci(OMN-18205): document merge-hold pin provenance

* ci(OMN-18205): rely on pin reachability gate
Product code merged via #3481; resolve all conflicts in favour of dev.
Delegation chain topics (OMN-18398/18419) already on dev — remove
duplicate allowlist entries from the branch's earlier additions.
@Patel230
Patel230 force-pushed the lakshman/omn-16964-ledger-audit-topics branch from cad3021 to fcc6bd7 Compare September 16, 2026 07:20
@Patel230

Copy link
Copy Markdown
Contributor Author

Closing: branch history corrupted by filter-branch rewrite. Recreating fresh.

@Patel230 Patel230 closed this Sep 16, 2026
ctx.verify_mode = ssl.CERT_NONE
sock = socket.create_connection((BASTION_HOST, BASTION_PORT), timeout=timeout)
try:
wrapped = ctx.wrap_socket(sock, server_hostname=sni)
the reader must not collapse them.
"""
path = record_path(state_dir, sha)
if not path.exists():
path = record_path(state_dir, sha)
if not path.exists():
return None
payload = json.loads(path.read_text(encoding="utf-8"))
jonahgabriel added a commit to OmniNode-ai/onex_change_control that referenced this pull request Sep 16, 2026
…ibase_infra#3551 (#9588)

* fix(OMN-16964): merge current OCC dev evidence for PR 9588

Onex-Lane: ledger-chain-writes-nothing-1625
Onex-Session: 0b9fe03e95e84563a664fbf9b4ce925a

* fix(OMN-16964): keep existing receipts immutable for PR 9588

Onex-Lane: ledger-chain-writes-nothing-1625
Onex-Session: 0b9fe03e95e84563a664fbf9b4ce925a

* fix(OMN-16964): drop unproven pending AC1 live proof

Onex-Lane: ledger-chain-writes-nothing-1625
Onex-Session: 0b9fe03e95e84563a664fbf9b4ce925a

---------

Co-authored-by: jonahgabriel <jonah@omninode.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants