Skip to content

fix(OMN-15732): break infra-dev merge deadlock -- exempt vendor files from OMN-15361 legacy-schema lint - #2683

Merged
jonahgabriel merged 3 commits into
devfrom
jonah/omn-15732-migration-domain-fix
Aug 8, 2026
Merged

jonahgabriel merged 3 commits into
devfrom
jonah/omn-15732-migration-domain-fix

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Aug 8, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

infra dev is merge-deadlocked as of 2026-08-08 13:35Z: node-migration-sync (OMN-13332) forces every open infra dev PR to vendor two upstream omnimarket migrations verbatim —

  • node_canary_score_reducer/0003_capability_scores_tenant_id_to_uuid.sql
  • node_projection_registration/0004_node_service_registry_no_force_rls.sql

— but vendoring them trips Application Database Domain Enforcement (OMN-15361), a required CI Summary STRICT_GATE_JOBS member, in the "Enforce schema qualification in changed SQL (trusted)" step of scripts/ci/check_application_database_sql.py. Not vendoring → red node-migration-sync. Vendoring → red OMN-15361. Both are hard required gates; neither can be waived. The failure reproduces identically on 5 other open PRs (#2681, #2680, #2677, #2676, #2668) plus #2682 (head 1aa87f7e2a, run 31259117513 / job 93107117177), which currently carries the only PR-side reproduction of the OMN-15361 half.

Root cause

Both files are new migrations in existing streams whose physical tables (public.capability_scores, public.node_service_registry) have been unqualified/default-schema since their original CREATE, pending the separate, larger governed OMN-15359 schema cutover. This is exactly the case _LEGACY_DEFAULT_SCHEMA_SQL_EXACT_PATHS in check_application_database_sql.py already exists to cover — it carries 11 precedent entries (OMN-15503, OMN-15655) exempting sibling follow-up migrations in the same physical-table-stays-in-public state, most directly node_canary_score_reducer/0002_capability_scores_tenant_id_and_rls.sql, which 0003 in this PR directly continues (same table, same fail-closed tenant-mapping pattern, next step of the OMN-15356 program).

Fix — STATUS: DRAFT, AC2 finding open, plan below supersedes the original claim

Correction (2026-08-08, post-merge-sweep review): the paragraph originally here claimed both new exemption entries were in-class with their cited precedents and that "the migration content is already correct for its precedented class." That claim is false for the 0003 entry. 0003_capability_scores_tenant_id_to_uuid.sql introduces a new function, house_tenant_map_slug_to_uuid, and the 0003 exemption-list entry lets that new function permanently escape ownership declaration — an OMN-15732 AC2 violation ("no new object may be exempted from ownership declaration via the legacy-default-schema list"). Every precedent in _LEGACY_DEFAULT_SCHEMA_SQL_EXACT_PATHS (OMN-15503, OMN-15655, 0002_capability_scores_tenant_id_and_rls.sql) exempts a pre-existing unqualified object carried forward from before OMN-15361 existed — none mints a new function. 0004's entry remains in-class (pre-existing dynamic-SQL block on an already-exempted table) and is unaffected by this correction.

Real plan (option c — content fix, not registry exemption): topology/application_database.py:113-118 declares four schemas: public, tenant, omninode_internal, platform_catalog. platform_catalog is bootstrapped at docker/migrations/forward/_ledger/bootstrap.sql:32 and omninode_internal at src/omnibase_infra/migration/cutover/sql/bootstrap.sql:6 — both already exist, independent of tenant-bootstrap sequencing and independent of the OMN-15359 schema-cutover decision. Qualifying house_tenant_map_slug_to_uuid into omninode_internal (or platform_catalog) and declaring ownership there satisfies OMN-15361 directly for the new function, with no exemption-list entry and no dependency on OMN-15359. This PR stays draft until that content fix supersedes the 0003 exemption entry.

  • Add both new file paths to the existing exemption set, with inline comments citing the precedent and this ticket. This does not touch validator logic, add a skip token, or weaken what the gate checks for any non-exempt file — but per the correction above, the 0003 entry itself is the wrong mechanism for a new object and is pending replacement by the option-(c) content fix.
  • Also vendors the two migrations for real (scripts/sync-node-migrations.sh, matching PR feat(OMN-15730): CI concurrency auto-cancel — omnibase_infra #2682's approach) and adds their declaration rows to docker/migrations/forward/_ledger/application-migrations.tsv (checksums 8bf81c98../adb21c75.. — byte-identical to the content already vendored on feat(OMN-15730): CI concurrency auto-cancel — omnibase_infra #2682, confirmed via sha256sum), bumping tests/unit/scripts/validation/test_application_migration_manifest.py's declaration-count assertion 94→96 — so this PR clears both sides of the deadlock, not just the gate in the abstract.

Validator proof

Reproduced the exact 5 findings from live CI job 93107117177 by running check_application_database_sql.py against the exact vendored SQL content from PR #2682 head 1aa87f7e2a (git show 1aa87f7e2a:<path>) with the pre-fix script:

application_database_sql_gate=FAIL
docker/migrations/forward/nodes/node_canary_score_reducer/0003_capability_scores_tenant_id_to_uuid.sql: application relation target 'house_tenant_map_slug_to_uuid' must be schema-qualified
docker/migrations/forward/nodes/node_canary_score_reducer/0003_capability_scores_tenant_id_to_uuid.sql: application relation target 'public.capability_scores' is prohibited in public
docker/migrations/forward/nodes/node_canary_score_reducer/0003_capability_scores_tenant_id_to_uuid.sql: application target public.capability_scores requires exactly one ownership declaration
docker/migrations/forward/nodes/node_canary_score_reducer/0003_capability_scores_tenant_id_to_uuid.sql: procedural block contains dynamic SQL whose relation targets cannot be proven statically
docker/migrations/forward/nodes/node_projection_registration/0004_node_service_registry_no_force_rls.sql: procedural block contains dynamic SQL whose relation targets cannot be proven statically

Byte-identical to the live log. Same content with this fix applied, using the exact 3-manifest CI argument set (omnimarket + both omninode_infra manifests):

application_database_sql_gate=PASS

At the real committed HEAD of this branch: same command → application_database_sql_gate=PASS.

Re-vendor cascade

None beyond the normal git-history mechanism. This fix changes no content on the two SQL files and no checksums vs the #2682 precedent, so it carries no separate TSV re-sync obligation for other branches. Once merged to dev, every other deadlocked PR (#2681/#2680/#2677/#2676/#2668) clears the OMN-15361 half of the deadlock as soon as it (a) vendors the same two files — forced by node-migration-sync regardless of this fix — and (b) is based on/rebased onto a dev commit containing this exemption-list change. The exemption keys on a fixed logical file path, not git ancestry or content hash, so no per-branch checksum resync is required.

Local proof

  • uv run ruff format / ruff check --fix — clean
  • uv run mypy scripts/ci/check_application_database_sql.py — clean
  • pre-commit run (full chain, real commit, no --no-verify) — all hooks passed, including ONEX Node Migration Vendor Sync Check (previously red without the vendored files)
  • uv run pytest tests/ci/test_application_database_sql_gate.py tests/ci/test_application_database_sql_authority_isolation.py tests/ci/test_application_database_domain_enforcement_contract.py tests/unit/validation/test_application_database_sql_enforcement_regressions.py tests/unit/scripts/validation/test_application_migration_manifest.py — 131 passed
  • governed pre-push impacted-test selector (OMN-13973) — 3011 passed, 5 skipped, 331.34s

Resolution (2026-08-08 adjudication, supersedes "Fix" section above)

STATUS UPDATE: AC2 finding CLOSED — not via the option-(c) content fix
described above.
That plan proposed qualifying
house_tenant_map_slug_to_uuid into omninode_internal or
platform_catalog on the infra side. Adjudication (full text: omnimarket
PR #2026 comment
#5226942902;
one row in docs/tracking/ROLLING_WORK_LEDGER.md,
[mergesweep-0808-domain-adj]) rejected both of those targets for a
node:% migration stream: _ALLOWED_DOMAINS /
schema_migrations_stream_domain_check restrict node streams to exactly
{tenant, omninode_internal}, and platform_catalog is topology domain
PLATFORM_CATALOG -- inadmissible at either enforcement point regardless
of which schema hosts the function. omninode_internal itself also fails
apply-time presence under the forward-migration stream (created only by
the cutover-only initializer -- confirmed at
src/omnibase_infra/migration/cutover/sql/bootstrap.sql:4-5, no
CREATE SCHEMA omninode_internal anywhere under
docker/migrations/forward/).

The actual fix landed upstream in omnimarket (PR #2026, revised): the
house_tenant_map_slug_to_uuid function had exactly one DDL-time consumer
and no runtime caller, so it was deleted outright and its closed mapping
inlined directly into 0003_capability_scores_tenant_id_to_uuid.sql's own
USING clause (fail-closed via a pre-guard RAISE plus a CASE-without-
ELSE, backed by the column's NOT NULL constraint). There is no
0004 file and no new catalog object of any kind -- so there is nothing
left to schema-qualify on either side.

No structural change to this PR as a result. The exemption entry this
PR already added to _LEGACY_DEFAULT_SCHEMA_SQL_EXACT_PATHS and the TSV
ledger row already name the exact pre-split path,
docker/migrations/forward/nodes/node_canary_score_reducer/0003_capability_scores_tenant_id_to_uuid.sql,
with domain=tenant -- both remain correct and become truthful (not
just permitted) once re-vendored, because the file's only target,
public.capability_scores, is itself topology domain TENANT
(src/omnibase_infra/topology/instances/{local,onex-dev,onex-prod}.yaml).
The comment block explaining the exemption
(scripts/ci/check_application_database_sql.py) was updated to drop the
now-inaccurate "the mapping function it defines" clause -- the file no
longer defines one. The TSV row's content checksum will be refreshed by
node-migration-sync once it re-vendors from the omnimarket branch; no
other file in this PR changes.

Update (2026-08-08, applied): re-vendored 0003_capability_scores_tenant_id_to_uuid.sql locally against omnimarket jonah/omn-15732-0003-optc@09c53bfd via scripts/sync-node-migrations.sh (OMNIMARKET_SRC override), refreshed the TSV row's checksum to d1eeefff11fbec255216154ba26aa6f0f29602a0b6fd8fd89e8de5d8d3282b2c, and added the Omnimarket-Source-Ref field above so node-migration-sync resolves the same unmerged branch instead of defaulting to dev (which does not yet carry the omnimarket-side fix). check_application_database_sql.py re-run against this content with the real ownership manifests: application_database_sql_gate=PASS. No exemption-registry or gate-logic edit.

Ticket: OMN-15732 (root-cause fix lane for the infra-dev deadlock diagnosed 2026-08-08 13:35Z; see docs/tracking/ROLLING_WORK_LEDGER.md [mergesweep-0808-deadlock] entries in omni_home for the full incident trail, including the earlier #2682 declaration-gap fix (OMN-15730) that first exposed this OMN-15361 side).

Omnimarket-Source-Ref: jonah/omn-15732-0003-optc
Evidence-Ticket: OMN-15732
Evidence-Source: OCC#6195

Summary by CodeRabbit

  • Database Updates

    • Converted capability score tenant identifiers to UUIDs for improved consistency and validation.
    • Preserved tenant isolation and access controls during the conversion.
    • Removed forced row-level security from the node service registry while retaining existing policies.
  • Maintenance

    • Updated migration validation and tracking to include the latest database changes.

…on-sync-forced vendor files from OMN-15361 legacy-schema lint

infra dev is merge-deadlocked: node-migration-sync (OMN-13332) forces every
open infra dev PR to vendor two upstream omnimarket migrations verbatim
(node_canary_score_reducer/0003_capability_scores_tenant_id_to_uuid.sql,
node_projection_registration/0004_node_service_registry_no_force_rls.sql),
but vendoring them trips "Application Database Domain Enforcement
(OMN-15361)" -- a required CI Summary STRICT_GATE_JOBS member -- in the
"Enforce schema qualification in changed SQL (trusted)" step of
scripts/ci/check_application_database_sql.py: not vendoring -> red
node-migration-sync; vendoring -> red OMN-15361. Both sides are hard
required gates, neither can be waived, and the failure reproduces
identically on 5 open PRs (2681/2680/2677/2676/2668) plus 2682 (head
1aa87f7, run 31259117513/job 93107117177), which currently carries the
only PR-side reproduction of the OMN-15361 half.

ROOT CAUSE: both files are new migrations in existing streams whose
physical tables (public.capability_scores, public.node_service_registry)
have been unqualified/default-schema since their original CREATE, pending
the separate, larger governed OMN-15359 schema cutover. This is the exact
case scripts/ci/check_application_database_sql.py's
_LEGACY_DEFAULT_SCHEMA_SQL_EXACT_PATHS exists to cover -- it already carries
11 precedent entries (OMN-15503, OMN-15655) exempting sibling follow-up
migrations in the same physical-table-stays-in-public state, most directly
node_canary_score_reducer/0002_capability_scores_tenant_id_and_rls.sql,
which 0003 in this fix directly continues (same table, same fail-closed
tenant-mapping pattern, next step of the OMN-15356 program).

FIX: add both new file paths to the existing exemption set, with inline
comments citing the precedent and this ticket. No SQL content changed at
the omnimarket source and no ownership-manifest edits were needed -- the
migration content is already correct for its precedented class; only the
infra-side CI exemption registry (the validator's own documented mechanism
for exactly this situation) was missing the two new entries. This does not
touch validator logic, add a skip token, or weaken what the gate checks for
any non-exempt file.

Also vendors the two migrations for real (via scripts/sync-node-migrations.sh,
matching infra PR #2682's approach) and adds their declaration rows to
docker/migrations/forward/_ledger/application-migrations.tsv (checksums
8bf81c98../adb21c75.. -- byte-identical to the content already vendored on
#2682, confirmed via sha256sum), bumping
tests/unit/scripts/validation/test_application_migration_manifest.py's
declaration-count assertion 94->96, so this PR itself clears both sides of
the deadlock (node-migration-sync AND OMN-15361) rather than only fixing
the gate in the abstract.

VALIDATOR PROOF (not just theoretical): reproduced the exact 5 findings
from live CI job 93107117177 by running check_application_database_sql.py
against the exact vendored SQL content from infra PR #2682 head 1aa87f7
(fetched via `git show 1aa87f7:<path>`) with the pre-fix script --
application_database_sql_gate=FAIL, violation lines byte-identical to the
live log. Same run with this fix applied -- application_database_sql_gate=PASS,
using the exact 3-manifest CI argument set (omnimarket +
both omninode_infra manifests).

RE-VENDOR CASCADE: none beyond the normal git-history mechanism. This fix
changes no *content* on the two SQL files and no checksums vs the #2682
precedent, so it carries no separate TSV re-sync obligation for other
branches. Once merged to infra dev, every other deadlocked PR
(2681/2680/2677/2676/2668) clears the OMN-15361 half of the deadlock as
soon as it (a) vendors the same two files -- forced by node-migration-sync
regardless of this fix -- and (b) is based on/rebased onto a dev commit
containing this exemption-list change; the exemption keys on a fixed
logical file path, not git ancestry or content hash, so no per-branch
checksum resync is required.

Local proof: uv run ruff format/check --fix (clean), uv run mypy (clean),
pre-commit run --files on all changed files (full chain, no --no-verify),
uv run pytest tests/ci/test_application_database_sql_gate.py
tests/ci/test_application_database_sql_authority_isolation.py
tests/ci/test_application_database_domain_enforcement_contract.py
tests/unit/validation/test_application_database_sql_enforcement_regressions.py
tests/unit/scripts/validation/test_application_migration_manifest.py --
all passed.

Ticket: OMN-15732
@onexbot-occ-writer

Copy link
Copy Markdown
Contributor

OCC autobind did not mint a companion for this PR: no changed-file candidate could be proven RED against the merge base, and emitting a PR-existence probe instead would be non-falsifiable evidence (OMN-15247). Hand-authored evidence is required.

@coderabbitai

coderabbitai Bot commented Aug 8, 2026 •

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 52 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b700a343-04db-40e4-b89e-7a04acb11439

📥 Commits

Reviewing files that changed from the base of the PR and between 9b83171 and e2c379c.

⛔ Files ignored due to path filters (1)
  • docker/migrations/forward/_ledger/application-migrations.tsv is excluded by !**/*.tsv
📒 Files selected for processing (2)
  • docker/migrations/forward/nodes/node_canary_score_reducer/0003_capability_scores_tenant_id_to_uuid.sql
  • scripts/ci/check_application_database_sql.py
📝 Walkthrough

Walkthrough

The pull request adds idempotent migrations for capability-score tenant UUID conversion and node service registry RLS behavior. It also updates SQL validation exemptions and the migration manifest test.

Changes

Database migration changes

Layer / File(s) Summary
Capability-score tenant UUID conversion
docker/migrations/forward/nodes/node_canary_score_reducer/0003_capability_scores_tenant_id_to_uuid.sql
Maps omninode to its canonical UUID, rejects unmapped values, converts tenant_id from TEXT to UUID, restores the UUID default, recreates tenant isolation, and grants SELECT to app_dashboard.
Node registry RLS adjustment
docker/migrations/forward/nodes/node_projection_registration/0004_node_service_registry_no_force_rls.sql
Removes FORCE ROW LEVEL SECURITY when node_service_registry exists and leaves RLS enabled.
Migration validation registration
scripts/ci/check_application_database_sql.py, tests/unit/scripts/validation/test_application_migration_manifest.py
Adds exemptions for the migrations and updates the expected declaration count from 94 to 96.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the merge deadlock fix and the exemption of vendor files from the legacy-schema lint.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch jonah/omn-15732-migration-domain-fix

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

✅ Hostile Reviewer — PASSED

Blocking findings (critical): 0
Total findings: 0
Models succeeded: qwen3-review,qwen3-review-b


Gate semantics (pilot phase)

Verdict Meaning Blocks merge?
passed No critical findings No
blocked CRITICAL findings found Yes
degraded All models unavailable (infra) No (pilot)

Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8468/OMN-8524)

jonahgabriel pushed a commit to OmniNode-ai/onex_change_control that referenced this pull request Aug 8, 2026
…e_infra#2683 (#6195)

* evidence: OCC companion pass 1 for OmniNode-ai/omnibase_infra#2683

* evidence: OCC companion self-bind for #6195

---------

Co-authored-by: node-occ-companion-effect <occ-companion-effect@omninode.ai>
@jonahgabriel
jonahgabriel marked this pull request as draft August 8, 2026 14:53
@jonahgabriel

Copy link
Copy Markdown
Collaborator Author

Held as DRAFT — AC2 violation found (OMN-15732).

The 0003 entry added to _LEGACY_DEFAULT_SCHEMA_SQL_EXACT_PATHS is not in the same class as its cited precedents. 0003_capability_scores_tenant_id_to_uuid.sql introduces a new function, house_tenant_map_slug_to_uuid, which the exemption list suppresses ownership-declaration checking for. Every precedent in the list (OMN-15503, OMN-15655, 0002_capability_scores_tenant_id_and_rls.sql) exempts existing unqualified objects carried forward from before OMN-15361 existed — none of them mint a brand-new function and route it around ownership declaration. This 0003 entry lets a new function permanently escape ownership declaration, which is exactly the OMN-15732 AC2 condition ("no new object may be exempted from ownership declaration via the legacy-default-schema list").

The PR body's claim — "the migration content is already correct for its precedented class; only the infra-side CI exemption registry ... was missing the two new entries" — is false for the 0003 half. It is not the same class: 0002 continues an existing unqualified table; 0003 is a new function with no qualification at all, exempted instead of declared. (0004's entry is in-class — a pre-existing dynamic-SQL block on an already-exempted table — and stays as-is.)

Content fix (option c), not exemption: topology/application_database.py:113-118 declares four schemas: public, tenant, omninode_internal, platform_catalog. platform_catalog is bootstrapped at docker/migrations/forward/_ledger/bootstrap.sql:32, and omninode_internal at src/omnibase_infra/migration/cutover/sql/bootstrap.sql:6 — both exist today, independent of the tenant-bootstrap sequencing and independent of the OMN-15359 schema-cutover decision. Qualifying house_tenant_map_slug_to_uuid into omninode_internal (or platform_catalog) and declaring ownership there satisfies OMN-15361 directly, with no new exemption-list entry and no dependency on OMN-15359 landing first.

This PR stays draft until the 0003 fix is content (schema-qualify + declare ownership), not registry (exempt), and the PR body is corrected to reflect that.

Checker source cited: omnibase_infra/scripts/ci/check_application_database_sql.py (exemption list) and src/omnibase_infra/validation/application_database_domain_enforcement.py (ownership-declaration enforcement).

…e mapping

omnimarket's OMN-15732 0003/0004 split was adjudicated and reverted
(no gate widened): platform_catalog is inadmissible for a node:%
migration stream, so the mapping function is inlined back into the
single 0003_capability_scores_tenant_id_to_uuid.sql file instead of
living as a standalone catalog object. No structural change here --
the existing _LEGACY_DEFAULT_SCHEMA_SQL_EXACT_PATHS entry and TSV
ledger row already name that exact pre-split path and stay a
checksum-only delta once node-migration-sync re-vendors. This commit
only corrects the comment's stale "mapping function it defines" clause.
…pping

Re-vendors docker/migrations/forward/nodes/node_canary_score_reducer/
0003_capability_scores_tenant_id_to_uuid.sql from omnimarket
jonah/omn-15732-0003-optc@09c53bfd (PR #2026, adjudicated 2026-08-08):
the standalone house_tenant_map_slug_to_uuid function is deleted and its
mapping inlined into this file's own ALTER COLUMN ... USING clause. The
previously-vendored copy here still carried the pre-adjudication
function-based content; scripts/sync-node-migrations.sh (OMNIMARKET_SRC
override) brings it back in sync.

Refreshes the TSV declaration row's checksum to match. No exemption-
registry, gate-logic, or domain/CHECK-constraint edit -- the existing
0003 exemption entry and domain=tenant declaration were already correct
for the single-file shape per omnibase_infra#2683's prior commit.

PR body separately gains an Omnimarket-Source-Ref field pinning
node-migration-sync to the same unmerged omnimarket branch, since dev
does not yet carry the omnimarket-side fix.

OMN-15732.
@jonahgabriel
jonahgabriel marked this pull request as ready for review August 8, 2026 17:42
@jonahgabriel
jonahgabriel merged commit 211e81e into dev Aug 8, 2026
159 of 162 checks passed
@jonahgabriel
jonahgabriel deleted the jonah/omn-15732-migration-domain-fix branch August 8, 2026 17:55
jonahgabriel added a commit that referenced this pull request Aug 8, 2026
…arder transform seam

Gateway lift Phase 0, item G0. HandlerForwardOutbound/HandlerConsumeInbound
are declared in node_bus_forwarder_effect/contract.yaml but had zero
production call sites -- ServiceGatewayForwarder re-derived the same
tenant-prefix transform inline via prefix_topic() at three call sites
(:78/:171/:312 pre-fix). This collapses that into one implementation:
_prepare_outbound/_prepare_inbound now build a ModelGatewayEnvelope and
delegate the actual prefix/strip transform + payload stamp to the
contract-declared handlers, while the service keeps ownership of the
untrusted-input trust-boundary checks (topic declared, envelope tags match
the config-bound tenant identity) that were never part of the handlers'
own contract.

All prefix_topic() call sites in service_gateway_forwarder.py are removed
(handlers own that transform now). strip_topic_prefix() stays in the
service as the bootstrap step needed to derive canonical_topic before
constructing the handler's required ModelGatewayEnvelope input -- this is
not a duplicate implementation, it feeds the same handler-owned validation.

Adds a seam test (test_gateway_forwarder_service_handler_seam.py) that
spies (wraps=, not mock-only) on the service's own handler instances to
prove the real service path invokes them, not just handler-isolated unit
tests.

MIGRATION-SYNC TRAP disclosure: omnibase_infra#2683 (OMN-15732
split-migration rework) has not merged to dev at time of this PR. The
repo-wide node-migration-sync pre-commit hook requires vendoring
node_canary_score_reducer/0003 and node_projection_registration/0004 from
omnimarket -- vendored here via scripts/sync-node-migrations.sh, with the
matching application-migrations.tsv declaration rows and the manifest
test's pinned count (94->96) taken verbatim from #2683's diff. These files
duplicate #2683's in-flight resolution; this branch should be rebased
onto dev after #2683 lands rather than carrying the duplicate. No exemption
or validator logic (_LEGACY_DEFAULT_SCHEMA_SQL_EXACT_PATHS or otherwise)
was touched.
jonahgabriel added a commit that referenced this pull request Aug 8, 2026
…ateway forwarder

The gateway forwarder's delivery-loop supervision previously raced
delivery.wait() against shutdown_event.wait() with no retry: any
cloud-leg delivery failure (broker drop, transient network fault)
propagated straight out of run_gateway_forwarder and terminated the
process (docs/design/2026-08-08-gateway-node-architecture-lift.md
axis #10, live restart-count evidence in that doc's 1.2).

- runtime/gateway_forwarder.py: new _supervise_gateway_delivery loop
  wraps the existing delivery.wait()/shutdown race. A delivery
  failure now retries with bounded exponential backoff + jitter
  instead of exiting. Reconnect state only resets after the delivery
  loop survives a full heartbeat_interval_seconds "recovery confirm"
  window without failing again (a bare delivery.start() succeeding
  only proves tasks were scheduled, not that the cloud leg is back).
  Process still exits on shutdown or on an unrecoverable
  (non-connectivity) failure -- config/heartbeat-task errors.
- Once a failure window crosses the contract-declared
  degraded_after_seconds, one DEGRADED status event publishes via
  the new ServiceGatewayForwarder.publish_status(), reusing the
  existing gateway-heartbeat.v1 event shape/topic
  (ModelGatewayHeartbeat gains status="degraded" +
  consecutive_failures + detail). Published on the LOCAL bus (not
  cloud) so DEGRADED stays observable while the cloud leg that
  caused it is itself unreachable.
- contract.yaml: liveness block gains
  reconnect_backoff_initial_seconds / _max_seconds / _jitter_seconds
  and degraded_after_seconds (contract_version/node_version patch
  bump 0.1.0 -> 0.1.1); ModelGatewayForwarderConfig gains the
  matching typed fields + a max>=initial validator.
- Data-plane loop (NodeGatewayDelivery, KafkaTransport) is untouched;
  supervision wraps it per scope.

Also vendors 2 omnimarket node migrations
(node_canary_score_reducer/0003, node_projection_registration/0004)
via scripts/sync-node-migrations.sh -- required by the repo-wide
node-migration-sync pre-commit/CI hook on any omnibase_infra branch.
infra#2683 (OMN-15732, split-migration rework) was NOT merged at
push time; these 2 files duplicate that in-flight resolution and
this PR must be rebased onto dev (picking up #2683's shape) once it
lands. No _LEGACY_DEFAULT_SCHEMA_SQL_EXACT_PATHS or exemption/
validator logic touched.

Ticket: OMN-15742 (G2, gateway-lift Phase 0)
jonahgabriel added a commit that referenced this pull request Aug 8, 2026
…warder (G1)

Replace the docker-compose gateway healthcheck test -f
/tmp/gateway-forwarder-ready sentinel-file check with a real produce+readback
probe (onex-gateway-canary-probe) that dials the local leg and cloud leg
independently, using the exact KafkaTransport/credentials real gateway
traffic uses, and exits non-zero when either legs path is dead even though
the forwarder process is still up -- the exact failure mode of the
2026-08-04 outage (ready-file present, forwarding 0% for 4 days, FailingStreak
57 with the old check still calling it healthy).

- New contract-declared gateway_forwarder.canary block (topic, cadence,
  produce/readback deadlines) is the sole authority for the canary topic and
  cadence, materialized into the runtime config the same way mirror_topics
  already is (canary_topic_set selector, no inline redeclaration).
- New ModelGatewayCanaryConfig; ModelGatewayForwarderConfig.canary is a
  required field, validated dedicated (must not collide with mirror_topics).
- New onex-gateway-canary-probe CLI (runtime/gateway_canary_probe.py):
  concurrent local+cloud leg checks, each producing a unique correlation-id
  record to the dedicated canary topic and confirming readback within the
  contract deadline; per-leg PASS/FAIL lines land in
  docker inspect ...State.Health.Log, distinguishing local healthy, cloud
  dead from both healthy per the tickets acceptance criterion.
- Cadence-bound via a --state-file cache so the healthcheck (15s interval)
  does not spam either broker; a real round trip only runs once per
  contract-declared canary.cadence_seconds (default 30s).
- docker-compose.gateway.yml healthcheck now runs the probe; timeout raised
  to 25s to cover a real (non-cached) check.

Built standalone against omnibase_infra dev, NOT on top of G0/OMN-15740
(PR #2689 is still open/unmerged at time of this build) -- disclosed per
operator instruction; no functional dependency on G0s transform-seam wiring,
since this ticket only touches the healthcheck/probe surface.

Branched from dev after PR #2683 (split-migration rework) merged, so the
node-migration-sync hook is satisfied by devs already-split migration
shape. That hook still flagged one unrelated live drift between omnimarket
canonical clone and infras vendored node_canary_score_reducer/0003 file
(the canonical clone was observed live-flapping between two content states
during this session -- another concurrent lane editing it); vendored via
scripts/sync-node-migrations.sh + matching _ledger/application-migrations.tsv
checksum, no hand edits, out of this tickets functional scope, disclosed.

Also regenerated docker/runners/runner-image.lock.json via
scripts/ci/runner_image_identity.py --mode generate: pyproject.toml gained
the onex-gateway-canary-probe script entry, which changes the
dependency-manifest binding input to the CI runner image identity.

Full local gates: ruff format/check clean, mypy --strict clean (2750 source
files), full local unit suite 24616 passed / 4 failed (all 4 confirmed
pre-existing xdist cross-worker env-var-leakage flakes unrelated to any file
this PR touches -- reproduced green serially with -p no:xdist).

Ticket: OMN-15741 (G1 of the gateway node-architecture-lift Phase 0,
docs/design/2026-08-08-gateway-node-architecture-lift.md commit 9422d30b1).
jonahgabriel added a commit that referenced this pull request Aug 8, 2026
…lock test fixtures

Two real pre-push failures caught locally (root-caused, not worked
around):

1. Rebasing this branch onto #2690 (G1) produced an add/add conflict
   on the vendored node_canary_score_reducer/0003 migration; resolving
   it left two stale trailing duplicate lines in
   application-migrations.tsv from this branch's own pre-#2683
   vendoring commit (0003 with a stale checksum, 0004 with a
   redundant-but-matching one). validate_application_migration_manifest.py
   fails closed on the checksum conflict. Removed both trailing dupes;
   the TSV is now byte-identical to dev's (post-#2683) manifest.

2. Three test_gateway_forwarder_runtime.py cases
   (test_runtime_config_rejects_inline_cloud_broker_literal,
   test_runtime_config_fails_closed_on_missing_broker_ref_map,
   test_runtime_config_rejects_inline_canary_literals) built minimal
   contract/config fixtures that predate G1's canary block landing on
   this branch. _materialize_contract_canary_config now runs before
   _materialize_cloud_broker_ref in load_gateway_forwarder_runtime_config,
   so the first two raised on the canary check before ever reaching the
   cloud-broker-ref behavior under test; the third didn't pass the now-
   required broker_ref_map_path kwarg. Updated all three fixtures to
   declare canary_topic_set/canary consistently with the other tests in
   this file and pass broker_ref_map_path.

Ticket: OMN-15743
jonahgabriel added a commit that referenced this pull request Aug 8, 2026
…warder (G1) (#2690)

Replace the docker-compose gateway healthcheck test -f
/tmp/gateway-forwarder-ready sentinel-file check with a real produce+readback
probe (onex-gateway-canary-probe) that dials the local leg and cloud leg
independently, using the exact KafkaTransport/credentials real gateway
traffic uses, and exits non-zero when either legs path is dead even though
the forwarder process is still up -- the exact failure mode of the
2026-08-04 outage (ready-file present, forwarding 0% for 4 days, FailingStreak
57 with the old check still calling it healthy).

- New contract-declared gateway_forwarder.canary block (topic, cadence,
  produce/readback deadlines) is the sole authority for the canary topic and
  cadence, materialized into the runtime config the same way mirror_topics
  already is (canary_topic_set selector, no inline redeclaration).
- New ModelGatewayCanaryConfig; ModelGatewayForwarderConfig.canary is a
  required field, validated dedicated (must not collide with mirror_topics).
- New onex-gateway-canary-probe CLI (runtime/gateway_canary_probe.py):
  concurrent local+cloud leg checks, each producing a unique correlation-id
  record to the dedicated canary topic and confirming readback within the
  contract deadline; per-leg PASS/FAIL lines land in
  docker inspect ...State.Health.Log, distinguishing local healthy, cloud
  dead from both healthy per the tickets acceptance criterion.
- Cadence-bound via a --state-file cache so the healthcheck (15s interval)
  does not spam either broker; a real round trip only runs once per
  contract-declared canary.cadence_seconds (default 30s).
- docker-compose.gateway.yml healthcheck now runs the probe; timeout raised
  to 25s to cover a real (non-cached) check.

Built standalone against omnibase_infra dev, NOT on top of G0/OMN-15740
(PR #2689 is still open/unmerged at time of this build) -- disclosed per
operator instruction; no functional dependency on G0s transform-seam wiring,
since this ticket only touches the healthcheck/probe surface.

Branched from dev after PR #2683 (split-migration rework) merged, so the
node-migration-sync hook is satisfied by devs already-split migration
shape. That hook still flagged one unrelated live drift between omnimarket
canonical clone and infras vendored node_canary_score_reducer/0003 file
(the canonical clone was observed live-flapping between two content states
during this session -- another concurrent lane editing it); vendored via
scripts/sync-node-migrations.sh + matching _ledger/application-migrations.tsv
checksum, no hand edits, out of this tickets functional scope, disclosed.

Also regenerated docker/runners/runner-image.lock.json via
scripts/ci/runner_image_identity.py --mode generate: pyproject.toml gained
the onex-gateway-canary-probe script entry, which changes the
dependency-manifest binding input to the CI runner image identity.

Full local gates: ruff format/check clean, mypy --strict clean (2750 source
files), full local unit suite 24616 passed / 4 failed (all 4 confirmed
pre-existing xdist cross-worker env-var-leakage flakes unrelated to any file
this PR touches -- reproduced green serially with -p no:xdist).

Ticket: OMN-15741 (G1 of the gateway node-architecture-lift Phase 0,
docs/design/2026-08-08-gateway-node-architecture-lift.md commit 9422d30b1).
jonahgabriel added a commit that referenced this pull request Aug 9, 2026
…arder transform seam (#2689)

Gateway lift Phase 0, item G0. HandlerForwardOutbound/HandlerConsumeInbound
are declared in node_bus_forwarder_effect/contract.yaml but had zero
production call sites -- ServiceGatewayForwarder re-derived the same
tenant-prefix transform inline via prefix_topic() at three call sites
(:78/:171/:312 pre-fix). This collapses that into one implementation:
_prepare_outbound/_prepare_inbound now build a ModelGatewayEnvelope and
delegate the actual prefix/strip transform + payload stamp to the
contract-declared handlers, while the service keeps ownership of the
untrusted-input trust-boundary checks (topic declared, envelope tags match
the config-bound tenant identity) that were never part of the handlers'
own contract.

All prefix_topic() call sites in service_gateway_forwarder.py are removed
(handlers own that transform now). strip_topic_prefix() stays in the
service as the bootstrap step needed to derive canonical_topic before
constructing the handler's required ModelGatewayEnvelope input -- this is
not a duplicate implementation, it feeds the same handler-owned validation.

Adds a seam test (test_gateway_forwarder_service_handler_seam.py) that
spies (wraps=, not mock-only) on the service's own handler instances to
prove the real service path invokes them, not just handler-isolated unit
tests.

MIGRATION-SYNC TRAP disclosure: omnibase_infra#2683 (OMN-15732
split-migration rework) has not merged to dev at time of this PR. The
repo-wide node-migration-sync pre-commit hook requires vendoring
node_canary_score_reducer/0003 and node_projection_registration/0004 from
omnimarket -- vendored here via scripts/sync-node-migrations.sh, with the
matching application-migrations.tsv declaration rows and the manifest
test's pinned count (94->96) taken verbatim from #2683's diff. These files
duplicate #2683's in-flight resolution; this branch should be rebased
onto dev after #2683 lands rather than carrying the duplicate. No exemption
or validator logic (_LEGACY_DEFAULT_SCHEMA_SQL_EXACT_PATHS or otherwise)
was touched.
jonahgabriel added a commit that referenced this pull request Aug 9, 2026
…ateway forwarder

The gateway forwarder's delivery-loop supervision previously raced
delivery.wait() against shutdown_event.wait() with no retry: any
cloud-leg delivery failure (broker drop, transient network fault)
propagated straight out of run_gateway_forwarder and terminated the
process (docs/design/2026-08-08-gateway-node-architecture-lift.md
axis #10, live restart-count evidence in that doc's 1.2).

- runtime/gateway_forwarder.py: new _supervise_gateway_delivery loop
  wraps the existing delivery.wait()/shutdown race. A delivery
  failure now retries with bounded exponential backoff + jitter
  instead of exiting. Reconnect state only resets after the delivery
  loop survives a full heartbeat_interval_seconds "recovery confirm"
  window without failing again (a bare delivery.start() succeeding
  only proves tasks were scheduled, not that the cloud leg is back).
  Process still exits on shutdown or on an unrecoverable
  (non-connectivity) failure -- config/heartbeat-task errors.
- Once a failure window crosses the contract-declared
  degraded_after_seconds, one DEGRADED status event publishes via
  the new ServiceGatewayForwarder.publish_status(), reusing the
  existing gateway-heartbeat.v1 event shape/topic
  (ModelGatewayHeartbeat gains status="degraded" +
  consecutive_failures + detail). Published on the LOCAL bus (not
  cloud) so DEGRADED stays observable while the cloud leg that
  caused it is itself unreachable.
- contract.yaml: liveness block gains
  reconnect_backoff_initial_seconds / _max_seconds / _jitter_seconds
  and degraded_after_seconds (contract_version/node_version patch
  bump 0.1.0 -> 0.1.1); ModelGatewayForwarderConfig gains the
  matching typed fields + a max>=initial validator.
- Data-plane loop (NodeGatewayDelivery, KafkaTransport) is untouched;
  supervision wraps it per scope.

Also vendors 2 omnimarket node migrations
(node_canary_score_reducer/0003, node_projection_registration/0004)
via scripts/sync-node-migrations.sh -- required by the repo-wide
node-migration-sync pre-commit/CI hook on any omnibase_infra branch.
infra#2683 (OMN-15732, split-migration rework) was NOT merged at
push time; these 2 files duplicate that in-flight resolution and
this PR must be rebased onto dev (picking up #2683's shape) once it
lands. No _LEGACY_DEFAULT_SCHEMA_SQL_EXACT_PATHS or exemption/
validator logic touched.

Ticket: OMN-15742 (G2, gateway-lift Phase 0)
jonahgabriel added a commit that referenced this pull request Aug 9, 2026
…ateway forwarder (#2688)

* fix(OMN-15742): bounded reconnect backoff + DEGRADED bus status for gateway forwarder

The gateway forwarder's delivery-loop supervision previously raced
delivery.wait() against shutdown_event.wait() with no retry: any
cloud-leg delivery failure (broker drop, transient network fault)
propagated straight out of run_gateway_forwarder and terminated the
process (docs/design/2026-08-08-gateway-node-architecture-lift.md
axis #10, live restart-count evidence in that doc's 1.2).

- runtime/gateway_forwarder.py: new _supervise_gateway_delivery loop
  wraps the existing delivery.wait()/shutdown race. A delivery
  failure now retries with bounded exponential backoff + jitter
  instead of exiting. Reconnect state only resets after the delivery
  loop survives a full heartbeat_interval_seconds "recovery confirm"
  window without failing again (a bare delivery.start() succeeding
  only proves tasks were scheduled, not that the cloud leg is back).
  Process still exits on shutdown or on an unrecoverable
  (non-connectivity) failure -- config/heartbeat-task errors.
- Once a failure window crosses the contract-declared
  degraded_after_seconds, one DEGRADED status event publishes via
  the new ServiceGatewayForwarder.publish_status(), reusing the
  existing gateway-heartbeat.v1 event shape/topic
  (ModelGatewayHeartbeat gains status="degraded" +
  consecutive_failures + detail). Published on the LOCAL bus (not
  cloud) so DEGRADED stays observable while the cloud leg that
  caused it is itself unreachable.
- contract.yaml: liveness block gains
  reconnect_backoff_initial_seconds / _max_seconds / _jitter_seconds
  and degraded_after_seconds (contract_version/node_version patch
  bump 0.1.0 -> 0.1.1); ModelGatewayForwarderConfig gains the
  matching typed fields + a max>=initial validator.
- Data-plane loop (NodeGatewayDelivery, KafkaTransport) is untouched;
  supervision wraps it per scope.

Also vendors 2 omnimarket node migrations
(node_canary_score_reducer/0003, node_projection_registration/0004)
via scripts/sync-node-migrations.sh -- required by the repo-wide
node-migration-sync pre-commit/CI hook on any omnibase_infra branch.
infra#2683 (OMN-15732, split-migration rework) was NOT merged at
push time; these 2 files duplicate that in-flight resolution and
this PR must be rebased onto dev (picking up #2683's shape) once it
lands. No _LEGACY_DEFAULT_SCHEMA_SQL_EXACT_PATHS or exemption/
validator logic touched.

Ticket: OMN-15742 (G2, gateway-lift Phase 0)

* fix(OMN-15742): stop outbound consumer loop re-forwarding local-only publishes

Reconciliation finding D1: publish_status (DEGRADED) publishes directly onto
the local bus's canonical outbound topic -- exactly the topic the
forwarder's own outbound consumer (NodeGatewayDelivery polling
local_consumer, the SAME KafkaTransport object local_bus publishes into in
runtime/gateway_forwarder.py:run_gateway_forwarder) is subscribed to. The
untagged envelope does not match _forward_outbound_message's loopback skip
(only "cloud-to-local"), so it falls through to _prepare_outbound and leaks
to the cloud leg -- directly contradicting
test_publish_status_degraded_goes_to_local_bus_not_cloud, which only passes
today because it uses a bare _MockGatewayBus with no consumer loop
(feedback_real_dispatch_path_tests gap).

Fix: local-only publishes are now stamped with
gateway_direction="local-mirror"; _forward_outbound_message and
validate_outbound_message skip both "cloud-to-local" (existing) and
"local-mirror" (new) via a shared _LOCAL_ONLY_DIRECTIONS set. The stamping
helper is a module-level function, not a method, to stay under the
ONEX Pattern Validation method-count threshold on ServiceGatewayForwarder.

New test test_real_outbound_consumer_loop_does_not_reforward_degraded_status
wires the REAL NodeGatewayDelivery consumer loop against a fake transport
that actually connects publish (local_bus) to poll (local_consumer) -- the
same object playing both roles, matching the real runtime wiring -- unlike
every existing test in this file, which uses two disconnected fakes and so
cannot observe this class of bug. Verified RED against the parent commit
(f0cdfe4, pre-fix): DEGRADED payload reached the cloud leg. GREEN after
this fix.

The matching heartbeat-local-mirror half of this bug (#2692, OMN-15570)
gets the same fix applied on that branch separately, since it stacks on
this one and its own diff must be independently correct.

Ticket: OMN-15742
Evidence-Ticket: OMN-15742

* fix(OMN-15742): stop test _Source.poll() from starving the asyncio event loop

Root cause of every anomalously long/hanging local pre-push run on this
branch throughout this session (observed: 10h33m on the original Mac before
this session started per the rolling ledger, then repeated 33min-2h+ hangs
on .200 across multiple push attempts -- all traced to the SAME mechanism
via a --timeout=60 diagnostic run that caught the stack mid-hang inside
NodeGatewayDelivery._run_direction's `while True:` loop).

_Source.poll() (a test fake in test_gateway_delivery_service.py) returned
`[]` with zero internal `await` suspension points. A coroutine with no real
suspension point does not yield control back to the asyncio event loop when
awaited -- so when test_real_outbound_consumer_loop_does_not_reforward_degraded_status
(added by the prior commit on this branch) drives a REAL
NodeGatewayDelivery task loop with _Source as the "idle" cloud consumer,
`_run_direction`'s `while True: await source.poll(...)` busy-spins the
event loop forever. `task.cancel()` only takes effect at the next real
suspension point, so the spinning task is uncancellable: `delivery.stop()`
(called in the test's `finally`) awaits `asyncio.gather(*tasks)` on tasks
that can never be scheduled to observe their own cancellation, deadlocking
the entire test process indefinitely -- explaining every "still running,
100% CPU, no progress" observation this session mistook for slowness rather
than a hang.

Fix: _Source.poll() now honors timeout_ms via asyncio.sleep(timeout_ms /
1000) before returning empty, matching how a real transport (and this same
file's _SharedLocalTransport.poll(), which correctly uses
asyncio.wait_for(..., timeout=...)) actually behaves. Every other _Source
usage in this file calls deliver_message() directly and never invokes
poll() at all, so this costs exactly one test ~50ms and changes nothing
else (verified: full file 5/5 passed in 0.20s, vs. hanging past a 60s
per-test timeout pre-fix).

Ticket: OMN-15742

* fix(OMN-15742): wire required canary field into remaining gateway test fixtures

Two gaps surfaced by the fixed full local suite run (previous commit
removed the busy-spin hang that was masking these):

- test_gateway_forwarder_config.py: the two reconnect_backoff tests
  (added by this branch's own reconnect-supervision commit, authored
  before this branch was rebased onto dev's #2690/OMN-15741 G1 commit)
  never got canary=_canary() added, unlike every other constructor call
  in the same file.
- test_gateway_forwarder_service_handler_seam.py: pre-existing dev-tip
  gap (unrelated to this branch's diff) -- same class of fix already
  applied independently on the OMN-15781 branch.

Ticket: OMN-15742
jonahgabriel added a commit that referenced this pull request Aug 9, 2026
…hdog (#2700)

* fix(OMN-15742): bounded reconnect backoff + DEGRADED bus status for gateway forwarder

The gateway forwarder's delivery-loop supervision previously raced
delivery.wait() against shutdown_event.wait() with no retry: any
cloud-leg delivery failure (broker drop, transient network fault)
propagated straight out of run_gateway_forwarder and terminated the
process (docs/design/2026-08-08-gateway-node-architecture-lift.md
axis #10, live restart-count evidence in that doc's 1.2).

- runtime/gateway_forwarder.py: new _supervise_gateway_delivery loop
  wraps the existing delivery.wait()/shutdown race. A delivery
  failure now retries with bounded exponential backoff + jitter
  instead of exiting. Reconnect state only resets after the delivery
  loop survives a full heartbeat_interval_seconds "recovery confirm"
  window without failing again (a bare delivery.start() succeeding
  only proves tasks were scheduled, not that the cloud leg is back).
  Process still exits on shutdown or on an unrecoverable
  (non-connectivity) failure -- config/heartbeat-task errors.
- Once a failure window crosses the contract-declared
  degraded_after_seconds, one DEGRADED status event publishes via
  the new ServiceGatewayForwarder.publish_status(), reusing the
  existing gateway-heartbeat.v1 event shape/topic
  (ModelGatewayHeartbeat gains status="degraded" +
  consecutive_failures + detail). Published on the LOCAL bus (not
  cloud) so DEGRADED stays observable while the cloud leg that
  caused it is itself unreachable.
- contract.yaml: liveness block gains
  reconnect_backoff_initial_seconds / _max_seconds / _jitter_seconds
  and degraded_after_seconds (contract_version/node_version patch
  bump 0.1.0 -> 0.1.1); ModelGatewayForwarderConfig gains the
  matching typed fields + a max>=initial validator.
- Data-plane loop (NodeGatewayDelivery, KafkaTransport) is untouched;
  supervision wraps it per scope.

Also vendors 2 omnimarket node migrations
(node_canary_score_reducer/0003, node_projection_registration/0004)
via scripts/sync-node-migrations.sh -- required by the repo-wide
node-migration-sync pre-commit/CI hook on any omnibase_infra branch.
infra#2683 (OMN-15732, split-migration rework) was NOT merged at
push time; these 2 files duplicate that in-flight resolution and
this PR must be rebased onto dev (picking up #2683's shape) once it
lands. No _LEGACY_DEFAULT_SCHEMA_SQL_EXACT_PATHS or exemption/
validator logic touched.

Ticket: OMN-15742 (G2, gateway-lift Phase 0)

* fix(OMN-15742): stop outbound consumer loop re-forwarding local-only publishes

Reconciliation finding D1: publish_status (DEGRADED) publishes directly onto
the local bus's canonical outbound topic -- exactly the topic the
forwarder's own outbound consumer (NodeGatewayDelivery polling
local_consumer, the SAME KafkaTransport object local_bus publishes into in
runtime/gateway_forwarder.py:run_gateway_forwarder) is subscribed to. The
untagged envelope does not match _forward_outbound_message's loopback skip
(only "cloud-to-local"), so it falls through to _prepare_outbound and leaks
to the cloud leg -- directly contradicting
test_publish_status_degraded_goes_to_local_bus_not_cloud, which only passes
today because it uses a bare _MockGatewayBus with no consumer loop
(feedback_real_dispatch_path_tests gap).

Fix: local-only publishes are now stamped with
gateway_direction="local-mirror"; _forward_outbound_message and
validate_outbound_message skip both "cloud-to-local" (existing) and
"local-mirror" (new) via a shared _LOCAL_ONLY_DIRECTIONS set. The stamping
helper is a module-level function, not a method, to stay under the
ONEX Pattern Validation method-count threshold on ServiceGatewayForwarder.

New test test_real_outbound_consumer_loop_does_not_reforward_degraded_status
wires the REAL NodeGatewayDelivery consumer loop against a fake transport
that actually connects publish (local_bus) to poll (local_consumer) -- the
same object playing both roles, matching the real runtime wiring -- unlike
every existing test in this file, which uses two disconnected fakes and so
cannot observe this class of bug. Verified RED against the parent commit
(f0cdfe4, pre-fix): DEGRADED payload reached the cloud leg. GREEN after
this fix.

The matching heartbeat-local-mirror half of this bug (#2692, OMN-15570)
gets the same fix applied on that branch separately, since it stacks on
this one and its own diff must be independently correct.

Ticket: OMN-15742
Evidence-Ticket: OMN-15742

* fix(OMN-15742): stop test _Source.poll() from starving the asyncio event loop

Root cause of every anomalously long/hanging local pre-push run on this
branch throughout this session (observed: 10h33m on the original Mac before
this session started per the rolling ledger, then repeated 33min-2h+ hangs
on .200 across multiple push attempts -- all traced to the SAME mechanism
via a --timeout=60 diagnostic run that caught the stack mid-hang inside
NodeGatewayDelivery._run_direction's `while True:` loop).

_Source.poll() (a test fake in test_gateway_delivery_service.py) returned
`[]` with zero internal `await` suspension points. A coroutine with no real
suspension point does not yield control back to the asyncio event loop when
awaited -- so when test_real_outbound_consumer_loop_does_not_reforward_degraded_status
(added by the prior commit on this branch) drives a REAL
NodeGatewayDelivery task loop with _Source as the "idle" cloud consumer,
`_run_direction`'s `while True: await source.poll(...)` busy-spins the
event loop forever. `task.cancel()` only takes effect at the next real
suspension point, so the spinning task is uncancellable: `delivery.stop()`
(called in the test's `finally`) awaits `asyncio.gather(*tasks)` on tasks
that can never be scheduled to observe their own cancellation, deadlocking
the entire test process indefinitely -- explaining every "still running,
100% CPU, no progress" observation this session mistook for slowness rather
than a hang.

Fix: _Source.poll() now honors timeout_ms via asyncio.sleep(timeout_ms /
1000) before returning empty, matching how a real transport (and this same
file's _SharedLocalTransport.poll(), which correctly uses
asyncio.wait_for(..., timeout=...)) actually behaves. Every other _Source
usage in this file calls deliver_message() directly and never invokes
poll() at all, so this costs exactly one test ~50ms and changes nothing
else (verified: full file 5/5 passed in 0.20s, vs. hanging past a 60s
per-test timeout pre-fix).

Ticket: OMN-15742

* fix(OMN-15742): wire required canary field into remaining gateway test fixtures

Two gaps surfaced by the fixed full local suite run (previous commit
removed the busy-spin hang that was masking these):

- test_gateway_forwarder_config.py: the two reconnect_backoff tests
  (added by this branch's own reconnect-supervision commit, authored
  before this branch was rebased onto dev's #2690/OMN-15741 G1 commit)
  never got canary=_canary() added, unlike every other constructor call
  in the same file.
- test_gateway_forwarder_service_handler_seam.py: pre-existing dev-tip
  gap (unrelated to this branch's diff) -- same class of fix already
  applied independently on the OMN-15781 branch.

Ticket: OMN-15742

* fix(OMN-15748): gateway poison-pill quarantine + membership-loss watchdog

decode_message() ran before the try/except that nacks, so any undecodable
record on a mirrored source topic crashed the process (live-observed
2026-08-08T16:50Z). Nacking it (the ordinary failure path) would also be
wrong for a permanently malformed record: nack seeks back to the same
offset and re-crashes forever. Quarantine instead: log, best-effort
dead-letter via the DLQ topic derived from the record's own topic, commit
past the offset, keep the loop alive.

Also adds an independent membership-loss watchdog to NodeGatewayDelivery.
The live 2026-08-09T10:03:07Z incident was a clean single LeaveGroup with
zero exception (aiokafka's client-side max_poll_interval_ms idle-eviction,
fired from a separate asyncio task than the poll loop) -- the existing
exception-triggered reconnect supervision in runtime/gateway_forwarder.py
cannot observe a task that is alive, hung, and never exceptions. The
watchdog tracks per-direction last-progress timestamps (updated on poll()
completion and deliver_message completion) plus a best-effort
consumer.assignment() membership probe, and on staleness force-recreates
just the affected direction's transport (a fresh client forces a new group
join, sidestepping the permanently-stuck idle clock) and publishes a
DEGRADED status. Reuses the previously contract-declared but unwired
max_silence_window_seconds config field as the staleness threshold.

TDD: both regression tests are RED against pre-fix source (verified via
git stash of the source-only diff) and GREEN after.

OMN-15748

* fix(OMN-15748): consumer-scoped watchdog recovery, no CancelledError escape

Fixes two verifier-confirmed defects in the membership-loss watchdog
(mergesweep-0809-poisonpath-verify):

- wait()/_run_direction: a watchdog stale_task.cancel() made the
  supervisor's delivery.wait() (a frozen asyncio.gather snapshot) observe
  a cancelled child and propagate CancelledError -- a BaseException that
  escapes gateway_forwarder.py's except Exception and kills the process on
  every recovery. _run_direction now catches CancelledError and returns
  cleanly when its direction is watchdog-recovering; wait() re-reads the
  live task set after every completion instead of gathering a one-time
  snapshot, so a swapped-in replacement task stays supervised.

- Recovery no longer calls close()+start() (which stops BOTH the consumer
  and producer on a KafkaTransport instance shared across directions --
  one direction's consumer is the same object the other direction's
  publish and status/heartbeat emission use as a producer). Adds
  KafkaTransport.restart_consumer(), a consumer-only recreate that never
  touches the producer; _recover_stalled_direction now calls it
  exclusively.

De-vacuums the watchdog regression test (arms delivery.wait() before the
stall, matching production ordering) and adds direct KafkaTransport-level
coverage proving the producer is untouched across a restart_consumer()
call.

* fix(OMN-15748): address CodeRabbit findings on watchdog wait() + log sanitization

Two live (non-outdated) CodeRabbit threads on infra#2700:

- wait()'s asyncio.wait was unbounded, waking only on a tracked task's
  completion. _recover_stalled_direction splices the replacement task into
  self._tasks strictly AFTER the stale task has already finished and this
  loop has already woken/rebuilt `watched` from the stale-only list -- so an
  unbounded wait then blocks on the remaining old tasks with no further
  wakeup until one of THEM completes. A real exception on the recovered
  direction would go unsupervised (only surfacing as an "exception was never
  retrieved" warning at GC). Fixed by bounding the wait with
  _watchdog_tick_seconds so the loop always revisits self._tasks, and by also
  picking up an already-completed-with-exception replacement on that rescan.
  Added test_watchdog_replacement_task_failure_still_reaches_wait, verified
  RED against the unbounded wait() (reverted locally, re-applied after) and
  GREEN after.

- _quarantine_undecodable_message logged the raw exception via exc_info=error,
  bypassing the sanitize_error_message() the DLQ payload path already uses a
  few lines up. A pydantic ValidationError from decode_message() embeds the
  offending record content, which is untrusted tenant bus traffic that can
  carry PII/credential-bearing values. Now logs the sanitized message,
  matching the DLQ path.

* test(OMN-15748): harden test_monitor_logs.py subprocess timeouts against CI fleet contention

The self-hosted CI fleet runs many concurrent pytest-split shards per job;
interpreter spawn/import can occasionally queue behind that CPU contention.
A hardcoded 10s subprocess.run(..., timeout=10) tripped this twice live on
unrelated PRs' Split 12/15 (infra#2697 and this PR, both times as three
subprocess.TimeoutExpired failures at exactly the 10s bound, unrelated to
either PR's own diff -- this file isn't touched by either PR's product
diff). Bumped to a shared _SUBPROCESS_TIMEOUT_SECONDS = 30 constant across
all three call sites: still tight enough to catch a genuine hang, with real
headroom against fleet contention.
jonahgabriel added a commit that referenced this pull request Aug 16, 2026
…lock test fixtures

Two real pre-push failures caught locally (root-caused, not worked
around):

1. Rebasing this branch onto #2690 (G1) produced an add/add conflict
   on the vendored node_canary_score_reducer/0003 migration; resolving
   it left two stale trailing duplicate lines in
   application-migrations.tsv from this branch's own pre-#2683
   vendoring commit (0003 with a stale checksum, 0004 with a
   redundant-but-matching one). validate_application_migration_manifest.py
   fails closed on the checksum conflict. Removed both trailing dupes;
   the TSV is now byte-identical to dev's (post-#2683) manifest.

2. Three test_gateway_forwarder_runtime.py cases
   (test_runtime_config_rejects_inline_cloud_broker_literal,
   test_runtime_config_fails_closed_on_missing_broker_ref_map,
   test_runtime_config_rejects_inline_canary_literals) built minimal
   contract/config fixtures that predate G1's canary block landing on
   this branch. _materialize_contract_canary_config now runs before
   _materialize_cloud_broker_ref in load_gateway_forwarder_runtime_config,
   so the first two raised on the canary check before ever reaching the
   cloud-broker-ref behavior under test; the third didn't pass the now-
   required broker_ref_map_path kwarg. Updated all three fixtures to
   declare canary_topic_set/canary consistently with the other tests in
   this file and pass broker_ref_map_path.

Ticket: OMN-15743
jonahgabriel added a commit that referenced this pull request Aug 16, 2026
…ry (#2687)

* fix(OMN-15743): resolve gateway cloud_broker_ref at the effect boundary

Deletes the hardcoded Docker extra_hosts bastion-IP override and the
duplicated cloud_bus.bootstrap_servers literal in beta-gateway-canary.yaml.
The cloud broker endpoint is now resolved from the node contract's
gateway_forwarder.cloud_leg.cloud_broker_ref at process startup, via a new
_materialize_cloud_broker_ref() boundary function in gateway_forwarder.py
that mirrors the existing _materialize_contract_mirror_topics() pattern:
contract is sole authority, resolved YAML may not redeclare the literal,
and resolution fails closed (ValueError, no fallback) if the ref doesn't
match the contract or the operator-supplied broker-ref map is missing or
has no entry for the ref.

Compose now mounts an operator-supplied GATEWAY_BROKER_REF_MAP_FILE (same
required-env-var-mount pattern already used for the AWS Roles Anywhere
credential files) instead of baking a bastion IP into extra_hosts. The CLI
gained a required --broker-ref-map flag with no default.

Also vendors 2 omnimarket node migrations (node_canary_score_reducer/0003,
node_projection_registration/0004) via scripts/sync-node-migrations.sh and
declares them in application-migrations.tsv -- required because infra#2683
(OMN-15732 split-migration rework) has not yet merged to dev; see PR body
for the disclosure and rebase note.

OMN-15743

* chore(OMN-15743): trigger CI re-run after Evidence-Source edit

* fix(OMN-15743): thread broker-ref-map through the canary probe (seam A)

load_gateway_forwarder_runtime_config now requires keyword-only
broker_ref_map_path (this PR). onex-gateway-canary-probe (OMN-15741,
PR #2690) called it single-arg -- a TypeError on every healthcheck
tick once both land. Pairs the caller update with the signature
change in the same PR so they can never split:

- gateway_canary_probe.py: new required --broker-ref-map CLI arg,
  threaded into the loader call.
- docker-compose.gateway.yml: healthcheck test cmd passes
  --broker-ref-map /run/gateway/broker-ref-map.yaml (the same file
  already bind-mounted for the main forwarder process).

HOLD: do not land until (a) #2690 (G1) has merged and (b) a durable
name-resolution path exists for all 6 broker FQDNs -- Route53 PHZ
(OMN-15739 S6.2) or the ref-map carrying all six -- since this PR
also deletes the extra_hosts pin that live outbound currently
depends on; #2686 is the interim extra_hosts stopgap and is not a
substitute for landing this before that path exists.

Ticket: OMN-15743

* fix(OMN-15743): repair stale migration-manifest duplicates + canary-block test fixtures

Two real pre-push failures caught locally (root-caused, not worked
around):

1. Rebasing this branch onto #2690 (G1) produced an add/add conflict
   on the vendored node_canary_score_reducer/0003 migration; resolving
   it left two stale trailing duplicate lines in
   application-migrations.tsv from this branch's own pre-#2683
   vendoring commit (0003 with a stale checksum, 0004 with a
   redundant-but-matching one). validate_application_migration_manifest.py
   fails closed on the checksum conflict. Removed both trailing dupes;
   the TSV is now byte-identical to dev's (post-#2683) manifest.

2. Three test_gateway_forwarder_runtime.py cases
   (test_runtime_config_rejects_inline_cloud_broker_literal,
   test_runtime_config_fails_closed_on_missing_broker_ref_map,
   test_runtime_config_rejects_inline_canary_literals) built minimal
   contract/config fixtures that predate G1's canary block landing on
   this branch. _materialize_contract_canary_config now runs before
   _materialize_cloud_broker_ref in load_gateway_forwarder_runtime_config,
   so the first two raised on the canary check before ever reaching the
   cloud-broker-ref behavior under test; the third didn't pass the now-
   required broker_ref_map_path kwarg. Updated all three fixtures to
   declare canary_topic_set/canary consistently with the other tests in
   this file and pass broker_ref_map_path.

Ticket: OMN-15743
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant