Repository navigation
fix(OMN-15742): bounded reconnect backoff + DEGRADED bus status for gateway forwarder - #2688
Merged
Merged
Conversation
📝 WalkthroughWalkthroughThe gateway forwarder adds configurable reconnect backoff, degraded heartbeat events, local-only status publication, retry supervision, recovery confirmation, and tests for validation, delivery isolation, status reporting, and task cleanup. ChangesGateway liveness supervision
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant GatewayRuntime
participant DeliverySupervisor
participant ServiceGatewayForwarder
participant LocalBus
GatewayRuntime->>DeliverySupervisor: supervise delivery loop
DeliverySupervisor->>DeliverySupervisor: retry with bounded backoff
DeliverySupervisor->>ServiceGatewayForwarder: publish degraded status
ServiceGatewayForwarder->>LocalBus: publish local-mirror heartbeat
DeliverySupervisor->>DeliverySupervisor: confirm recovery
DeliverySupervisor->>ServiceGatewayForwarder: publish active status
ServiceGatewayForwarder->>LocalBus: publish local-mirror heartbeat
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
Contributor
❓ Hostile Reviewer — UNKNOWNBlocking findings (critical): 0 Gate semantics (pilot phase)
Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8468/OMN-8524) |
jonahgabriel
added a commit
to OmniNode-ai/onex_change_control
that referenced
this pull request
Aug 8, 2026
#6206) * evidence: OCC companion pass 1 for OmniNode-ai/omnibase_infra#2688 * evidence: OCC companion self-bind for #6206 * fix(OMN-15742): replace non-falsifiable deploy-assessment probe with live gh-api content check The autobind-generated dod-deploy-assessment check used 'gh pr diff | grep -c' which greps the PR diff text itself -- not falsifiable against the deployed/pushed system state, and rejected by the OMN-14443 deploy-gate falsifiability ratchet. Replaced with 'gh api .../contents/<path>?ref=<head_sha> | grep -q <symbol>', which fetches the real pushed file content from GitHub and asserts the new _supervise_gateway_delivery reconnect-supervision symbol is present -- exit status genuinely depends on the state of the pushed commit. * fix(OMN-15742): regenerate dod-deploy-assessment receipt for the falsifiable gh-api probe Tool-computed contract_sha256/contract_entry_sha256 (compute_contract_sha256 / compute_contract_entry_sha256 from omnibase_core.validation.validator_receipt_gate against the amended contracts/OMN-15742.yaml), not hand-authored. check_value, commit_sha, and outputs reflect the real gh-api probe I ran and verified locally (exit_code 0, 2 live matches for _supervise_gateway_delivery). * style(OMN-15742): yamlfmt normalization of the amended contract * style(OMN-15742): yamlfmt normalization of the amended receipt * fix(OMN-15742): correct contract_sha256 to match the yamlfmt-normalized contract file (whole-file hash is byte-sensitive; per-entry hash is canonical-parse and unchanged) * fix(OMN-15742): restore --- document-start marker (yamlfmt lint requirement) * fix(OMN-15742): restore --- document-start marker (yamlfmt lint requirement) * fix(OMN-15742): recompute contract_sha256 against the final yamlfmt-normalized contract * fix(OMN-15742): remove yamlfmt sentinel corruption (OMN-15479) -- rewrite summary as a single-line quoted scalar, not a folded > block * fix(OMN-15742): recompute contract_sha256 against sentinel-free contract --------- Co-authored-by: node-occ-companion-effect <occ-companion-effect@omninode.ai> Co-authored-by: Jonah Gray <jonah@omninode.ai>
jonahgabriel
force-pushed
the
jonah/omn-15742-gateway-reconnect-supervision
branch
from
August 8, 2026 22:24
09b17db to
f0cdfe4
Compare
This was referenced Aug 9, 2026
Merged
…ateway forwarder The gateway forwarder's delivery-loop supervision previously raced delivery.wait() against shutdown_event.wait() with no retry: any cloud-leg delivery failure (broker drop, transient network fault) propagated straight out of run_gateway_forwarder and terminated the process (docs/design/2026-08-08-gateway-node-architecture-lift.md axis #10, live restart-count evidence in that doc's 1.2). - runtime/gateway_forwarder.py: new _supervise_gateway_delivery loop wraps the existing delivery.wait()/shutdown race. A delivery failure now retries with bounded exponential backoff + jitter instead of exiting. Reconnect state only resets after the delivery loop survives a full heartbeat_interval_seconds "recovery confirm" window without failing again (a bare delivery.start() succeeding only proves tasks were scheduled, not that the cloud leg is back). Process still exits on shutdown or on an unrecoverable (non-connectivity) failure -- config/heartbeat-task errors. - Once a failure window crosses the contract-declared degraded_after_seconds, one DEGRADED status event publishes via the new ServiceGatewayForwarder.publish_status(), reusing the existing gateway-heartbeat.v1 event shape/topic (ModelGatewayHeartbeat gains status="degraded" + consecutive_failures + detail). Published on the LOCAL bus (not cloud) so DEGRADED stays observable while the cloud leg that caused it is itself unreachable. - contract.yaml: liveness block gains reconnect_backoff_initial_seconds / _max_seconds / _jitter_seconds and degraded_after_seconds (contract_version/node_version patch bump 0.1.0 -> 0.1.1); ModelGatewayForwarderConfig gains the matching typed fields + a max>=initial validator. - Data-plane loop (NodeGatewayDelivery, KafkaTransport) is untouched; supervision wraps it per scope. Also vendors 2 omnimarket node migrations (node_canary_score_reducer/0003, node_projection_registration/0004) via scripts/sync-node-migrations.sh -- required by the repo-wide node-migration-sync pre-commit/CI hook on any omnibase_infra branch. infra#2683 (OMN-15732, split-migration rework) was NOT merged at push time; these 2 files duplicate that in-flight resolution and this PR must be rebased onto dev (picking up #2683's shape) once it lands. No _LEGACY_DEFAULT_SCHEMA_SQL_EXACT_PATHS or exemption/ validator logic touched. Ticket: OMN-15742 (G2, gateway-lift Phase 0)
…publishes Reconciliation finding D1: publish_status (DEGRADED) publishes directly onto the local bus's canonical outbound topic -- exactly the topic the forwarder's own outbound consumer (NodeGatewayDelivery polling local_consumer, the SAME KafkaTransport object local_bus publishes into in runtime/gateway_forwarder.py:run_gateway_forwarder) is subscribed to. The untagged envelope does not match _forward_outbound_message's loopback skip (only "cloud-to-local"), so it falls through to _prepare_outbound and leaks to the cloud leg -- directly contradicting test_publish_status_degraded_goes_to_local_bus_not_cloud, which only passes today because it uses a bare _MockGatewayBus with no consumer loop (feedback_real_dispatch_path_tests gap). Fix: local-only publishes are now stamped with gateway_direction="local-mirror"; _forward_outbound_message and validate_outbound_message skip both "cloud-to-local" (existing) and "local-mirror" (new) via a shared _LOCAL_ONLY_DIRECTIONS set. The stamping helper is a module-level function, not a method, to stay under the ONEX Pattern Validation method-count threshold on ServiceGatewayForwarder. New test test_real_outbound_consumer_loop_does_not_reforward_degraded_status wires the REAL NodeGatewayDelivery consumer loop against a fake transport that actually connects publish (local_bus) to poll (local_consumer) -- the same object playing both roles, matching the real runtime wiring -- unlike every existing test in this file, which uses two disconnected fakes and so cannot observe this class of bug. Verified RED against the parent commit (f0cdfe4, pre-fix): DEGRADED payload reached the cloud leg. GREEN after this fix. The matching heartbeat-local-mirror half of this bug (#2692, OMN-15570) gets the same fix applied on that branch separately, since it stacks on this one and its own diff must be independently correct. Ticket: OMN-15742 Evidence-Ticket: OMN-15742
…ent loop Root cause of every anomalously long/hanging local pre-push run on this branch throughout this session (observed: 10h33m on the original Mac before this session started per the rolling ledger, then repeated 33min-2h+ hangs on .200 across multiple push attempts -- all traced to the SAME mechanism via a --timeout=60 diagnostic run that caught the stack mid-hang inside NodeGatewayDelivery._run_direction's `while True:` loop). _Source.poll() (a test fake in test_gateway_delivery_service.py) returned `[]` with zero internal `await` suspension points. A coroutine with no real suspension point does not yield control back to the asyncio event loop when awaited -- so when test_real_outbound_consumer_loop_does_not_reforward_degraded_status (added by the prior commit on this branch) drives a REAL NodeGatewayDelivery task loop with _Source as the "idle" cloud consumer, `_run_direction`'s `while True: await source.poll(...)` busy-spins the event loop forever. `task.cancel()` only takes effect at the next real suspension point, so the spinning task is uncancellable: `delivery.stop()` (called in the test's `finally`) awaits `asyncio.gather(*tasks)` on tasks that can never be scheduled to observe their own cancellation, deadlocking the entire test process indefinitely -- explaining every "still running, 100% CPU, no progress" observation this session mistook for slowness rather than a hang. Fix: _Source.poll() now honors timeout_ms via asyncio.sleep(timeout_ms / 1000) before returning empty, matching how a real transport (and this same file's _SharedLocalTransport.poll(), which correctly uses asyncio.wait_for(..., timeout=...)) actually behaves. Every other _Source usage in this file calls deliver_message() directly and never invokes poll() at all, so this costs exactly one test ~50ms and changes nothing else (verified: full file 5/5 passed in 0.20s, vs. hanging past a 60s per-test timeout pre-fix). Ticket: OMN-15742
…t fixtures Two gaps surfaced by the fixed full local suite run (previous commit removed the busy-spin hang that was masking these): - test_gateway_forwarder_config.py: the two reconnect_backoff tests (added by this branch's own reconnect-supervision commit, authored before this branch was rebased onto dev's #2690/OMN-15741 G1 commit) never got canary=_canary() added, unlike every other constructor call in the same file. - test_gateway_forwarder_service_handler_seam.py: pre-existing dev-tip gap (unrelated to this branch's diff) -- same class of fix already applied independently on the OMN-15781 branch. Ticket: OMN-15742
jonahgabriel
force-pushed
the
jonah/omn-15742-gateway-reconnect-supervision
branch
from
August 9, 2026 14:30
f0cdfe4 to
611c8a1
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/omnibase_infra/runtime/gateway_forwarder.py`:
- Around line 342-358: Sanitize the delivery exception detail before both the
warning logger call and the degraded status publication in the gateway delivery
loop. Use the repository’s existing sanitization utility on the exception text,
then reuse the sanitized, stable diagnostic value for the logged error and the
_publish_gateway_status detail instead of raw exc or str(exc).
- Around line 407-412: Bound the best-effort status publication around
forwarder.publish_status so _publish_with_delivery_retry cannot retry
RuntimeHostError indefinitely. Apply a finite deadline or attempt budget to this
call, handle expiry or exhaustion, and allow the reconnect supervisor to
continue its cloud-leg supervision.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: e24d4740-0265-49c6-a546-69c969b180ef
📒 Files selected for processing (10)
src/omnibase_infra/nodes/node_bus_forwarder_effect/contract.yamlsrc/omnibase_infra/nodes/node_bus_forwarder_effect/models/model_gateway_forwarder_config.pysrc/omnibase_infra/nodes/node_bus_forwarder_effect/models/model_gateway_heartbeat.pysrc/omnibase_infra/nodes/node_bus_forwarder_effect/services/service_gateway_forwarder.pysrc/omnibase_infra/runtime/gateway_forwarder.pytests/unit/nodes/node_bus_forwarder_effect/test_gateway_delivery_service.pytests/unit/nodes/node_bus_forwarder_effect/test_gateway_forwarder_config.pytests/unit/nodes/node_bus_forwarder_effect/test_gateway_forwarder_service.pytests/unit/nodes/node_bus_forwarder_effect/test_gateway_forwarder_service_handler_seam.pytests/unit/runtime/test_gateway_forwarder_runtime.py
Comment on lines
+342
to
+358
| logger.warning( | ||
| "Gateway delivery loop failed; reconnect attempt=%d " | ||
| "elapsed_seconds=%.1f error_type=%s error=%s", | ||
| consecutive_failures, | ||
| elapsed_seconds, | ||
| type(exc).__name__, | ||
| exc, | ||
| ) | ||
|
|
||
| degraded_threshold = config.degraded_after_seconds | ||
| if not degraded_emitted and elapsed_seconds >= degraded_threshold: | ||
| await _publish_gateway_status( | ||
| forwarder, | ||
| status="degraded", | ||
| consecutive_failures=consecutive_failures, | ||
| detail=f"{type(exc).__name__}: {exc}", | ||
| ) |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Sanitize the delivery failure detail before logging or publishing it.
Line 348 logs raw exc. Line 357 persists raw str(exc) in a local heartbeat event. delivery.wait() can raise exception text that contains credentials, connection strings, or PII. Use the repository sanitization utility and publish a stable diagnostic value.
As per coding guidelines, “Error messages must never expose passwords, API keys, PII, or credential-bearing connection strings; use the repository sanitization utilities.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/omnibase_infra/runtime/gateway_forwarder.py` around lines 342 - 358,
Sanitize the delivery exception detail before both the warning logger call and
the degraded status publication in the gateway delivery loop. Use the
repository’s existing sanitization utility on the exception text, then reuse the
sanitized, stable diagnostic value for the logged error and the
_publish_gateway_status detail instead of raw exc or str(exc).
Source: Coding guidelines
Comment on lines
+407
to
+412
| try: | ||
| await forwarder.publish_status( | ||
| status, | ||
| consecutive_failures=consecutive_failures, | ||
| detail=detail, | ||
| ) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Keep status publication bounded.
publish_status() uses _publish_with_delivery_retry(), which retries RuntimeHostError indefinitely. If the local bus is unavailable, Line 408 never returns and the reconnect supervisor stops retrying the cloud leg. Apply a finite deadline or attempt budget to this best-effort status publish, then continue supervision.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/omnibase_infra/runtime/gateway_forwarder.py` around lines 407 - 412,
Bound the best-effort status publication around forwarder.publish_status so
_publish_with_delivery_retry cannot retry RuntimeHostError indefinitely. Apply a
finite deadline or attempt budget to this call, handle expiry or exhaustion, and
allow the reconnect supervisor to continue its cloud-leg supervision.
This was referenced Aug 9, 2026
jonahgabriel
added a commit
that referenced
this pull request
Aug 9, 2026
…ient pin-reachability timeout Tests (Split 1/15) failed on tests/integration/ci/test_pin_reachability_live_omn15538.py with "transport error: The read operation timed out" resolving the pre-existing core-ref pin in ci.yml/hostile-reviewer.yml (unrelated to this PR's diff -- neither line is touched here; the stale pin itself is OMN-15725's scope, PR #2680 still open). Confirmed transient, not a durable unreachable-pin state: sibling PR #2688/#2697 passed this same check hours ago; sibling PR #2701 failed the identical check in the same ~30-40min window as this PR, consistent with a shared transient GitHub API rate-limit/timeout window (cf. OMN-14352) rather than a code-caused or permanently-broken pin.
jonahgabriel
added a commit
that referenced
this pull request
Aug 10, 2026
…d PR events (#2703) * fix(OMN-14241): occ-preflight eval-path callers must listen for `edited` PR events ci.yml (implicit default types, no `edited`) and hostile-reviewer.yml (`types: [opened, synchronize, reopened]`) each declare a top-level `occ-preflight` job calling the canonical reusable occ-preflight.yml, but neither retriggers when a PR body is edited without a new commit. The reusable workflow already live-fetches the current PR body via `gh pr view` on every run it does execute -- the defect was never a frozen-body read, it was that no new run fired at all when a body-only edit (typically stamping `Evidence-Source: OCC#<n>` after the companion OCC PR merges) landed. The last commit-triggered run -- correctly FAILURE before the stamp existed -- then sits as the permanent `occ-preflight / eligibility` status, and CI Summary fails closed on it forever without a manual empty-commit retrigger. Live-root-caused on infra#2696 and #2694 (2026-08-09): both PRs stamped Evidence-Source via a body edit; CI's and Hostile Reviewer's occ-preflight copies stayed FAILURE from before the edit while call-reject-skip.yml's copy (which already lists `edited`) correctly re-ran and passed. Adds tests/ci/test_occ_eval_path_trigger_coverage.py (RED before this change on both workflows, GREEN after), sibling to the existing test_occ_born_path_trigger_coverage.py (OMN-14987) guard for the autobind/companion-effect minting workflows. OMN-14241 (adopted existing Backlog ticket; relatedTo OMN-15725/OMN-15214 lineage). No debounce/diff guard added -- today's actual behavior already pays a full-CI-run cost per stamp fix via manual empty-commit retriggers, so automating the same retrigger via `edited` is net-neutral CI cost, not new cost; it removes the manual-toil failure mode currently blocking both PRs. * chore(OMN-14241): [mergesweep-0809-infraunblock] retrigger CI Deploy Agent Tests (OMN-15378) failed once with a docker-teardown flake unrelated to this PR's diff (subprocess.TimeoutExpired on `docker rm -f` in test_executor_digest_verification_image_id_omn15181.py, a file this PR never touches). First-strike retry per two-strike allowance. * chore(OMN-14241): [mergesweep-0809-infraunblock] retrigger CodeRabbit gate gate/CodeRabbit Thread Check's prior run (31333667026) was CANCELLED by a concurrent same-SHA run collision, not a real failure (log confirmed "Unresolved CodeRabbit threads: 0"). An issue_comment retrigger attempt resolved to the wrong head SHA (dev tip, not this PR's head) due to how cr-thread-gate-caller.yml's issue_comment path checks out ref -- a genuine workflow quirk, not applicable to this PR's actual commit. Using a synchronize-triggering empty commit instead, which correctly targets this PR's head per the workflow's pull_request trigger. * chore(OMN-14241): [mergesweep-0809-infraunblock] retrigger CI Summary poll CI Summary FAILED (run 31334333041) purely on poller-deadline timing: its own log shows CodeQL still listed as "missing/pending" at the 20:37:31Z deadline, but CodeQL's own run (31334296327) completed SUCCESS at 20:36:49Z -- 42 seconds earlier. A fresh poll will observe CodeQL as already-passed. Not a repeat of the same failure class (prior strikes were Deploy Agent Tests docker-teardown timeout and a CodeRabbit gate wrong-SHA issue) -- this is a first-instance poller/job-completion race. * chore(OMN-14241): [mergesweep-0809-infraunblock] retrigger past known flake OMN-15749 Deploy Agent Tests failed twice on TestRepoDigestsOnContainerIsRealDockerFailureMode (docker create / docker rm -f timing out at 30s). Root-caused, not transient hand-waving: this is the pre-existing, already-ticketed OMN-15749 ("deploy-agent digest-verification fixture tests time out on real docker create/docker rm -f -- 30s too short under runner contention", Backlog, unrelated to this PR's diff -- this PR touches zero deploy-agent files). Confirmed non-systemic: sibling PRs #2701/#2702/#2700 all passed the identical test on concurrent runs at the same time, so this is isolated runner-instance docker contention, not a fleet-wide or code-caused failure. Retrying past the known flake rather than blocking an unrelated CI-workflow-trigger fix on it. * chore(OMN-14241): [mergesweep-0809-infraunblock] retrigger past transient pin-reachability timeout Tests (Split 1/15) failed on tests/integration/ci/test_pin_reachability_live_omn15538.py with "transport error: The read operation timed out" resolving the pre-existing core-ref pin in ci.yml/hostile-reviewer.yml (unrelated to this PR's diff -- neither line is touched here; the stale pin itself is OMN-15725's scope, PR #2680 still open). Confirmed transient, not a durable unreachable-pin state: sibling PR #2688/#2697 passed this same check hours ago; sibling PR #2701 failed the identical check in the same ~30-40min window as this PR, consistent with a shared transient GitHub API rate-limit/timeout window (cf. OMN-14352) rather than a code-caused or permanently-broken pin.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LANDING ORDER (mandatory, read before merging)
Landing order: #2689 -> #2688 -> #2692; #2690 -> #2687 (HELD). Content-stacked despite base=dev — out-of-order squash-merge folds multiple tickets into one PR.
Summary
OMN-15742 (G2, gateway-lift Phase 0). Source of truth:
docs/design/2026-08-08-gateway-node-architecture-lift.md(commit9422d30b1) axis #10 + §1.2 live restart evidence, anddocs/design/2026-08-08-gateway-path-implementation-inventory.md(b9d439c79) §5.Ticket: OMN-15742
Evidence-Source: OCC#6206
Evidence-Ticket: OMN-15742
Replaces the terminal-exit-on-cloud-leg-failure supervision in
src/omnibase_infra/runtime/gateway_forwarder.pywith a bounded reconnect-backoff loop and a bus-observableDEGRADEDstatus:_supervise_gateway_delivery()wraps the existingNodeGatewayDelivery.wait()/shutdown_event.wait()race. A delivery failure (cloud-leg drop, transient network fault) now retries with bounded exponential backoff + jitter (contract-declaredreconnect_backoff_initial_seconds/_max_seconds/_jitter_seconds) instead of propagating out ofrun_gateway_forwarderand killing the process.consecutive_failures/first_failure_at) only resets after the delivery loop survives a fullheartbeat_interval_seconds"recovery confirm" window without failing again — a baredelivery.start()call succeeding only proves the coroutines were rescheduled, not that the cloud leg is actually reachable, so it is deliberately not treated as recovery on its own.degraded_after_seconds, oneDEGRADEDstatus event publishes via the newServiceGatewayForwarder.publish_status(), reusing the existingonex.evt.omnibase-infra.gateway-heartbeat.v1event shape (ModelGatewayHeartbeatgainsstatus="degraded"+consecutive_failures+detail). Published on the local bus (not cloud, unlikepublish_heartbeat) soDEGRADEDstays observable while the cloud leg that caused it is itself unreachable.contract.yamlliveness block gains the 4 new reconnect/degradation knobs (contract_version/node_version patch bump0.1.0→0.1.1);ModelGatewayForwarderConfiggains the matching typed fields + amax >= initialvalidator.NodeGatewayDelivery,KafkaTransport) is untouched — supervision wraps it, per scope. No restructuring.Migration-sync trap (disclosed)
infra#2683(OMN-15732, split-migration rework) was not merged at push time (verified viagh pr view 2683— stillOPEN). This branch therefore vendored the 2 files the repo-wide node-migration-sync hook required (scripts/sync-node-migrations.sh):nodes/node_canary_score_reducer/0003_capability_scores_tenant_id_to_uuid.sqlandnodes/node_projection_registration/0004_node_service_registry_no_force_rls.sql, plus their TSV declarations indocker/migrations/forward/_ledger/application-migrations.tsv(required separately byvalidate_application_migration_manifest.py, only surfaced on the full local pre-push gate). These 2 files duplicate the in-flight #2683 resolution — this PR must be rebased ontodev(picking up #2683's shape) once it lands. No_LEGACY_DEFAULT_SCHEMA_SQL_EXACT_PATHSor exemption/validator logic was touched.Known live CI consequence of this trap (disclosed, not worked around):
Application Database Domain Enforcement (OMN-15361)is RED on this PR. Root cause per its own log: both vendored files fail static domain-ownership/dynamic-SQL analysis (capability_scores requires exactly one ownership declaration; both files contain "dynamic SQL whose relation targets cannot be proven statically"). These are unmodified copies of omnimarket-owned migrations vendored verbatim by the sync script — I did not hand-edit their SQL, and per the hard rule I did not touch any domain-enforcement exemption/validator logic to force this green. This gate is very likely exactly what #2683's split-migration rework exists to fix; the correct remediation is rebasing onto dev after #2683 lands, not patching around this gate here.CI report (honest, live-verified — updated after PR open)
.venv, this Mac — machine was contended with 3 other parallel gateway-lift lanes, hence the ~9.5min wall time):23174 passed, 40 skipped(0 failed) after the migration-manifest fix below. First local run surfaced 1 pre-existing migration-vendor-sync failure (fixed by vendoring) and 2 manifest-declaration failures (fixed) — both root-caused and fixed in this branch, not worked around.src/files:Success: no issues found.ONEX Pattern Validation,ONEX Any Type Validation,ONEX Node Migration Vendor Sync Check,Topic Contract Parity Gate.full_suite_reason: shared_module, contract.yaml touched) and passed — expected, correct behavior per repo rule feat: Complete infrastructure containers operational with Docker secrets #4.deploy-gate: PASSline below is stale/wrong for the current head. A new local commit7e99107c1(fix: stop outbound consumer loop re-forwarding local-only publishes) exists on this branch but had not reachedoriginas of this correction — local pre-push gate was still running (60+ min elapsed, in progress). Livegh pr checks 2688on the still-current remote headf0cdfe43showsdeploy-gate / deploy-gate: FAILandCI Summary: FAIL(re-verified just now). Correct status: deploy-gate is FAIL pending fresh CI on7e99107c1once that commit's push completes and a new CI run starts. Do not read the PASS claim two lines below as current.deploy-gate / deploy-gate: PASS (after repairing the auto-minted OCC contract's non-falsifiabledod-deploy-assessmentprobe — see below).occ-preflight / eligibility: PASS (Evidence-Source autobind landed OCC#6206).Application Database Domain Enforcement (OMN-15361): FAIL, expected/disclosed migration-sync-trap consequence (see above) — not a defect in this PR's own code.CI Summary,Enable Auto-Merge,OCC Companion Merged Gate: currently red/pending on stale sub-checks from the pre-Evidence-Source run window; a full-workflow rerun is in flight as of this update to refresh them against the current (green) OCC state. Will not chase these further in this build lane — they are receipt-plumbing status, not code correctness, and this workflow does not merge.Tests (Split N/15)CI fleet is running as of this update.dod-deploy-assessmentcheck usedgh pr diff | grep -c, which greps the PR diff text itself — not falsifiable against deployed state, and rejected by theOMN-14443deploy-gate falsifiability ratchet. Replaced (3 follow-up commits directly on the bot'sauto/omninode-ai-omnibase_infra-pr-2688-occ-autobindbranch — permitted per the hard rule allowing an OCC companion to be authored/amended when the gate requires it) withgh api .../contents/<path>?ref=<head_sha> | grep -q '_supervise_gateway_delivery', a live fetch of the actually-pushed file whose exit status depends on real GitHub state.contract_sha256/contract_entry_sha256were tool-computed viaomnibase_core.validation.validator_receipt_gate.compute_contract_sha256/compute_contract_entry_sha256, never hand-authored. One of those follow-ups also hit and fixed a documented yamlfmt v0.21 sentinel-corruption bug (OMN-15479,#magic___^_^___lineinjected into a folded>block scalar) by rewriting the field as a single-line quoted scalar instead..201/AWS mutation was performed; build + PR only.Test plan
test_supervise_gateway_delivery_retries_without_raising,test_supervise_gateway_delivery_emits_degraded_then_recovers(tests/unit/runtime/test_gateway_forwarder_runtime.py) — drive the real_supervise_gateway_delivery()against fake delivery/forwarder doubles, asserting restart counts, backoff behavior, and the["degraded", "active"]publish sequence.test_publish_status_degraded_goes_to_local_bus_not_cloud,test_publish_status_active_defaults_zero_failures_and_no_detail(tests/unit/nodes/node_bus_forwarder_effect/test_gateway_forwarder_service.py).test_config_requires_reconnect_backoff_max_at_least_initial_delay,test_config_reconnect_defaults_match_contract(tests/unit/nodes/node_bus_forwarder_effect/test_gateway_forwarder_config.py).publish_heartbeatshape/topic untouched).CI status — final honest snapshot at end of this build lane
As of the last
gh pr checks 2688read:SUPERSEDED — see the 2026-08-08 correction above the "Live remote CI" section. The table below reflects the state as of the prior head; it is not current for
7e99107c1.deploy-gate / deploy-gatef0cdfe43, now re-measured FAIL live; pending fresh CI on7e99107c1occ-preflight / eligibilityApplication Database Domain Enforcement (OMN-15361)Tests (Split 1/15)##[error]The runner has received a shutdown signal/The operation was canceled) triggered by re-running the whole workflow while a prior partial rerun was still in flight — not a code failure. The local full unit suite (23174 passed, 0 failed) already exercises this same test set green.CI Tests Gate,CI Summary,verify / verifyEnable Auto-MergeHonest bottom line: this PR's own code is proven (full local suite green, mypy --strict clean, all new/existing unit tests for this change green, deploy-gate green). The 2 remaining CI reds are (1) the disclosed, already-explained migration-sync-trap gap that #2683 is designed to close, and (2) a runner-level cancellation from my own re-run action, not a defect surfaced by the code in this diff. I stopped chasing further reruns per the two-strike rule (this is CI-infra flake, not a repeat code failure) and am reporting the state as-is rather than working around it. This workflow does not merge; a human/downstream lane should re-run the cancelled split once, and re-evaluate OMN-15361 after rebasing onto dev post-#2683.
Summary by CodeRabbit
New Features
Bug Fixes
Tests