Repository navigation
fix(OMN-15741): path-verifying canary healthcheck for the gateway forwarder (G1) - #2690
Merged
jonahgabriel merged 1 commit intoAug 8, 2026
Merged
Conversation
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 54 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (18)
Comment |
jonahgabriel
added a commit
to OmniNode-ai/onex_change_control
that referenced
this pull request
Aug 8, 2026
…omnibase_infra#2690 (#6212) * evidence(OMN-15741): OCC contract + falsifiable deploy-scope DoD for OmniNode-ai/omnibase_infra#2690 Net-new contract file only. Deploy-scope DoD item probes the actual pushed content of the gateway canary probe module and the docker-compose healthcheck at PR #2690's head commit (11548f36f) via a live gh api content fetch -- exit status depends on the real state of the pushed commit, not a self-authored receipt, per the deploy-gate falsifiability ratchet (OMN-14443). Ticket: OMN-15741. * evidence(OMN-15741): add self-bind item + PASS receipts for all dod_evidence checks Adds occ-self-bind-pr-6212 (binds this contract to its own OCC companion PR, matching the OMN-15742 precedent) and PASS receipts for all four dod_evidence items, computed via omnibase_core.validation.validator_receipt_gate (compute_contract_sha256 / compute_contract_entry_sha256) and verified live against omnibase_core.validation.validator_occ_merge_eligibility locally before pushing (eligible=true). Ticket: OMN-15741.
Contributor
✅ Hostile Reviewer — PASSEDBlocking findings (critical): 0 Gate semantics (pilot phase)
Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8468/OMN-8524) |
…warder (G1) Replace the docker-compose gateway healthcheck test -f /tmp/gateway-forwarder-ready sentinel-file check with a real produce+readback probe (onex-gateway-canary-probe) that dials the local leg and cloud leg independently, using the exact KafkaTransport/credentials real gateway traffic uses, and exits non-zero when either legs path is dead even though the forwarder process is still up -- the exact failure mode of the 2026-08-04 outage (ready-file present, forwarding 0% for 4 days, FailingStreak 57 with the old check still calling it healthy). - New contract-declared gateway_forwarder.canary block (topic, cadence, produce/readback deadlines) is the sole authority for the canary topic and cadence, materialized into the runtime config the same way mirror_topics already is (canary_topic_set selector, no inline redeclaration). - New ModelGatewayCanaryConfig; ModelGatewayForwarderConfig.canary is a required field, validated dedicated (must not collide with mirror_topics). - New onex-gateway-canary-probe CLI (runtime/gateway_canary_probe.py): concurrent local+cloud leg checks, each producing a unique correlation-id record to the dedicated canary topic and confirming readback within the contract deadline; per-leg PASS/FAIL lines land in docker inspect ...State.Health.Log, distinguishing local healthy, cloud dead from both healthy per the tickets acceptance criterion. - Cadence-bound via a --state-file cache so the healthcheck (15s interval) does not spam either broker; a real round trip only runs once per contract-declared canary.cadence_seconds (default 30s). - docker-compose.gateway.yml healthcheck now runs the probe; timeout raised to 25s to cover a real (non-cached) check. Built standalone against omnibase_infra dev, NOT on top of G0/OMN-15740 (PR #2689 is still open/unmerged at time of this build) -- disclosed per operator instruction; no functional dependency on G0s transform-seam wiring, since this ticket only touches the healthcheck/probe surface. Branched from dev after PR #2683 (split-migration rework) merged, so the node-migration-sync hook is satisfied by devs already-split migration shape. That hook still flagged one unrelated live drift between omnimarket canonical clone and infras vendored node_canary_score_reducer/0003 file (the canonical clone was observed live-flapping between two content states during this session -- another concurrent lane editing it); vendored via scripts/sync-node-migrations.sh + matching _ledger/application-migrations.tsv checksum, no hand edits, out of this tickets functional scope, disclosed. Also regenerated docker/runners/runner-image.lock.json via scripts/ci/runner_image_identity.py --mode generate: pyproject.toml gained the onex-gateway-canary-probe script entry, which changes the dependency-manifest binding input to the CI runner image identity. Full local gates: ruff format/check clean, mypy --strict clean (2750 source files), full local unit suite 24616 passed / 4 failed (all 4 confirmed pre-existing xdist cross-worker env-var-leakage flakes unrelated to any file this PR touches -- reproduced green serially with -p no:xdist). Ticket: OMN-15741 (G1 of the gateway node-architecture-lift Phase 0, docs/design/2026-08-08-gateway-node-architecture-lift.md commit 9422d30b1).
jonahgabriel
force-pushed
the
jonah/omn-15741-gateway-path-verifying-healthcheck
branch
from
August 8, 2026 22:24
11548f3 to
38cddb5
Compare
jonahgabriel
added a commit
that referenced
this pull request
Aug 8, 2026
load_gateway_forwarder_runtime_config now requires keyword-only broker_ref_map_path (this PR). onex-gateway-canary-probe (OMN-15741, PR #2690) called it single-arg -- a TypeError on every healthcheck tick once both land. Pairs the caller update with the signature change in the same PR so they can never split: - gateway_canary_probe.py: new required --broker-ref-map CLI arg, threaded into the loader call. - docker-compose.gateway.yml: healthcheck test cmd passes --broker-ref-map /run/gateway/broker-ref-map.yaml (the same file already bind-mounted for the main forwarder process). HOLD: do not land until (a) #2690 (G1) has merged and (b) a durable name-resolution path exists for all 6 broker FQDNs -- Route53 PHZ (OMN-15739 S6.2) or the ref-map carrying all six -- since this PR also deletes the extra_hosts pin that live outbound currently depends on; #2686 is the interim extra_hosts stopgap and is not a substitute for landing this before that path exists. Ticket: OMN-15743
jonahgabriel
added a commit
that referenced
this pull request
Aug 8, 2026
…lock test fixtures Two real pre-push failures caught locally (root-caused, not worked around): 1. Rebasing this branch onto #2690 (G1) produced an add/add conflict on the vendored node_canary_score_reducer/0003 migration; resolving it left two stale trailing duplicate lines in application-migrations.tsv from this branch's own pre-#2683 vendoring commit (0003 with a stale checksum, 0004 with a redundant-but-matching one). validate_application_migration_manifest.py fails closed on the checksum conflict. Removed both trailing dupes; the TSV is now byte-identical to dev's (post-#2683) manifest. 2. Three test_gateway_forwarder_runtime.py cases (test_runtime_config_rejects_inline_cloud_broker_literal, test_runtime_config_fails_closed_on_missing_broker_ref_map, test_runtime_config_rejects_inline_canary_literals) built minimal contract/config fixtures that predate G1's canary block landing on this branch. _materialize_contract_canary_config now runs before _materialize_cloud_broker_ref in load_gateway_forwarder_runtime_config, so the first two raised on the canary check before ever reaching the cloud-broker-ref behavior under test; the third didn't pass the now- required broker_ref_map_path kwarg. Updated all three fixtures to declare canary_topic_set/canary consistently with the other tests in this file and pass broker_ref_map_path. Ticket: OMN-15743
jonahgabriel
deleted the
jonah/omn-15741-gateway-path-verifying-healthcheck
branch
August 8, 2026 23:54
This was referenced Aug 9, 2026
Merged
jonahgabriel
added a commit
that referenced
this pull request
Aug 9, 2026
…t fixtures Two gaps surfaced by the fixed full local suite run (previous commit removed the busy-spin hang that was masking these): - test_gateway_forwarder_config.py: the two reconnect_backoff tests (added by this branch's own reconnect-supervision commit, authored before this branch was rebased onto dev's #2690/OMN-15741 G1 commit) never got canary=_canary() added, unlike every other constructor call in the same file. - test_gateway_forwarder_service_handler_seam.py: pre-existing dev-tip gap (unrelated to this branch's diff) -- same class of fix already applied independently on the OMN-15781 branch. Ticket: OMN-15742
jonahgabriel
added a commit
that referenced
this pull request
Aug 9, 2026
…ixture Rebased onto origin/dev (4 commits: #2689 OMN-15740, #2690 OMN-15741 G1, #2691 lane-census, #2682 CI concurrency) to pick up dev's current tip before merge, closing the gap the PR-merge-ref CI checkout exposed: CI runs against branch+dev merged, not the raw branch head, so a test file added by #2689 (test_gateway_forwarder_service_handler_seam.py) was exercised by CI even though it did not exist on this branch's own tree. That file's _config() fixture predates #2690/G1's new required `canary` field on ModelGatewayForwarderConfig, so it failed 3 tests with a Pydantic "field required" error on any PR whose merge-ref included both commits -- a genuinely pre-existing bug on dev's current tip (confirmed via git stash against origin/dev by a sibling lane earlier today), not something introduced by this PR. Cherry-picked the isolated one-file fixture fix already authored for it (identical hunk from open PR #2697/OMN-15781, which bundles it with unrelated gateway auto_offset_reset work this ticket does not touch) rather than waiting on that PR to land first. Schema fingerprint re-verified unchanged (still 83 migration files, 8d57e961...) after rebase -- no new migrations landed. Local proof: tests/unit/nodes/node_bus_forwarder_effect/test_gateway_forwarder_service_handler_seam.py (3/3 passed), tests/integration/migrations/test_098_omninode_internal_schema_omn15359.py + tests/unit/topology/test_application_database_table_grants.py (57/57 passed). OMN-15359
jonahgabriel
added a commit
that referenced
this pull request
Aug 9, 2026
…physical bridge (#2674) * feat(OMN-15359): physically create omninode_internal schema + extend physical bridge P2-P4 build slice: absorbs the OMN-15426 48-table gap (rolling ledger 2026-08-03T19:2xZ) where handler_wiring issued schema-qualified SQL against contract-declared omninode_internal for the omninode_runtime table domain while the schema physically never existed anywhere in the migration corpus (every table lands unqualified in public). - docker/migrations/forward/098_create_omninode_internal_schema.sql: additive CREATE SCHEMA IF NOT EXISTS omninode_internal inside omnidash_analytics. No table created/moved/altered. Proven live against a real ephemeral Postgres cluster (5 integration tests: created, additive/zero-table, idempotent re-apply, rollback drops the empty schema, rollback fails closed via RESTRICT once a table has landed). - physical_schema_mapping.py: INTERNAL_TABLES_PHYSICALLY_IN_PUBLIC_UNTIL_OMN15359 (41 tables), mirroring the already-shipped TENANT bridge. Grant derivation and handler_wiring now resolve these tables against their real physical location (public) instead of a schema that holds nothing yet. - omninode_runtime gains USAGE ON SCHEMA public in all 3 topology source instances (local/onex-dev/onex-prod) -- a separate real gap the bridge alone doesn't close, mirroring tenant_projection_writer's existing grant. TABLE grants regenerated via generate_application_database_table_grants.py --write (live omnimarket sibling, 57 declarations, --check --prove clean across all 7 profiles) and docker/catalog/database-topology/*.yaml re-rendered. - Updated 5 pre-existing topology tests whose assertions depended on omninode_internal tables resolving against the (formerly nonexistent) omninode_internal grant schema; all now correctly account for the bridge. Deferred (named, not silent): per-family physical table copy (owned by the OMN-15420 cutover-journal machinery via P5 tickets OMN-15426/OMN-15360), owner_omninode_internal/omninode_runtime role creation, and the tenant-side physical move. Does not touch delegation_workflow_state (OMN-15337 lane, ruling R-q) or the OMN-15423 event_bus_events/schema_migrations dispositions -- neither table is in the omninode_runtime grant set this PR bridges. Evidence-Source: OMN-15359 * fix(OMN-15359): repair 2 seam bugs blocking infra#2674 merge Two independent bugs on jonah/omn-15359-p2p4-internal-schema-physical-bridge, both verified live before and after fix: 1. Schema fingerprint artifact was stale (migration_file_count: 82, generated 2026-08-03) relative to the branch's own 83 migration files (098_create_omninode_internal_schema.sql plus 2 vendored node migrations). `check_schema_fingerprint.py verify` reproduced the exact CI failure locally (expected 8d57e961... vs committed 40607d243...); `stamp` regenerates it to match. 2. docker/domain-adapter-proof/prove.py's INTERNAL_TABLE constant still said "generation_events", but this same PR's earlier commit renamed the fixture topology's omninode_internal-domain proof grant target to "future_internal_projection" (tests/fixtures/application_relation_ownership/ topology.yaml) -- generation_events is now bridged to public via INTERNAL_TABLES_PHYSICALLY_IN_PUBLIC_UNTIL_OMN15359, so it no longer belongs in the omninode_internal-native proof path. prove.py's raw DDL/GRANT block still worked (it doesn't read topology), but the topology-driven privilege check in handler_wiring.py's _require_projection_binding_privileges correctly consulted the fixture and found no grant for generation_events, raising exactly the live CI failure: "Projection binding 'omninode_runtime_service' ... lacks declared read privileges ... SELECT on table omninode_internal.generation_events". Renamed the constant (and the one hardcoded SQL string in the cross-domain isolation check) to match the fixture. Also added prove.py to the existing S608 per-file-ignores precedent (controlled module constant, not external input, matching the two existing integration-test entries). Reran docker/domain-adapter-proof/compose.yml locally end-to-end against a real Postgres 16 container after each edit: PASS (OMN-15421 rebuilt- container proof exit 0). Does not touch the two remaining red checks (CI Summary -- cascades from these plus a stale companion-OCC read, should clear on rerun once this lands; OCC Companion Merged Gate -- companion OCC#6185 closed unmerged against a now-stale head SHA, needs a fresh mint against this new head via the autobind/companion-effect pipeline on push). OMN-15359 * fix(OMN-15359): rebase onto dev + repair pre-existing broken canary fixture Rebased onto origin/dev (4 commits: #2689 OMN-15740, #2690 OMN-15741 G1, #2691 lane-census, #2682 CI concurrency) to pick up dev's current tip before merge, closing the gap the PR-merge-ref CI checkout exposed: CI runs against branch+dev merged, not the raw branch head, so a test file added by #2689 (test_gateway_forwarder_service_handler_seam.py) was exercised by CI even though it did not exist on this branch's own tree. That file's _config() fixture predates #2690/G1's new required `canary` field on ModelGatewayForwarderConfig, so it failed 3 tests with a Pydantic "field required" error on any PR whose merge-ref included both commits -- a genuinely pre-existing bug on dev's current tip (confirmed via git stash against origin/dev by a sibling lane earlier today), not something introduced by this PR. Cherry-picked the isolated one-file fixture fix already authored for it (identical hunk from open PR #2697/OMN-15781, which bundles it with unrelated gateway auto_offset_reset work this ticket does not touch) rather than waiting on that PR to land first. Schema fingerprint re-verified unchanged (still 83 migration files, 8d57e961...) after rebase -- no new migrations landed. Local proof: tests/unit/nodes/node_bus_forwarder_effect/test_gateway_forwarder_service_handler_seam.py (3/3 passed), tests/integration/migrations/test_098_omninode_internal_schema_omn15359.py + tests/unit/topology/test_application_database_table_grants.py (57/57 passed). OMN-15359 * fix(OMN-15359): regenerate stale runner-image identity lock Root-caused the "runner image shared_env_digest is stale" failure that was blocking CI Tests Gate (via runner-image-build-smoke and repeated Tests-split identity assertions) -- NOT a runner-fleet flake as it first appeared alongside genuine network-transport flakes today. This PR's own earlier commit added a pyproject.toml entry ([tool.ruff.lint.per-file- ignores] for docker/domain-adapter-proof/prove.py), and shared_env_digest is computed (scripts/ci/runner_image_identity.py::compute_shared_env_digest -> ci_env_digest.compute_digest) partly from pyproject.toml bytes -- so editing that file legitimately re-binds the runner image identity and the lock file needed regenerating in the same PR, per the module's own documented contract ("the lock file records the bound identity; CI verifies the recorded digest against the recomputed one"). Reproduced locally: `runner_image_identity.py --mode verify` failed with the exact recorded/recomputed pair from CI (e0abb945.../b5c24d81...); `--mode generate` + re-verify fixed it. tests/ci/test_runner_image_identity.py 12/12 passed after. OMN-15359 * chore(OMN-15359): retrigger CI (stuck Topic Drift Check job on prior attempt)
jonahgabriel
added a commit
that referenced
this pull request
Aug 9, 2026
…ateway forwarder (#2688) * fix(OMN-15742): bounded reconnect backoff + DEGRADED bus status for gateway forwarder The gateway forwarder's delivery-loop supervision previously raced delivery.wait() against shutdown_event.wait() with no retry: any cloud-leg delivery failure (broker drop, transient network fault) propagated straight out of run_gateway_forwarder and terminated the process (docs/design/2026-08-08-gateway-node-architecture-lift.md axis #10, live restart-count evidence in that doc's 1.2). - runtime/gateway_forwarder.py: new _supervise_gateway_delivery loop wraps the existing delivery.wait()/shutdown race. A delivery failure now retries with bounded exponential backoff + jitter instead of exiting. Reconnect state only resets after the delivery loop survives a full heartbeat_interval_seconds "recovery confirm" window without failing again (a bare delivery.start() succeeding only proves tasks were scheduled, not that the cloud leg is back). Process still exits on shutdown or on an unrecoverable (non-connectivity) failure -- config/heartbeat-task errors. - Once a failure window crosses the contract-declared degraded_after_seconds, one DEGRADED status event publishes via the new ServiceGatewayForwarder.publish_status(), reusing the existing gateway-heartbeat.v1 event shape/topic (ModelGatewayHeartbeat gains status="degraded" + consecutive_failures + detail). Published on the LOCAL bus (not cloud) so DEGRADED stays observable while the cloud leg that caused it is itself unreachable. - contract.yaml: liveness block gains reconnect_backoff_initial_seconds / _max_seconds / _jitter_seconds and degraded_after_seconds (contract_version/node_version patch bump 0.1.0 -> 0.1.1); ModelGatewayForwarderConfig gains the matching typed fields + a max>=initial validator. - Data-plane loop (NodeGatewayDelivery, KafkaTransport) is untouched; supervision wraps it per scope. Also vendors 2 omnimarket node migrations (node_canary_score_reducer/0003, node_projection_registration/0004) via scripts/sync-node-migrations.sh -- required by the repo-wide node-migration-sync pre-commit/CI hook on any omnibase_infra branch. infra#2683 (OMN-15732, split-migration rework) was NOT merged at push time; these 2 files duplicate that in-flight resolution and this PR must be rebased onto dev (picking up #2683's shape) once it lands. No _LEGACY_DEFAULT_SCHEMA_SQL_EXACT_PATHS or exemption/ validator logic touched. Ticket: OMN-15742 (G2, gateway-lift Phase 0) * fix(OMN-15742): stop outbound consumer loop re-forwarding local-only publishes Reconciliation finding D1: publish_status (DEGRADED) publishes directly onto the local bus's canonical outbound topic -- exactly the topic the forwarder's own outbound consumer (NodeGatewayDelivery polling local_consumer, the SAME KafkaTransport object local_bus publishes into in runtime/gateway_forwarder.py:run_gateway_forwarder) is subscribed to. The untagged envelope does not match _forward_outbound_message's loopback skip (only "cloud-to-local"), so it falls through to _prepare_outbound and leaks to the cloud leg -- directly contradicting test_publish_status_degraded_goes_to_local_bus_not_cloud, which only passes today because it uses a bare _MockGatewayBus with no consumer loop (feedback_real_dispatch_path_tests gap). Fix: local-only publishes are now stamped with gateway_direction="local-mirror"; _forward_outbound_message and validate_outbound_message skip both "cloud-to-local" (existing) and "local-mirror" (new) via a shared _LOCAL_ONLY_DIRECTIONS set. The stamping helper is a module-level function, not a method, to stay under the ONEX Pattern Validation method-count threshold on ServiceGatewayForwarder. New test test_real_outbound_consumer_loop_does_not_reforward_degraded_status wires the REAL NodeGatewayDelivery consumer loop against a fake transport that actually connects publish (local_bus) to poll (local_consumer) -- the same object playing both roles, matching the real runtime wiring -- unlike every existing test in this file, which uses two disconnected fakes and so cannot observe this class of bug. Verified RED against the parent commit (f0cdfe4, pre-fix): DEGRADED payload reached the cloud leg. GREEN after this fix. The matching heartbeat-local-mirror half of this bug (#2692, OMN-15570) gets the same fix applied on that branch separately, since it stacks on this one and its own diff must be independently correct. Ticket: OMN-15742 Evidence-Ticket: OMN-15742 * fix(OMN-15742): stop test _Source.poll() from starving the asyncio event loop Root cause of every anomalously long/hanging local pre-push run on this branch throughout this session (observed: 10h33m on the original Mac before this session started per the rolling ledger, then repeated 33min-2h+ hangs on .200 across multiple push attempts -- all traced to the SAME mechanism via a --timeout=60 diagnostic run that caught the stack mid-hang inside NodeGatewayDelivery._run_direction's `while True:` loop). _Source.poll() (a test fake in test_gateway_delivery_service.py) returned `[]` with zero internal `await` suspension points. A coroutine with no real suspension point does not yield control back to the asyncio event loop when awaited -- so when test_real_outbound_consumer_loop_does_not_reforward_degraded_status (added by the prior commit on this branch) drives a REAL NodeGatewayDelivery task loop with _Source as the "idle" cloud consumer, `_run_direction`'s `while True: await source.poll(...)` busy-spins the event loop forever. `task.cancel()` only takes effect at the next real suspension point, so the spinning task is uncancellable: `delivery.stop()` (called in the test's `finally`) awaits `asyncio.gather(*tasks)` on tasks that can never be scheduled to observe their own cancellation, deadlocking the entire test process indefinitely -- explaining every "still running, 100% CPU, no progress" observation this session mistook for slowness rather than a hang. Fix: _Source.poll() now honors timeout_ms via asyncio.sleep(timeout_ms / 1000) before returning empty, matching how a real transport (and this same file's _SharedLocalTransport.poll(), which correctly uses asyncio.wait_for(..., timeout=...)) actually behaves. Every other _Source usage in this file calls deliver_message() directly and never invokes poll() at all, so this costs exactly one test ~50ms and changes nothing else (verified: full file 5/5 passed in 0.20s, vs. hanging past a 60s per-test timeout pre-fix). Ticket: OMN-15742 * fix(OMN-15742): wire required canary field into remaining gateway test fixtures Two gaps surfaced by the fixed full local suite run (previous commit removed the busy-spin hang that was masking these): - test_gateway_forwarder_config.py: the two reconnect_backoff tests (added by this branch's own reconnect-supervision commit, authored before this branch was rebased onto dev's #2690/OMN-15741 G1 commit) never got canary=_canary() added, unlike every other constructor call in the same file. - test_gateway_forwarder_service_handler_seam.py: pre-existing dev-tip gap (unrelated to this branch's diff) -- same class of fix already applied independently on the OMN-15781 branch. Ticket: OMN-15742
jonahgabriel
added a commit
that referenced
this pull request
Aug 9, 2026
…hdog (#2700) * fix(OMN-15742): bounded reconnect backoff + DEGRADED bus status for gateway forwarder The gateway forwarder's delivery-loop supervision previously raced delivery.wait() against shutdown_event.wait() with no retry: any cloud-leg delivery failure (broker drop, transient network fault) propagated straight out of run_gateway_forwarder and terminated the process (docs/design/2026-08-08-gateway-node-architecture-lift.md axis #10, live restart-count evidence in that doc's 1.2). - runtime/gateway_forwarder.py: new _supervise_gateway_delivery loop wraps the existing delivery.wait()/shutdown race. A delivery failure now retries with bounded exponential backoff + jitter instead of exiting. Reconnect state only resets after the delivery loop survives a full heartbeat_interval_seconds "recovery confirm" window without failing again (a bare delivery.start() succeeding only proves tasks were scheduled, not that the cloud leg is back). Process still exits on shutdown or on an unrecoverable (non-connectivity) failure -- config/heartbeat-task errors. - Once a failure window crosses the contract-declared degraded_after_seconds, one DEGRADED status event publishes via the new ServiceGatewayForwarder.publish_status(), reusing the existing gateway-heartbeat.v1 event shape/topic (ModelGatewayHeartbeat gains status="degraded" + consecutive_failures + detail). Published on the LOCAL bus (not cloud) so DEGRADED stays observable while the cloud leg that caused it is itself unreachable. - contract.yaml: liveness block gains reconnect_backoff_initial_seconds / _max_seconds / _jitter_seconds and degraded_after_seconds (contract_version/node_version patch bump 0.1.0 -> 0.1.1); ModelGatewayForwarderConfig gains the matching typed fields + a max>=initial validator. - Data-plane loop (NodeGatewayDelivery, KafkaTransport) is untouched; supervision wraps it per scope. Also vendors 2 omnimarket node migrations (node_canary_score_reducer/0003, node_projection_registration/0004) via scripts/sync-node-migrations.sh -- required by the repo-wide node-migration-sync pre-commit/CI hook on any omnibase_infra branch. infra#2683 (OMN-15732, split-migration rework) was NOT merged at push time; these 2 files duplicate that in-flight resolution and this PR must be rebased onto dev (picking up #2683's shape) once it lands. No _LEGACY_DEFAULT_SCHEMA_SQL_EXACT_PATHS or exemption/ validator logic touched. Ticket: OMN-15742 (G2, gateway-lift Phase 0) * fix(OMN-15742): stop outbound consumer loop re-forwarding local-only publishes Reconciliation finding D1: publish_status (DEGRADED) publishes directly onto the local bus's canonical outbound topic -- exactly the topic the forwarder's own outbound consumer (NodeGatewayDelivery polling local_consumer, the SAME KafkaTransport object local_bus publishes into in runtime/gateway_forwarder.py:run_gateway_forwarder) is subscribed to. The untagged envelope does not match _forward_outbound_message's loopback skip (only "cloud-to-local"), so it falls through to _prepare_outbound and leaks to the cloud leg -- directly contradicting test_publish_status_degraded_goes_to_local_bus_not_cloud, which only passes today because it uses a bare _MockGatewayBus with no consumer loop (feedback_real_dispatch_path_tests gap). Fix: local-only publishes are now stamped with gateway_direction="local-mirror"; _forward_outbound_message and validate_outbound_message skip both "cloud-to-local" (existing) and "local-mirror" (new) via a shared _LOCAL_ONLY_DIRECTIONS set. The stamping helper is a module-level function, not a method, to stay under the ONEX Pattern Validation method-count threshold on ServiceGatewayForwarder. New test test_real_outbound_consumer_loop_does_not_reforward_degraded_status wires the REAL NodeGatewayDelivery consumer loop against a fake transport that actually connects publish (local_bus) to poll (local_consumer) -- the same object playing both roles, matching the real runtime wiring -- unlike every existing test in this file, which uses two disconnected fakes and so cannot observe this class of bug. Verified RED against the parent commit (f0cdfe4, pre-fix): DEGRADED payload reached the cloud leg. GREEN after this fix. The matching heartbeat-local-mirror half of this bug (#2692, OMN-15570) gets the same fix applied on that branch separately, since it stacks on this one and its own diff must be independently correct. Ticket: OMN-15742 Evidence-Ticket: OMN-15742 * fix(OMN-15742): stop test _Source.poll() from starving the asyncio event loop Root cause of every anomalously long/hanging local pre-push run on this branch throughout this session (observed: 10h33m on the original Mac before this session started per the rolling ledger, then repeated 33min-2h+ hangs on .200 across multiple push attempts -- all traced to the SAME mechanism via a --timeout=60 diagnostic run that caught the stack mid-hang inside NodeGatewayDelivery._run_direction's `while True:` loop). _Source.poll() (a test fake in test_gateway_delivery_service.py) returned `[]` with zero internal `await` suspension points. A coroutine with no real suspension point does not yield control back to the asyncio event loop when awaited -- so when test_real_outbound_consumer_loop_does_not_reforward_degraded_status (added by the prior commit on this branch) drives a REAL NodeGatewayDelivery task loop with _Source as the "idle" cloud consumer, `_run_direction`'s `while True: await source.poll(...)` busy-spins the event loop forever. `task.cancel()` only takes effect at the next real suspension point, so the spinning task is uncancellable: `delivery.stop()` (called in the test's `finally`) awaits `asyncio.gather(*tasks)` on tasks that can never be scheduled to observe their own cancellation, deadlocking the entire test process indefinitely -- explaining every "still running, 100% CPU, no progress" observation this session mistook for slowness rather than a hang. Fix: _Source.poll() now honors timeout_ms via asyncio.sleep(timeout_ms / 1000) before returning empty, matching how a real transport (and this same file's _SharedLocalTransport.poll(), which correctly uses asyncio.wait_for(..., timeout=...)) actually behaves. Every other _Source usage in this file calls deliver_message() directly and never invokes poll() at all, so this costs exactly one test ~50ms and changes nothing else (verified: full file 5/5 passed in 0.20s, vs. hanging past a 60s per-test timeout pre-fix). Ticket: OMN-15742 * fix(OMN-15742): wire required canary field into remaining gateway test fixtures Two gaps surfaced by the fixed full local suite run (previous commit removed the busy-spin hang that was masking these): - test_gateway_forwarder_config.py: the two reconnect_backoff tests (added by this branch's own reconnect-supervision commit, authored before this branch was rebased onto dev's #2690/OMN-15741 G1 commit) never got canary=_canary() added, unlike every other constructor call in the same file. - test_gateway_forwarder_service_handler_seam.py: pre-existing dev-tip gap (unrelated to this branch's diff) -- same class of fix already applied independently on the OMN-15781 branch. Ticket: OMN-15742 * fix(OMN-15748): gateway poison-pill quarantine + membership-loss watchdog decode_message() ran before the try/except that nacks, so any undecodable record on a mirrored source topic crashed the process (live-observed 2026-08-08T16:50Z). Nacking it (the ordinary failure path) would also be wrong for a permanently malformed record: nack seeks back to the same offset and re-crashes forever. Quarantine instead: log, best-effort dead-letter via the DLQ topic derived from the record's own topic, commit past the offset, keep the loop alive. Also adds an independent membership-loss watchdog to NodeGatewayDelivery. The live 2026-08-09T10:03:07Z incident was a clean single LeaveGroup with zero exception (aiokafka's client-side max_poll_interval_ms idle-eviction, fired from a separate asyncio task than the poll loop) -- the existing exception-triggered reconnect supervision in runtime/gateway_forwarder.py cannot observe a task that is alive, hung, and never exceptions. The watchdog tracks per-direction last-progress timestamps (updated on poll() completion and deliver_message completion) plus a best-effort consumer.assignment() membership probe, and on staleness force-recreates just the affected direction's transport (a fresh client forces a new group join, sidestepping the permanently-stuck idle clock) and publishes a DEGRADED status. Reuses the previously contract-declared but unwired max_silence_window_seconds config field as the staleness threshold. TDD: both regression tests are RED against pre-fix source (verified via git stash of the source-only diff) and GREEN after. OMN-15748 * fix(OMN-15748): consumer-scoped watchdog recovery, no CancelledError escape Fixes two verifier-confirmed defects in the membership-loss watchdog (mergesweep-0809-poisonpath-verify): - wait()/_run_direction: a watchdog stale_task.cancel() made the supervisor's delivery.wait() (a frozen asyncio.gather snapshot) observe a cancelled child and propagate CancelledError -- a BaseException that escapes gateway_forwarder.py's except Exception and kills the process on every recovery. _run_direction now catches CancelledError and returns cleanly when its direction is watchdog-recovering; wait() re-reads the live task set after every completion instead of gathering a one-time snapshot, so a swapped-in replacement task stays supervised. - Recovery no longer calls close()+start() (which stops BOTH the consumer and producer on a KafkaTransport instance shared across directions -- one direction's consumer is the same object the other direction's publish and status/heartbeat emission use as a producer). Adds KafkaTransport.restart_consumer(), a consumer-only recreate that never touches the producer; _recover_stalled_direction now calls it exclusively. De-vacuums the watchdog regression test (arms delivery.wait() before the stall, matching production ordering) and adds direct KafkaTransport-level coverage proving the producer is untouched across a restart_consumer() call. * fix(OMN-15748): address CodeRabbit findings on watchdog wait() + log sanitization Two live (non-outdated) CodeRabbit threads on infra#2700: - wait()'s asyncio.wait was unbounded, waking only on a tracked task's completion. _recover_stalled_direction splices the replacement task into self._tasks strictly AFTER the stale task has already finished and this loop has already woken/rebuilt `watched` from the stale-only list -- so an unbounded wait then blocks on the remaining old tasks with no further wakeup until one of THEM completes. A real exception on the recovered direction would go unsupervised (only surfacing as an "exception was never retrieved" warning at GC). Fixed by bounding the wait with _watchdog_tick_seconds so the loop always revisits self._tasks, and by also picking up an already-completed-with-exception replacement on that rescan. Added test_watchdog_replacement_task_failure_still_reaches_wait, verified RED against the unbounded wait() (reverted locally, re-applied after) and GREEN after. - _quarantine_undecodable_message logged the raw exception via exc_info=error, bypassing the sanitize_error_message() the DLQ payload path already uses a few lines up. A pydantic ValidationError from decode_message() embeds the offending record content, which is untrusted tenant bus traffic that can carry PII/credential-bearing values. Now logs the sanitized message, matching the DLQ path. * test(OMN-15748): harden test_monitor_logs.py subprocess timeouts against CI fleet contention The self-hosted CI fleet runs many concurrent pytest-split shards per job; interpreter spawn/import can occasionally queue behind that CPU contention. A hardcoded 10s subprocess.run(..., timeout=10) tripped this twice live on unrelated PRs' Split 12/15 (infra#2697 and this PR, both times as three subprocess.TimeoutExpired failures at exactly the 10s bound, unrelated to either PR's own diff -- this file isn't touched by either PR's product diff). Bumped to a shared _SUBPROCESS_TIMEOUT_SECONDS = 30 constant across all three call sites: still tight enough to catch a genuine hang, with real headroom against fleet contention.
jonahgabriel
added a commit
that referenced
this pull request
Aug 10, 2026
…rops backlog during outage windows (#2697) * fix(OMN-15781): gateway forwarder auto_offset_reset=latest silently drops backlog during outage windows beta-gateway-canary.yaml set auto_offset_reset="latest" on both local_bus and cloud_bus legs, overriding KafkaTransport's own documented "earliest" default for a pull-based at-least-once consumer. enable_auto_commit=false only protects offsets already inside the consumer's read window -- a "latest" leg still silently skips any backlog produced while the consumer group was unjoined (crash, LeaveGroup, cold restart before rejoin), and auto_offset_reset also fires mid-session on OffsetOutOfRangeError, not only on first boot. Live-observed during the 2026-08-04->08-08 gateway outage investigation (OMN-15742): the delegation-completed.v1 MSK topic showed BEGIN=28=END by the time this was root-caused -- 28 events unrecoverable even after a reconnect fix, because they were never held past "latest" in the first place. Fix: - beta-gateway-canary.yaml: both legs -> auto_offset_reset: "earliest". - ModelGatewayForwarderRuntimeConfig._validate_resolved_legs gains a fail-fast check (same pattern as the existing enable_auto_commit check) rejecting any resolved leg with auto_offset_reset != "earliest", so this can't silently regress. - TDD: new unit tests for the validator (reject latest on either leg, accept earliest); test_staging_canary_resolves_topics_from_node_contract now asserts both legs resolve to "earliest" from the live deployed YAML. - Real-dispatch-path proof (tests/integration/gateway/test_gateway_offset_recovery.py, pytest.mark.kafka, broker-gated/skipped without a reachable broker): produces a message to a topic before a consumer group ever joins, then proves a fresh KafkaTransport consumer on that group delivers it under "earliest" and silently skips it under "latest" -- run live against the real .201 broker in this session, both cases verified (2 passed). Also fixes a pre-existing (dev-tip, unrelated to this change, confirmed via git stash) broken fixture in test_gateway_forwarder_service_handler_seam.py: _config() was missing the now-required ModelGatewayForwarderConfig.canary field (added by #2690/OMN-15741, G1, merged after this test's last edit). Scope: config + fail-fast validator + tests only. Does not touch NodeGatewayDelivery/KafkaTransport data-plane code, and does not recover the already-purged 28 events on the live topic (unrecoverable). Ticket: OMN-15781 Evidence-Source: this session, live code read + live broker proof against 192.168.86.201:39092 (2 passed, test_gateway_offset_recovery.py) * fix(OMN-15781): update gateway canary probe test fixtures for auto_offset_reset=earliest ModelGatewayForwarderRuntimeConfig now rejects auto_offset_reset != "earliest" on either leg (previous commit). test_gateway_canary_probe.py's _bus_config() helper and 3 inline cloud_bus fixtures relied on the model default ("latest"), so they started failing validation. Add auto_offset_reset="earliest" explicitly everywhere this file constructs a ModelKafkaEventBusConfig for a gateway leg -- caught by running the full local suite before push (3 failed pre-fix, 7 passed post-fix in tests/unit/runtime/test_gateway_canary_probe.py). Ticket: OMN-15781
jonahgabriel
added a commit
that referenced
this pull request
Aug 16, 2026
load_gateway_forwarder_runtime_config now requires keyword-only broker_ref_map_path (this PR). onex-gateway-canary-probe (OMN-15741, PR #2690) called it single-arg -- a TypeError on every healthcheck tick once both land. Pairs the caller update with the signature change in the same PR so they can never split: - gateway_canary_probe.py: new required --broker-ref-map CLI arg, threaded into the loader call. - docker-compose.gateway.yml: healthcheck test cmd passes --broker-ref-map /run/gateway/broker-ref-map.yaml (the same file already bind-mounted for the main forwarder process). HOLD: do not land until (a) #2690 (G1) has merged and (b) a durable name-resolution path exists for all 6 broker FQDNs -- Route53 PHZ (OMN-15739 S6.2) or the ref-map carrying all six -- since this PR also deletes the extra_hosts pin that live outbound currently depends on; #2686 is the interim extra_hosts stopgap and is not a substitute for landing this before that path exists. Ticket: OMN-15743
jonahgabriel
added a commit
that referenced
this pull request
Aug 16, 2026
…lock test fixtures Two real pre-push failures caught locally (root-caused, not worked around): 1. Rebasing this branch onto #2690 (G1) produced an add/add conflict on the vendored node_canary_score_reducer/0003 migration; resolving it left two stale trailing duplicate lines in application-migrations.tsv from this branch's own pre-#2683 vendoring commit (0003 with a stale checksum, 0004 with a redundant-but-matching one). validate_application_migration_manifest.py fails closed on the checksum conflict. Removed both trailing dupes; the TSV is now byte-identical to dev's (post-#2683) manifest. 2. Three test_gateway_forwarder_runtime.py cases (test_runtime_config_rejects_inline_cloud_broker_literal, test_runtime_config_fails_closed_on_missing_broker_ref_map, test_runtime_config_rejects_inline_canary_literals) built minimal contract/config fixtures that predate G1's canary block landing on this branch. _materialize_contract_canary_config now runs before _materialize_cloud_broker_ref in load_gateway_forwarder_runtime_config, so the first two raised on the canary check before ever reaching the cloud-broker-ref behavior under test; the third didn't pass the now- required broker_ref_map_path kwarg. Updated all three fixtures to declare canary_topic_set/canary consistently with the other tests in this file and pass broker_ref_map_path. Ticket: OMN-15743
jonahgabriel
added a commit
that referenced
this pull request
Aug 16, 2026
…ry (#2687) * fix(OMN-15743): resolve gateway cloud_broker_ref at the effect boundary Deletes the hardcoded Docker extra_hosts bastion-IP override and the duplicated cloud_bus.bootstrap_servers literal in beta-gateway-canary.yaml. The cloud broker endpoint is now resolved from the node contract's gateway_forwarder.cloud_leg.cloud_broker_ref at process startup, via a new _materialize_cloud_broker_ref() boundary function in gateway_forwarder.py that mirrors the existing _materialize_contract_mirror_topics() pattern: contract is sole authority, resolved YAML may not redeclare the literal, and resolution fails closed (ValueError, no fallback) if the ref doesn't match the contract or the operator-supplied broker-ref map is missing or has no entry for the ref. Compose now mounts an operator-supplied GATEWAY_BROKER_REF_MAP_FILE (same required-env-var-mount pattern already used for the AWS Roles Anywhere credential files) instead of baking a bastion IP into extra_hosts. The CLI gained a required --broker-ref-map flag with no default. Also vendors 2 omnimarket node migrations (node_canary_score_reducer/0003, node_projection_registration/0004) via scripts/sync-node-migrations.sh and declares them in application-migrations.tsv -- required because infra#2683 (OMN-15732 split-migration rework) has not yet merged to dev; see PR body for the disclosure and rebase note. OMN-15743 * chore(OMN-15743): trigger CI re-run after Evidence-Source edit * fix(OMN-15743): thread broker-ref-map through the canary probe (seam A) load_gateway_forwarder_runtime_config now requires keyword-only broker_ref_map_path (this PR). onex-gateway-canary-probe (OMN-15741, PR #2690) called it single-arg -- a TypeError on every healthcheck tick once both land. Pairs the caller update with the signature change in the same PR so they can never split: - gateway_canary_probe.py: new required --broker-ref-map CLI arg, threaded into the loader call. - docker-compose.gateway.yml: healthcheck test cmd passes --broker-ref-map /run/gateway/broker-ref-map.yaml (the same file already bind-mounted for the main forwarder process). HOLD: do not land until (a) #2690 (G1) has merged and (b) a durable name-resolution path exists for all 6 broker FQDNs -- Route53 PHZ (OMN-15739 S6.2) or the ref-map carrying all six -- since this PR also deletes the extra_hosts pin that live outbound currently depends on; #2686 is the interim extra_hosts stopgap and is not a substitute for landing this before that path exists. Ticket: OMN-15743 * fix(OMN-15743): repair stale migration-manifest duplicates + canary-block test fixtures Two real pre-push failures caught locally (root-caused, not worked around): 1. Rebasing this branch onto #2690 (G1) produced an add/add conflict on the vendored node_canary_score_reducer/0003 migration; resolving it left two stale trailing duplicate lines in application-migrations.tsv from this branch's own pre-#2683 vendoring commit (0003 with a stale checksum, 0004 with a redundant-but-matching one). validate_application_migration_manifest.py fails closed on the checksum conflict. Removed both trailing dupes; the TSV is now byte-identical to dev's (post-#2683) manifest. 2. Three test_gateway_forwarder_runtime.py cases (test_runtime_config_rejects_inline_cloud_broker_literal, test_runtime_config_fails_closed_on_missing_broker_ref_map, test_runtime_config_rejects_inline_canary_literals) built minimal contract/config fixtures that predate G1's canary block landing on this branch. _materialize_contract_canary_config now runs before _materialize_cloud_broker_ref in load_gateway_forwarder_runtime_config, so the first two raised on the canary check before ever reaching the cloud-broker-ref behavior under test; the third didn't pass the now- required broker_ref_map_path kwarg. Updated all three fixtures to declare canary_topic_set/canary consistently with the other tests in this file and pass broker_ref_map_path. Ticket: OMN-15743
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LANDING ORDER (mandatory, read before merging)
Landing order: #2689 -> #2688 -> #2692; #2690 -> #2687 (HELD). Content-stacked despite base=dev — out-of-order squash-merge folds multiple tickets into one PR.
Summary
OMN-15741 — G1 of the gateway node-architecture-lift Phase 0
(
docs/design/2026-08-08-gateway-node-architecture-lift.md§3, commit9422d30b1).Replaces the gateway container healthcheck's
test -f /tmp/gateway-forwarder-ready(a sentinel file written once at process start, never re-verified) with a real
path-verifying probe (
onex-gateway-canary-probe) that produces a tiny canaryrecord to a dedicated topic and confirms readback across both the local leg
and the cloud leg, using the exact
KafkaTransport+ credentials real gatewaytraffic uses. This is the failure mode of the 2026-08-04 outage: the process was
up and the ready-file was present for 4 days while the cloud leg forwarded 0%,
because the only health signal could not distinguish "connected and forwarding"
from "started once, now silently broken" (
FailingStreakin the container's ownhealth log never moved the needle either — the old check tested the sentinel
file, not the wire).
What changed
gateway_forwarder.canaryblock(
node_bus_forwarder_effect/contract.yaml):topic,cadence_seconds,produce_deadline_seconds,readback_deadline_seconds. Sole authority for thecanary topic/cadence, materialized into the runtime config the same way
mirror_topicsalready is (canary_topic_setselector — resolved deploymentYAML may not redeclare the block inline).
ModelGatewayCanaryConfig;ModelGatewayForwarderConfig.canaryis arequired field, validated dedicated (must not collide with
mirror_topics).onex-gateway-canary-probeCLI (src/omnibase_infra/runtime/gateway_canary_probe.py):concurrent local-leg + cloud-leg checks, each producing a record with a unique
correlation id to the dedicated canary topic and confirming readback within the
contract deadline via the same
KafkaTransportclass real traffic uses. Per-legPASS/FAILlines land indocker inspect ...State.Health.Log, so the healthcheckdistinguishes "local leg healthy, cloud leg dead" from "both legs healthy" — the
ticket's acceptance criterion.
--state-filecache means only one realproduce+readback round trip runs per contract-declared
canary.cadence_seconds(default 30s); the Docker healthcheck's 15s interval mostly serves the cached
result. The canary topic is dedicated (never mirrors tenant traffic) and records
are tiny (a 32-byte correlation id).
docker-compose.gateway.ymlhealthcheck now runs the probe;timeoutraisedfrom 3s to 25s to cover a real (non-cached) check.
Scope / dependency disclosure
omnibase_infradev, NOT on top of G0 (OMN-15740,PR fix(OMN-15740): wire gateway COMPUTE handlers into ServiceGatewayForwarder transform seam #2689) — fix(OMN-15740): wire gateway COMPUTE handlers into ServiceGatewayForwarder transform seam #2689 is still open/unmerged at the time of this build. Per
operator instruction this lane proceeds standalone; there is no functional
dependency on G0's transform-seam wiring since this ticket only touches the
healthcheck/probe surface, not
ServiceGatewayForwarder's transform methods.devafter PR fix(OMN-15732): break infra-dev merge deadlock -- exempt vendor files from OMN-15361 legacy-schema lint #2683 (OMN-15732split-migration rework) merged, so the node-migration-sync pre-commit hook is
satisfied by dev's already-split shape — no vendoring of duplicate/in-flight
migrations was needed for that reason. The hook still flagged one unrelated
live drift:
docker/migrations/forward/nodes/node_canary_score_reducer/0003_*.sqlvs. the live
omnimarketcanonical clone. That clone was observed live-flappingbetween two content states during this session (another concurrent lane
editing it) — vendored + checksummed atomically via
scripts/sync-node-migrations.sh(no hand edits to the SQL), with the matching_ledger/application-migrations.tsvchecksum updated. Out of this ticket'sfunctional scope; included only because the vendor-sync hook is a hard gate.
_LEGACY_DEFAULT_SCHEMA_SQL_EXACT_PATHSand all exemption/validator logic werenot touched.
docker/runners/runner-image.lock.jsonwas regenerated(
scripts/ci/runner_image_identity.py --mode generate) becausepyproject.tomlgained the
onex-gateway-canary-probescript entry, which is adependency-manifest binding input to the CI runner image identity.
CI report (honest, live-verified)
Full local gates run on
.200(stickybeatz-studio), per repo rule 11a:ruff format --check/ruff check— clean (repo-widesrc/+tests/).mypy --strict— clean, 2750 source files.pytest tests/ -m unit) — 24616 passed. 4 failuresobserved once under
-n auto(tests/unit/runtime/test_dependency_materializer.py,tests/unit/runtime/test_model_kafka_producer_config_max_request_size.pyx2,tests/unit/runtime/test_kafka_bootstrap_no_localhost_fallback.py) — reproducedgreen when run serially (
-p no:xdist); confirmed pre-existing cross-workerenv-var-leakage flakes, unrelated to any file this PR touches (none of the four
files are in this diff).
tests/unit/nodes/node_bus_forwarder_effect/,tests/unit/runtime/test_gateway_forwarder_runtime.py,tests/unit/runtime/test_gateway_canary_probe.py,tests/integration/gateway/test_bus_forwarder_contract_integration.py) —51/51 passed, including new coverage for the probe's local-vs-cloud
distinguishing behavior, cadence caching, and the contract-authority
canary_topic_setmaterialization (mirrors the existingmirror_topic_setpattern's redeclare-rejection test).
ruff format,ruff check,mypy, ONEX ArchitectureLayer Validation, the governed impacted-test selector — escalated to the full
suite because
pyproject.tomlchanged — Deploy-scope DoD parity) all passed;the deploy-scope hook reported
NOTICE_COMPANION_UNMERGED(expected/non-blocking:no local OCC contract for OMN-15741/OMN-15740 yet — the hosted
deploy-gateresolves the OCC companion via Evidence-Source and makes the final call).
No live
.201/AWS mutation performed. PR opened only — not merged.Not done in this PR
devis informational-only per repo doctrine;deploy-gateis required andcurrently
NOTICEs rather than fails locally on the missing local OCC contract).If the hosted
deploy-gatecheck comes back red requiring an OCC companion withdeploy-scope DoD evidence, that is a real follow-up, not silently skipped here.
Ticket: OMN-15741
Evidence-Ticket: OMN-15741
Evidence-Source: OCC#6212