Skip to content

fix(OMN-20590): the lane's main runtime publishes the runtime tick, as an envelope the consumers accept - #4575

Merged
jonahgabriel merged 3 commits into
devfrom
jonah/omn-20590-runtime-tick-producer
Oct 5, 2026
Merged

jonahgabriel merged 3 commits into
devfrom
jonah/omn-20590-runtime-tick-producer

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Closes OMN-20590

Summary

This pull request starts the runtime tick producer that seven existing contract subscribers wait on, which no runtime started until now. It also publishes each tick as an envelope with the event_type those contracts route on, so the tick carries its own lineage. The implementation introduces an opt-in mechanism controlled by the ONEX_RUNTIME_SCHEDULER_ENABLED environment variable. Only the main role is permitted to publish ticks to prevent duplicate emissions in shared environments. The change is validated through unit tests and a controlled lab proof on the dev-202 lane.

Symptom (read live 2026-10-05)

Seven contracts subscribe onex.intent.platform.runtime-tick.v1 and nothing published it. HIGH-WATERMARK / LOG-START on the tick topic: .201 dev lane 0 / 0, .201 stability-test 0 / 0, k3s onex-lab 0 / 0 (positive control, onex.evt.platform.node-heartbeat.v1 on the same brokers: 592547, 278271, 230843). dev-202 held 126 records, every one a synthetic tick hand-published by another lane (two tick ids, 63 copies each).

Cause

  1. RuntimeScheduler (src/omnibase_infra/runtime/runtime_scheduler.py, OMN-953) had no instantiation site in src. service_kernel.py only starts ContractRegistrationEventRouter, whose tick is in-process (metadata topic __internal_tick__) and never reaches the bus. No compose, onex-lab, or onex-dev overlay sets any ONEX_RUNTIME_SCHEDULER_ variable, meaning there was no switch to turn on.
  2. emit_tick published a bare ModelRuntimeTick through publish() with header event_type runtime.tick.v1, while every subscribing contract routes on event_type platform.runtime-tick. The two consume paths treat a bare tick differently: EventBusSubcontractWiring._deserialize_to_envelope refuses it outright (ModelEventEnvelope.model_validate fails on the missing payload key, checked 2026-10-05), and the auto-wiring consume boundary, which the seven tick subscribers use, wraps it in a synthesized envelope whose minted id carries no lineage (OMN-18958). A scratch run of the integration chain below with a bare tick did reach the poller handler, so part 2 is a shape and lineage defect, not the reason nothing fired. Part 1 alone is why no lane ever saw a tick.

Change

  • src/omnibase_infra/runtime/runtime_scheduler.py: emit_tick now publishes ModelEventEnvelope[ModelRuntimeTick] with event_type platform.runtime-tick (constant RUNTIME_TICK_EVENT_TYPE), source_tool runtime-scheduler.<scheduler_id>, via publish_envelope(envelope=, topic=, key=scheduler_id). The constructor takes ProtocolEventBusLike instead of EventBusKafka. New function start_lane_runtime_scheduler(event_bus, runtime_profile) returns None for any profile other than main; for main, it returns None and logs a WARNING naming the tick topic and the switch when ONEX_RUNTIME_SCHEDULER_ENABLED is not true; otherwise it starts and returns the scheduler.
  • runtime_profile.py: resolve_runtime_scheduler_enabled() reads ONEX_RUNTIME_SCHEDULER_ENABLED at the existing lane-policy env boundary. Unset or blank is false; true/1/yes/on and false/0/no/off are accepted; anything else raises ProtocolConfigurationError instead of reading as false.
  • service_kernel.py: Step 9.6b starts the producer after the contract registry consumers; both shutdown paths call its idempotent stop().
  • docker/docker-compose.dev-202.yml: omninode-runtime sets ONEX_RUNTIME_SCHEDULER_ENABLED "true"; runtime-effects sets ONEX_DEAD_LETTER_ARCHIVE_DIR=/app/data/dead-letter-archive and ONEX_CONSUMER_FLOW_ARCHIVE_DIR=/app/data/consumer-flow-archive, because both prune effects resolve those names on every tick and on dev-202 each tick raised KeyError ONEX_CONSUMER_FLOW_ARCHIVE_DIR and was dead-lettered.

Why opt-in per lane

Only the main role publishes, so roles sharing an env anchor never double ticks. The first ticks a lane receives activate every subscriber, including two prunes configured dry_run false and the delegation reaper, so a lane changes only by an explicit overlay edit.

Tests

  • tests/unit/runtime/test_runtime_scheduler_wiring.py: Validates switch parsing including refusal of unparseable values; main+switch starts and publishes an envelope; effects/workers/projection-api never start one; main without switch starts none and warns naming topic and switch; dev-202 overlay sets the switch on omninode-runtime and the two archive dirs on runtime-effects; the switch appears in docker-compose.dev-202.yml only; an envelope survives the consumer deserializer shape.
  • tests/unit/runtime/test_runtime_scheduler.py: Moved from publish to publish_envelope and gained test_tick_published_as_envelope_the_consumers_accept.
  • Focused run: 121 passed (scheduler, scheduler_valkey, wiring), plus test_omn19494_compose_runtime_lane 73 passed with wiring.
  • tests/integration/runtime/test_runtime_tick_producer_golden_chain_omn20590.py: start_lane_runtime_scheduler (main, switch on) -> emit_tick -> in-memory bus -> auto-wired runtime-tick subscription -> HandlerGitHubApiPoll -> result topic, 1 PR polled, no errors. Passes together with test_github_api_poll_golden_chain.
  • RED on parent: the new unit and integration modules fail at collection because resolve_runtime_scheduler_enabled, start_lane_runtime_scheduler and RUNTIME_TICK_EVENT_TYPE do not exist there.

Lab proof (dev-202)

  • Hotpatch of the main runtime only (omninode-dev-202-runtime) with this branch's three Python files; image copies of those files were byte-identical to the base (md5 34f49241..., 368d6931..., 51961e74...).
  • Switch exported in the entrypoint for the window; tick topic pointed at a scratch topic onex.intent.platform.runtime-tick-proof-omn20590.v1 so a concurrent lane's replay-storm measurement on the real tick topic was not disturbed.
  • Boot log 09:55:36Z: "Runtime tick producer started: scheduler_id=dev-202-omn20590-proof topic=onex.intent.platform.runtime-tick-proof-omn20590.v1 interval_ms=1000"; kernel started in 326.516s (the lane's normal boot, previous boot 325.357s).
  • Scratch topic HWM 14 at 09:55:51Z and 23 at 09:56:01Z (delta 9 in 10 s).
  • Three records consumed and decoded inside the container with ModelEventEnvelope[object].model_validate then ModelRuntimeTick.model_validate: event_type platform.runtime-tick, scheduler_id dev-202-omn20590-proof, seq 21, 22, 23, monotonic.
  • runtime-effects and runtime-worker logged no producer start.
  • Same-shaped envelope ticks already on the real topic reached the registration orchestrator ("RuntimeTick timeout coordination completed") with 0 deserialize errors in the main runtime since 09:00Z.
  • Restore: original four files put back from the in-container backup (md5 matches image), runtime restarted 09:56:29Z and healthy again (boot 324.070s, no producer start logged), scratch topic stopped at HWM 47 and deleted, backups removed.

Not changed

  • dev-202 is the only lane opted in, and test_overlay_flag_is_set_by_dev_202_alone pins that list, so the next lane is a reviewed diff.
  • .201 dev lane, .201 stability, judge, lakshman, k3s onex-lab and onex-prod overlays; they publish no tick until an overlay edit opts them in.
  • The .201 lane hosts the Tech Week demo and was only read.

Landing order

Independent of other open PRs; the prune handler-side lazy archive read and DLQ replay dedup belong to OMN-20589, which this PR does not touch.

Evidence-Ticket: OMN-20590
Evidence-Source: OCC#12894

…s an envelope the consumers accept

RuntimeScheduler (OMN-953) had no instantiation site, so nothing published
onex.intent.platform.runtime-tick.v1 on any lane, and its emit_tick published a
bare ModelRuntimeTick that every auto-wired consumer refuses. The scheduler now
publishes a ModelEventEnvelope with event_type platform.runtime-tick through
publish_envelope; the kernel starts one in the main profile when the lane
overlay sets ONEX_RUNTIME_SCHEDULER_ENABLED, and warns at boot when it does
not. dev-202 is the one lane that opts in.

Onex-Lane: runtime-tick-9143
Onex-Session: 4e719e884f5d4b20a09ca235eed9662e
…the ticks now reach

Both prune effects resolve their local archive directory on every tick. On
dev-202 neither name was set, so each tick raised KeyError and was
dead-lettered. The directories live on the role's own data volume.

Onex-Lane: runtime-tick-9143
Onex-Session: 4e719e884f5d4b20a09ca235eed9662e
@jonahgabriel
jonahgabriel enabled auto-merge October 5, 2026 10:02
…andler end to end

Adds the integration chain start_lane_runtime_scheduler -> emit_tick -> bus
-> auto-wired runtime-tick subscription -> HandlerGitHubApiPoll. Corrects the
envelope rationale: a bare tick is refused by the subcontract-wiring path and
wrapped with a lineage-less minted id at the auto-wiring boundary; it is not
refused by every consumer.

Onex-Lane: runtime-tick-9143
Onex-Session: 4e719e884f5d4b20a09ca235eed9662e
@jonahgabriel

Copy link
Copy Markdown
Collaborator Author

Lab: host=h202 lane=dev-202 (compose project omnibase-infra-dev-202) head=f2eb0ca9c command=hotpatch of omninode-dev-202-runtime with this branch's runtime_scheduler.py, runtime_profile.py, service_kernel.py plus ONEX_RUNTIME_SCHEDULER_ENABLED=true exported in the entrypoint, tick topic pointed at the scratch topic onex.intent.platform.runtime-tick-proof-omn20590.v1, then rpk topic describe twice 10 s apart and an in-container decode. observed=kernel log 09:55:36Z 'Runtime tick producer started: scheduler_id=dev-202-omn20590-proof ... interval_ms=1000'; scratch HWM 14 -> 23 in 10 s; records decode as ModelEventEnvelope with event_type platform.runtime-tick and a ModelRuntimeTick payload, seq 21-23 monotonic; effects and worker roles started no producer. Restored at 09:56:29Z (md5s match the image, healthy after a 324.070 s boot, no producer started, scratch topic deleted). The later head adds only the integration test and comment and docstring corrections, no runtime code change.

jonahgabriel pushed a commit to OmniNode-ai/onex_change_control that referenced this pull request Oct 5, 2026
…12894)

* evidence(OMN-20590): author OCC companion for OmniNode-ai/omnibase_infra#4575

OCC companion by node_pr_lifecycle_fix_effect (OMN-13317 F1 / OMN-13990 / OMN-14285). Product PR head f2eb0ca9c141158d43fb344f34dcf3e64854a49a.

* evidence(OMN-20590): self-bind OCC#12894 + rebind contract_sha256

---------

Co-authored-by: omnimarket-bot <bot@omninode.ai>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hostile Reviewer — adversarial findings (OMN-17492)

Models succeeded: qwen3-review, gpt-oss-review
Models failed: none
New finding threads: 2
Deduped (already posted on this PR): 0
Below quorum (one model only, reported not threaded): 14
Nit-level findings suppressed: 1

The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.

Below quorum: 14 finding(s) raised by one model only (OMN-18479)

These are reported and NOT dropped, but they get no thread and do not block: a single model's finding no other model reproduced is not evidence enough to stop a merge. Read them; act on them if they are right.

  • [CRITICAL] src/omnibase_infra/runtime/runtime_profile.py (gpt-oss-review) — Missing import for os in runtime_profile | The new function resolve_runtime_scheduler_enabled uses os.environ but the module does not import the os package. Invoking the function will raise a NameErro
  • [CRITICAL] tests/unit/runtime/test_runtime_scheduler.py (gpt-oss-review) — Unit test mock missing publish attribute | tests/unit/runtime/test_runtime_scheduler.py defines mock_event_bus with only a publish_envelope AsyncMock. The test test_tick_published_as_envelope_the_cons
  • [MAJOR] docker/docker-compose.dev-202.yml (qwen3-review) — Docker compose exposes port on 172.17.0.1 (Docker bridge gateway) | The diff adds ports 172.17.0.1:61085:8085 and 172.17.0.1:61086:8085 to the omninode-runtime and runtime-effects services. Binding to
  • [MAJOR] tests/unit/runtime/test_runtime_scheduler_wiring.py (qwen3-review) — No test for the case where a non-main role has an invalid flag value | The test suite covers the case where a main role with an invalid flag value raises ProtocolConfigurationError, but does not test
  • [MAJOR] tests/unit/runtime/test_runtime_scheduler.py (gpt-oss-review) — Incorrect assertion on publish method in envelope test | The same test also checks that no publish call occurred via mock_event_bus.publish.assert_not_called(), but the scheduler now uses publish_enve
  • [MAJOR] tests/integration/runtime/test_runtime_tick_producer_golden_chain_omn20590.py (gpt-oss-review) — Integration test does not clean up scheduler on exception | tests/integration/runtime/test_runtime_tick_producer_golden_chain_omn20590.py creates a scheduler and calls await scheduler.stop() in a fina
  • [MINOR] src/omnibase_infra/runtime/runtime_profile.py (qwen3-review) — resolve_runtime_scheduler_enabled reads env var on every call | The function reads os.environ.get() on every invocation. In the current code, it is called once per boot in start_lane_runtime_scheduler
  • [MINOR] tests/unit/runtime/test_runtime_scheduler_wiring.py (qwen3-review) — Test uses SUFFIX_RUNTIME_TICK but scheduler config uses tick_topic | The test test_main_profile_with_flag_starts_and_publishes_envelope asserts that the topic equals SUFFIX_RUNTIME_TICK, but the sched
  • [MINOR] tests/integration/runtime/test_runtime_tick_producer_golden_chain_omn20590.py (qwen3-review) — Integration test does not verify that the scheduler loop is actually running | The integration test test_lane_producer_tick_drives_a_tick_subscriber calls scheduler.emit_tick() manually and then stops
  • [MINOR] src/omnibase_infra/runtime/runtime_scheduler.py (qwen3-review) — TICK_PRODUCER_PROFILE is a magic string 'main' | The constant TICK_PRODUCER_PROFILE is set to 'main', which is a string literal that must match the value returned by resolve_runtime_profile_name(). If
  • [MINOR] src/omnibase_infra/runtime/runtime_scheduler.py (qwen3-review) — No test for the case where the event bus is not started | The start_lane_runtime_scheduler function assumes that the event bus is already started. If the bus is not started, the scheduler's start() ma
  • [MINOR] src/omnibase_infra/runtime/runtime_scheduler.py (gpt-oss-review) — Event type constant may not match contract expectations | RUNTIME_TICK_EVENT_TYPE is set to "platform.runtime-tick" while existing contracts and documentation reference "runtime.tick.v1" as the event
  • [MINOR] tests/unit/runtime/test_runtime_scheduler_wiring.py (gpt-oss-review) — Unnecessary JSON round‑trip in envelope validation test | tests/unit/runtime/test_runtime_scheduler_wiring.py serialises and deserialises the envelope via json.dumps/json.loads to validate shape. This
  • [MINOR] src/omnibase_infra/runtime/runtime_scheduler.py (gpt-oss-review) — Transport type hard‑coded to KAFKA in error context | RuntimeScheduler.emit_tick constructs ModelInfraErrorContext with transport_type=EnumInfraTransportType.KAFKA regardless of the actual event bus i

Findings demoted from threads (anchor rejected)

  • [MAJOR] hostile-reviewer (qwen3-review)

    start_lane_runtime_scheduler raises ProtocolConfigurationError for non-main roles | The function calls resolve_runtime_scheduler_enabled() before checking the runtime_profile. If a non-main role (e.g., 'effects', 'workers') boots on a lane where ONEX_RUNTIME_SCHEDULER_ENABLED is set to an invalid value like 'ture', the function will raise ProtocolConfigurationError instead of returning None. This means a typo in the env var on a lane will crash non-main roles that should be unaffected by the tick producer s

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

  • [MAJOR] hostile-reviewer (gpt-oss-review)

    Potential mismatch of ModelEventEnvelope field name | RuntimeScheduler.emit_tick constructs ModelEventEnvelope using the keyword argument source_tool, but the typical schema for ModelEventEnvelope uses a field named source. If the model does not define source_tool, instantiation will fail at runtime, preventing tick emission. | Evidence: runtime_scheduler.py: envelope = ModelEventEnvelope(..., source_tool=f"runtime-scheduler.{self.scheduler_id}", ...) | Fix: Verify the ModelEventEnvelope definition; if the

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

✅ Hostile Reviewer — REVIEWED

Critical findings: 2
Major findings: 6
Total findings: 17
Models succeeded: qwen3-review,gpt-oss-review
Models unavailable: none

Action required: findings were posted as review threads. Address or reject each one, then resolve its thread — the Hostile Review Thread Gate fails while hostile-reviewer threads are unresolved (OMN-17492).


Semantics (OMN-17492 — the model finds, thread resolution gates)

Surface Meaning Blocks merge?
Review threads Per-finding, posted by the reviewer No (informational)
Hostile Review Thread Gate Deterministic: unresolved hostile-reviewer threads exist Fails until resolved (not yet a required context)
degraded verdict Fewer than 2 models succeeded (infra) Fails this job with a named reason

Powered by omniintelligence.review_pairing.cli_review — multi-model adversarial review: qwen3-review (Qwen3.8-27B), gpt-oss-review (gpt-oss-120b) (OMN-8468/OMN-8524/OMN-17492)

@jonahgabriel
jonahgabriel added this pull request to the merge queue Oct 5, 2026
Merged via the queue into dev with commit fcf5109 Oct 5, 2026
269 of 281 checks passed
@jonahgabriel
jonahgabriel deleted the jonah/omn-20590-runtime-tick-producer branch October 5, 2026 11:37
@jonahgabriel

Copy link
Copy Markdown
Collaborator Author

Hostile-review disposition (review 5413249388). Both threaded findings were demoted into the review body ("anchor rejected"), so no thread exists to reply under; every finding is answered here. This PR merged at f2eb0ca before the answer; the fix is in omnibase_infra#4579. The per-lane opt-in is unchanged there: dev-202 is still the only overlay that sets the switch.

Demoted from threads

  1. fp=4e7a3db20db2 [MAJOR] a non-main role refuses boot on a bad switch value. Right. start_lane_runtime_scheduler called resolve_runtime_scheduler_enabled() before the role check, so ONEX_RUNTIME_SCHEDULER_ENABLED=ture on a shared compose environment stopped effects, workers and projection-api, which never act on it. Fixed in fix(OMN-20590): only the main role reads the runtime tick switch, so a typo cannot stop a secondary role's boot #4579: the switch is read after the role check; main still refuses. RED-proven by test_unparseable_flag_leaves_a_secondary_role_running[...] (3 failed on dev, pass on fix(OMN-20590): only the main role reads the runtime tick switch, so a typo cannot stop a secondary role's boot #4579) and probed in the dev-202 runtime container.
  2. fp=f38b672bd66d [MAJOR] ModelEventEnvelope has source, not source_tool. Wrong. Against the pinned omnibase_core 0.47.32: 'source_tool' in ModelEventEnvelope.model_fields is True and 'source' is False. The lab readback on this PR decoded the published records as ModelEventEnvelope with event_type platform.runtime-tick.

Below quorum

  1. [CRITICAL] os not imported in runtime_profile. Wrong: import os is line 62 of runtime_profile.py, used by the existing resolvers.
  2. [CRITICAL] mock bus lacks publish. Wrong: the fixture is AsyncMock(spec=ProtocolEventBusLike) and ProtocolEventBusLike declares both publish and publish_envelope, so bus.publish exists and records no calls.
  3. [MAJOR] compose binds 172.17.0.1. Not in this diff: the 172.17.0.1:61085/61086 port lines are pre-existing context in docker-compose.dev-202.yml. This PR adds only environment keys.
  4. [MAJOR] no test for a non-main role with a bad value. Right; added with the fix in fix(OMN-20590): only the main role reads the runtime tick switch, so a typo cannot stop a secondary role's boot #4579.
  5. [MAJOR] publish.assert_not_called() is wrong. It is the point of the test: the scheduler must call publish_envelope and never the bare publish it used before (RED on the parent).
  6. [MAJOR] integration test does not stop the scheduler on exception. The finding's own evidence says scheduler.stop() is in a finally, which runs on exception.
  7. [MINOR] env read on every call. Called once per boot from bootstrap(); there is nothing to cache.
  8. [MINOR] test asserts SUFFIX_RUNTIME_TICK while the scheduler uses tick_topic. Deliberate: ModelRuntimeSchedulerConfig.default().tick_topic is onex.intent.platform.runtime-tick.v1, equal to SUFFIX_RUNTIME_TICK, and the assertion pins the default to the topic the subscribers read.
  9. [MINOR] integration test does not prove the loop runs. The interval loop is covered by the existing _wait_for_ticks tests in test_runtime_scheduler.py; the golden-chain test proves the published shape drives a subscriber. The lane readback on this PR shows the loop live: scratch-topic HWM 14 to 23 in 10 s at interval 1000 ms.
  10. [MINOR] TICK_PRODUCER_PROFILE = "main" is a magic string. It names the registry's main profile. fix(OMN-20590): only the main role reads the runtime tick switch, so a typo cannot stop a secondary role's boot #4579 adds a test that it resolves through load_runtime_profile, so it cannot drift from the registry unnoticed.
  11. [MINOR] no test for an unstarted bus. The kernel starts the bus before step 9.6b, and start_lane_runtime_scheduler is called only there. A publish on an unstarted bus fails in emit_tick and goes through the scheduler's existing circuit breaker and failure path, which this PR does not change.
  12. [MINOR] event type platform.runtime-tick vs runtime.tick.v1. Wrong direction: the subscribing contracts route on event_type: "platform.runtime-tick" (node_registration_orchestrator/contract.yaml:422, node_github_pr_poller_effect/contract.yaml:57). runtime.tick.v1 was the old header value no consumer routes on.
  13. [MINOR] JSON round-trip in the envelope test. Deliberate: it proves the envelope survives the serialization a Kafka bus applies, which a model-to-model check would not.
  14. [MINOR] transport_type=KAFKA in the error context. Pre-existing line, unchanged here; on every lane the bus is Kafka. Only tests and local runs use the in-memory bus.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant