Repository navigation
feat(OMN-17304): onex delegate resolves its execution locus — client mode caller, fail-closed dispatch gate - #3361
Conversation
…he lane is live, and refuses instead of degrading (OMN-17295) Requires omnibase-core >= 0.47.2 (RuntimeLocal client mode, core PR for OMN-17304). MUST NOT merge before that release and the pin bump. THE DEFECT Every "dev-lane probe" written as `onex delegate ... --bus kafka` was vacuous. --bus selects the TRANSPORT; it never relocated execution. The orchestrator resolved out of the caller's own venv and decided there, while the receipt was read as a statement about .201. Worse than "it ran locally": it ran TWICE. RuntimeLocal subscribed the entry handler to the command topic it was publishing to, so Redpanda fanned every command to both the lane's consumer group and the CLI's own disjoint one. Measured on the dev lane 2026-08-31T20:06Z (correlation 993819e0-f3e5-4064-a1dd-0b85ee263105): the lane group advanced 151 -> 152 on the real command while the Mac CLI group advanced 53 -> 54, executing a 98-message-old backlog command. The CLI then returned a terminal that was not its own; post-#3064 the correlation join caught it and the run exited `status: error` rather than printing a stranger's answer. WHAT LANDS 1. `--locus {auto,in-process,deployed-lane}`. Locus is now a RESOLVED property, not a side effect of the transport flag. `auto` follows the transport, because that is the real relationship: a shared bus is the only kind another runtime can consume from. An explicit --locus announces itself as an override, so a receipt can tell "the authority chose this" from "a human typed it". 2. A fail-closed dispatch gate. Before publishing, the exact command topic — read from the CONTRACT, never named in this repo — must have a live STABLE consumer group bound to it. Three outcomes, two refuse: * live groups -> proceed, and they go into the record * broker answered, nothing bound -> refuse ("start the runtime") * broker could not be asked -> refuse ("fix the broker address") It never falls back to in-process. A degraded lane probe is worse than no lane probe, because it still prints a receipt. 3. `live_consumer_groups()` in backend_probe: the same wiring-truth signal as `_has_live_consumer_group`, but tri-state. The existing helper collapses "nobody is bound" and "I could not find out" into False, which is correct for a health GRADE and fatal for a dispatch gate. It also asks for cluster metadata FIRST: against a closed port librdkafka retries in the background and `list_consumer_groups` resolves to a clean EMPTY listing — observed live 2026-08-31, and it reads exactly like "no consumers". 4. Locus provenance in the receipt and the capture log: `handler_locus` (read back from the runtime's OWN workflow_result.json, not from the request), `wire_correlation_id`, `orchestrator_distribution` (name, version AND location), and `dispatch_target` (topic, broker, live consumer groups). A LIVENESS CHECK ON THE WRONG TOPIC `resolve_default_bus` passes `authority_topic=SUFFIX_DELEGATION_REQUEST` (`onex.cmd.omnibase-infra.delegation-request.v1`) and its docstring claims liveness on "the exact topic this delegation is about to publish to". It is not: this path publishes to the delegate contract's own `onex.cmd.omnimarket.delegate-skill.v1`. Both carry STABLE groups on the dev lane (verified live), so the OMN-16529 AUTHORITATIVE grade has been passing for reasons unrelated to whether the command would be served. The new gate reads the topic from the contract so the two cannot disagree. RED-THEN-GREEN `tests/unit/cli/test_delegate_locus.py` — 13 tests, RED before the change (`ImportError: cannot import name 'ConsumerGroupLivenessUnknownError'`), green after. Five are counter-assertions on the in-process path: an in-process run must not acquire a broker dependency, must still host its handlers, and must still resolve. Without them "always refuse" and "never dispatch" would satisfy the fail-closed tests while deleting the offline flow. TESTS RESTATED, NOT DELETED `TestBusHelpTextTellsTheTruth` asserted the string "does not change where the work runs" — true when written, false the moment a real remote mode exists. It now asserts the pair (--bus is transport, --locus owns locus) plus the fail-closed promise, so the help text cannot lie in either direction. `TestBusSelection` / `TestExplicitOverrideProvenance` pin `--locus in-process`: their subject is TRANSPORT plumbing, and inheriting the new default would drag a live-consumer precondition into an assertion about a dict. `_override_records` now matches transport override lines only — OMN-17304 added a second override axis, and a counter-assertion that fails on an unrelated locus line has stopped testing what it claims to test. RECEIPT-LEVEL IDENTITY IS NOT THIS PR OMN-17295 AC2 asks the receipt to name the orchestrator. On the dispatched path it now does. On an in-process SUCCESS the receipt's `result` is the node's typed handler result, so `ModelReceiptRuntimeSummary` is not used and the locus lives in the capture log and workflow_result.json instead. The universal answer is `ModelSkillResult.runtime_identity` — OMN-17308, In Progress in omnibase_core on `model_skill_result.py`, whose `ModelRuntimeIdentity.execution_locus` names the venv/container this process lives in. This PR deliberately does NOT add a competing field, and names its own `handler_locus` for the disjoint question "did I do the work". LIVE PROOF Post-fix, correlation f99d5485-1d18-476e-861b-0e81835752cf, `--locus deployed-lane` against the .201 dev lane: * Mac CLI command-topic group: 54 -> 54 (UNCHANGED — never subscribed) * lane group: 152 -> 153 * `docker logs omninode-runtime{,-effects} --since 20m | grep <corr>`: 19 / 7 * receipt: status success, handler_locus `dispatched`, wire_correlation_id == this run's own correlation, dispatch_target naming the live lane group * the lane answered on the local rung (qwen3.8 @ 192.168.86.201:8000), 1 attempt, quality 1.0, zero escalations, zero metered providers Refusals, same build: an unreachable broker and `--locus deployed-lane --bus inmemory` both exit 1 with distinct diagnoses and zero stdout. REBASED 2026-09-09 onto origin/dev (255 commits). Three conflicts, all against work that landed after this commit was written: * cli_delegate.py DEFAULT_BUS comment -- kept dev's AC3 tier-0 wording (#3123, 21aaec9) and added LOCUS_CHOICES beside it. * cli_delegate.py --bus help -- dev's post-AC3 text still asserted "it does not change where the work runs: the delegate orchestrator always executes in-process in this CLI". That sentence is false the moment --locus exists, which the ticket predicted. Restated: --bus is transport, --locus owns locus, and the configured-authority resolution AC3 landed is preserved verbatim. * receipt_mode.py run_receipt_mode signature and test_cli_delegate.py imports -- union, both additive. Verified after rebase: ruff check + format clean, mypy --strict clean on src/omnibase_infra/{cli,backends,enums} (the 52 remaining errors are all in a sibling omnibase_core worktree checkout, none in this repo), and tests/unit/cli/test_delegate_locus.py + test_cli_delegate.py -> 89 passed.
There was a problem hiding this comment.
Hostile Reviewer — adversarial findings (OMN-17492)
Models succeeded: glm-review
Models failed: codex
New finding threads: 1
Deduped (already posted on this PR): 0
Nit-level findings suppressed: 2
The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.
Findings not anchored to a changed file
-
[MAJOR] hostile-reviewer (glm-review)
TOCTOU race between consumer-group liveness check and publish | The fail-closed gate verifies a STABLE consumer group exists, then publishes at some later point (after receipt setup, capture logging, and runtime construction). A group can rebalance, leave, or die between the check and the publish; the command then lands on an unattended topic, which is precisely the scenario the gate exists to prevent. The check is evidence of a past instant, not a guarantee about the publish moment, and nothing in the diff
Resolve this thread when addressed — the
Hostile Review Thread Gateblocks while hostile-reviewer threads are unresolved (OMN-17492). -
[MINOR] hostile-reviewer (glm-review)
AUTO resolves to DEPLOYED_LANE for existing --bus kafka callers | Any existing invocation that used --bus kafka for its documented original purpose (route this run's events through the shared broker while executing in-process) now silently becomes a remote dispatch, adds a multi-second broker interrogation, and refuses outright when KAFKA_BOOTSTRAP_SERVERS is unset or the topic check fails. This is a breaking change to the CLI's observable contract hidden behind a new default. The help text was updated, but
Resolve this thread when addressed — the
Hostile Review Thread Gateblocks while hostile-reviewer threads are unresolved (OMN-17492). -
[MINOR] hostile-reviewer (glm-review)
Per-group AttributeError swallowed, partial answer treated as complete | Inside the group loop, any AttributeError on group.group_id, group.state, or state.name causes the group to be silently skipped via continue. The function's own philosophy rejects partial answers for the listing level (listing.errors) but not at the group level: malformed entries are dropped and the result may report zero live groups, triggering a refusal that misattributes the cause to 'no live consumer group is bound' rather than 'a
Resolve this thread when addressed — the
Hostile Review Thread Gateblocks while hostile-reviewer threads are unresolved (OMN-17492). -
[MINOR] hostile-reviewer (glm-review)
Stacked timeouts outside the hard-timeout window can block ~16s | The liveness path performs AdminClient construction, list_topics(timeout=5), list_consumer_groups(request_timeout=5), and future.result(timeout=6) sequentially, all before the _hard_timeout window is entered (which is timeout + grace). Worst case, a refusing-by-hang broker delays the command by roughly 16 seconds with no alarm coverage, and the operator's --timeout no longer bounds end-to-end latency. | Evidence: admin.list_topics(timeout=tim
Resolve this thread when addressed — the
Hostile Review Thread Gateblocks while hostile-reviewer threads are unresolved (OMN-17492). -
[MINOR] hostile-reviewer (glm-review)
orchestrator_distribution conflates distribution name with importable module | importlib.util.find_spec(_ORCHESTRATOR_DISTRIBUTION) assumes a top-level importable module named identically to the distribution. If the omnimarket distribution ships an import package of a different name (common), find_spec returns None and every receipt reports '(unresolved)', undermining the field's stated purpose of exposing where the deciding code lives. Distribution name and import name are distinct concepts in importlib.me
Resolve this thread when addressed — the
Hostile Review Thread Gateblocks while hostile-reviewer threads are unresolved (OMN-17492).
Findings demoted from threads (anchor rejected)
-
[MAJOR] hostile-reviewer (glm-review)
live_consumer_groups, the load-bearing gate, has no tests | Every test in test_delegate_locus.py stubs live_consumer_groups. The actual broker-interrogation logic, metadata-first ordering, listing.errors handling, STABLE-state filtering, suffix matching, and the empty-listing-on-unreachable-broker hazard documented at length in the code comments, ships with zero coverage. The comment 'Observed live 2026-08-31 against a closed port' describes a manual observation standing in for a regression test. A refactor
Resolve this thread when addressed — the
Hostile Review Thread Gateblocks while hostile-reviewer threads are unresolved (OMN-17492).
|
| Surface | Meaning | Blocks merge? |
|---|---|---|
| Review threads | Per-finding, posted by the reviewer | No (informational) |
Hostile Review Thread Gate |
Deterministic: unresolved hostile-reviewer threads exist | Fails until resolved (not yet a required context) |
degraded verdict |
Fewer than 2 models succeeded (infra) | No |
Powered by omniintelligence.review_pairing.cli_review — multi-model adversarial review: qwen3-review, qwen3-review-b, glm-review (OMN-8468/OMN-8524/OMN-17492)
|
No OCC evidence companion was minted for this PR. this PR is already bound to OCC#8792; its evidence companion exists and nothing needs authoring To clear this: Nothing to do — the companion already exists. Reported by |
Closes OMN-17304 AC4 (caller side).
Evidence-Source: OCC#8792
Evidence-Ticket: OMN-17304
What this is
The AC4 client-mode caller.
omnibase_core#1633shippedRuntimeLocal(host_handlers=...)on 2026-09-01 andomnibase_infrahas had zero callers for it since —git grep -c host_handlers origin/dev -- srcreturned 0 against a positive control of 2RuntimeLocal(construction sites. Soonex delegatestill hosted the entry handler on every run, which is exactly the defect OMN-17304 opened on.This commit was written 2026-08-31 and sat unpushed on a local worktree until now. It is rebased onto current
devhere, not re-derived.What lands
--locus {auto,in-process,deployed-lane}. Locus becomes a resolved property rather than a side effect of--bus.autofollows the resolved transport; an explicit--locusannounces itself as an override.live_consumer_groups()inbackend_probe: tri-state, where_has_live_consumer_groupcollapses "nobody bound" and "could not find out" into False.handler_locus(read back from the runtime's ownworkflow_result.json),wire_correlation_id,orchestrator_distribution,dispatch_target.inmemorykeeps host-and-publish byte-for-byte; five of the 13 new tests are counter-assertions on the in-process path, so "always refuse" cannot pass them.Rebase resolution (3 conflicts, all against work that landed after this commit was written)
cli_delegate.pyDEFAULT_BUScomment21aaec90); addedLOCUS_CHOICESbeside itcli_delegate.py--bushelp--locusexists — the ticket predicted this wording would go stale. Restated:--busis transport,--locusowns locus; the configured-authority resolution AC3 landed is preserved verbatimreceipt_mode.pysignature,test_cli_delegate.pyimportsVerification
h105.2— 27382 tests green on1540b007969461c15f837227cc033e9c0b756e01(529s, suite log sha256194f6eb5…). Not narrowed, not skipped.uv run pytest tests/unit/cli/test_delegate_locus.py tests/unit/cli/test_cli_delegate.py -q→ 89 passed in 4.24s.ruff check+ruff formatclean;mypy --strictclean onsrc/omnibase_infra/{cli,backends,enums}.omnibase-core==0.47.5;RuntimeLocal.__init__exposeshost_handlers: bool = True, read live from the installed distribution.Lab readback — NOT obtained, and why (rule 24d, stated rather than borrowed)
AC4's locus proof (fresh correlation, non-zero
docker logs omninode-runtime{,-effects} --since 20m | grep <corr>, no committed-offset movement on the Mac CLI's group) needs a live delegation against the.201dev lane. It could not be taken in this pass.The dev-lane broker
omnibase-infra-redpandarestarted2026-09-09T01:48:44Zwith SASL enabled:Every unauthenticated Kafka client is now refused at connect — reproduced three ways:
rpk group listinside the container (broker closed the connection immediately after a request was issued … is SASL missing?), the same with-X brokers=localhost:9092and:19092, and aconfluent_kafkaAdminClientfrom this host against192.168.86.201:19092(_TRANSPORT … Broker transport failure). Containers arehealthyand the runtime is consuming normally, so this is an auth-surface change, not an outage. No SASL credentials are configured on the launching host.So the locus readback is deferred to a follow-up on the ticket, and this PR does not assert a lab proof it did not obtain. What it does assert is the unit-level proof above, which is what CI can adjudicate.
Blast radius
TestBusSelection/TestExplicitOverrideProvenancenow pin--locus in-process— their subject is transport plumbing, and inheriting the new default would drag a live-consumer precondition into an assertion about a dict.TestBusHelpTextTellsTheTruthasserts the pair (transport vs locus) plus the fail-closed promise, so the help text cannot lie in either direction._override_recordsmatches transport override lines only, since a second override axis now exists.