Skip to content

feat(OMN-17304): onex delegate resolves its execution locus — client mode caller, fail-closed dispatch gate - #3361

Merged
jonahgabriel merged 1 commit into
devfrom
jonah/omn-17304-delegate-lane-locus
Sep 9, 2026
Merged

jonahgabriel merged 1 commit into
devfrom
jonah/omn-17304-delegate-lane-locus

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Sep 9, 2026 •

Copy link
Copy Markdown
Collaborator

Closes OMN-17304 AC4 (caller side).

Evidence-Source: OCC#8792

Evidence-Ticket: OMN-17304

What this is

The AC4 client-mode caller. omnibase_core#1633 shipped RuntimeLocal(host_handlers=...) on 2026-09-01 and omnibase_infra has had zero callers for it since — git grep -c host_handlers origin/dev -- src returned 0 against a positive control of 2 RuntimeLocal( construction sites. So onex delegate still hosted the entry handler on every run, which is exactly the defect OMN-17304 opened on.

This commit was written 2026-08-31 and sat unpushed on a local worktree until now. It is rebased onto current dev here, not re-derived.

What lands

  1. --locus {auto,in-process,deployed-lane}. Locus becomes a resolved property rather than a side effect of --bus. auto follows the resolved transport; an explicit --locus announces itself as an override.
  2. A fail-closed dispatch gate: before publishing, the command topic — read from the contract, never named in this repo — must have a live STABLE consumer group bound to it. Broker answered with nothing bound, or broker unreachable, both refuse. It never silently falls back to in-process.
  3. live_consumer_groups() in backend_probe: tri-state, where _has_live_consumer_group collapses "nobody bound" and "could not find out" into False.
  4. Locus provenance in the receipt and capture log: handler_locus (read back from the runtime's own workflow_result.json), wire_correlation_id, orchestrator_distribution, dispatch_target.

inmemory keeps host-and-publish byte-for-byte; five of the 13 new tests are counter-assertions on the in-process path, so "always refuse" cannot pass them.

Rebase resolution (3 conflicts, all against work that landed after this commit was written)

file resolution
cli_delegate.py DEFAULT_BUS comment kept dev's AC3 tier-0 wording (#3123, 21aaec90); added LOCUS_CHOICES beside it
cli_delegate.py --bus help dev's post-AC3 text still asserted "it does not change where the work runs: the delegate orchestrator always executes in-process in this CLI". That sentence is false the moment --locus exists — the ticket predicted this wording would go stale. Restated: --bus is transport, --locus owns locus; the configured-authority resolution AC3 landed is preserved verbatim
receipt_mode.py signature, test_cli_delegate.py imports union; both sides additive

Verification

  • Governed pre-push: FULL unit suite satisfied by the lab run on h105.2 — 27382 tests green on 1540b007969461c15f837227cc033e9c0b756e01 (529s, suite log sha256 194f6eb5…). Not narrowed, not skipped.
  • uv run pytest tests/unit/cli/test_delegate_locus.py tests/unit/cli/test_cli_delegate.py -q → 89 passed in 4.24s.
  • ruff check + ruff format clean; mypy --strict clean on src/omnibase_infra/{cli,backends,enums}.
  • Precondition satisfied: this repo pins omnibase-core==0.47.5; RuntimeLocal.__init__ exposes host_handlers: bool = True, read live from the installed distribution.

Lab readback — NOT obtained, and why (rule 24d, stated rather than borrowed)

AC4's locus proof (fresh correlation, non-zero docker logs omninode-runtime{,-effects} --since 20m | grep <corr>, no committed-offset movement on the Mac CLI's group) needs a live delegation against the .201 dev lane. It could not be taken in this pass.

The dev-lane broker omnibase-infra-redpanda restarted 2026-09-09T01:48:44Z with SASL enabled:

enable_sasl: True, sasl_mechanisms: ['SCRAM'], superusers: ['onex-dev-lane']

Every unauthenticated Kafka client is now refused at connect — reproduced three ways: rpk group list inside the container (broker closed the connection immediately after a request was issued … is SASL missing?), the same with -X brokers=localhost:9092 and :19092, and a confluent_kafka AdminClient from this host against 192.168.86.201:19092 (_TRANSPORT … Broker transport failure). Containers are healthy and the runtime is consuming normally, so this is an auth-surface change, not an outage. No SASL credentials are configured on the launching host.

So the locus readback is deferred to a follow-up on the ticket, and this PR does not assert a lab proof it did not obtain. What it does assert is the unit-level proof above, which is what CI can adjudicate.

Blast radius

TestBusSelection / TestExplicitOverrideProvenance now pin --locus in-process — their subject is transport plumbing, and inheriting the new default would drag a live-consumer precondition into an assertion about a dict. TestBusHelpTextTellsTheTruth asserts the pair (transport vs locus) plus the fail-closed promise, so the help text cannot lie in either direction. _override_records matches transport override lines only, since a second override axis now exists.

…he lane is live, and refuses instead of degrading (OMN-17295)

Requires omnibase-core >= 0.47.2 (RuntimeLocal client mode, core PR for
OMN-17304). MUST NOT merge before that release and the pin bump.

THE DEFECT

Every "dev-lane probe" written as `onex delegate ... --bus kafka` was vacuous.
--bus selects the TRANSPORT; it never relocated execution. The orchestrator
resolved out of the caller's own venv and decided there, while the receipt was
read as a statement about .201.

Worse than "it ran locally": it ran TWICE. RuntimeLocal subscribed the entry
handler to the command topic it was publishing to, so Redpanda fanned every
command to both the lane's consumer group and the CLI's own disjoint one.
Measured on the dev lane 2026-08-31T20:06Z (correlation
993819e0-f3e5-4064-a1dd-0b85ee263105): the lane group advanced 151 -> 152 on
the real command while the Mac CLI group advanced 53 -> 54, executing a
98-message-old backlog command. The CLI then returned a terminal that was not
its own; post-#3064 the correlation join caught it and the run exited `status:
error` rather than printing a stranger's answer.

WHAT LANDS

1. `--locus {auto,in-process,deployed-lane}`. Locus is now a RESOLVED property,
   not a side effect of the transport flag. `auto` follows the transport,
   because that is the real relationship: a shared bus is the only kind another
   runtime can consume from. An explicit --locus announces itself as an
   override, so a receipt can tell "the authority chose this" from "a human
   typed it".

2. A fail-closed dispatch gate. Before publishing, the exact command topic —
   read from the CONTRACT, never named in this repo — must have a live STABLE
   consumer group bound to it. Three outcomes, two refuse:
     * live groups  -> proceed, and they go into the record
     * broker answered, nothing bound -> refuse ("start the runtime")
     * broker could not be asked      -> refuse ("fix the broker address")
   It never falls back to in-process. A degraded lane probe is worse than no
   lane probe, because it still prints a receipt.

3. `live_consumer_groups()` in backend_probe: the same wiring-truth signal as
   `_has_live_consumer_group`, but tri-state. The existing helper collapses
   "nobody is bound" and "I could not find out" into False, which is correct
   for a health GRADE and fatal for a dispatch gate. It also asks for cluster
   metadata FIRST: against a closed port librdkafka retries in the background
   and `list_consumer_groups` resolves to a clean EMPTY listing — observed live
   2026-08-31, and it reads exactly like "no consumers".

4. Locus provenance in the receipt and the capture log: `handler_locus`
   (read back from the runtime's OWN workflow_result.json, not from the
   request), `wire_correlation_id`, `orchestrator_distribution` (name, version
   AND location), and `dispatch_target` (topic, broker, live consumer groups).

A LIVENESS CHECK ON THE WRONG TOPIC

`resolve_default_bus` passes `authority_topic=SUFFIX_DELEGATION_REQUEST`
(`onex.cmd.omnibase-infra.delegation-request.v1`) and its docstring claims
liveness on "the exact topic this delegation is about to publish to". It is not:
this path publishes to the delegate contract's own
`onex.cmd.omnimarket.delegate-skill.v1`. Both carry STABLE groups on the dev
lane (verified live), so the OMN-16529 AUTHORITATIVE grade has been passing for
reasons unrelated to whether the command would be served. The new gate reads the
topic from the contract so the two cannot disagree.

RED-THEN-GREEN

`tests/unit/cli/test_delegate_locus.py` — 13 tests, RED before the change
(`ImportError: cannot import name 'ConsumerGroupLivenessUnknownError'`), green
after. Five are counter-assertions on the in-process path: an in-process run
must not acquire a broker dependency, must still host its handlers, and must
still resolve. Without them "always refuse" and "never dispatch" would satisfy
the fail-closed tests while deleting the offline flow.

TESTS RESTATED, NOT DELETED

`TestBusHelpTextTellsTheTruth` asserted the string "does not change where the
work runs" — true when written, false the moment a real remote mode exists. It
now asserts the pair (--bus is transport, --locus owns locus) plus the
fail-closed promise, so the help text cannot lie in either direction.
`TestBusSelection` / `TestExplicitOverrideProvenance` pin `--locus in-process`:
their subject is TRANSPORT plumbing, and inheriting the new default would drag
a live-consumer precondition into an assertion about a dict.
`_override_records` now matches transport override lines only — OMN-17304 added
a second override axis, and a counter-assertion that fails on an unrelated
locus line has stopped testing what it claims to test.

RECEIPT-LEVEL IDENTITY IS NOT THIS PR

OMN-17295 AC2 asks the receipt to name the orchestrator. On the dispatched path
it now does. On an in-process SUCCESS the receipt's `result` is the node's typed
handler result, so `ModelReceiptRuntimeSummary` is not used and the locus lives
in the capture log and workflow_result.json instead. The universal answer is
`ModelSkillResult.runtime_identity` — OMN-17308, In Progress in omnibase_core on
`model_skill_result.py`, whose `ModelRuntimeIdentity.execution_locus` names the
venv/container this process lives in. This PR deliberately does NOT add a
competing field, and names its own `handler_locus` for the disjoint question
"did I do the work".

LIVE PROOF

Post-fix, correlation f99d5485-1d18-476e-861b-0e81835752cf, `--locus
deployed-lane` against the .201 dev lane:
  * Mac CLI command-topic group: 54 -> 54 (UNCHANGED — never subscribed)
  * lane group:                 152 -> 153
  * `docker logs omninode-runtime{,-effects} --since 20m | grep <corr>`: 19 / 7
  * receipt: status success, handler_locus `dispatched`, wire_correlation_id ==
    this run's own correlation, dispatch_target naming the live lane group
  * the lane answered on the local rung (qwen3.8 @ 192.168.86.201:8000), 1
    attempt, quality 1.0, zero escalations, zero metered providers
Refusals, same build: an unreachable broker and `--locus deployed-lane --bus
inmemory` both exit 1 with distinct diagnoses and zero stdout.

REBASED 2026-09-09 onto origin/dev (255 commits). Three conflicts, all against
work that landed after this commit was written:
  * cli_delegate.py DEFAULT_BUS comment -- kept dev's AC3 tier-0 wording
    (#3123, 21aaec9) and added LOCUS_CHOICES beside it.
  * cli_delegate.py --bus help -- dev's post-AC3 text still asserted "it does
    not change where the work runs: the delegate orchestrator always executes
    in-process in this CLI". That sentence is false the moment --locus exists,
    which the ticket predicted. Restated: --bus is transport, --locus owns
    locus, and the configured-authority resolution AC3 landed is preserved
    verbatim.
  * receipt_mode.py run_receipt_mode signature and test_cli_delegate.py imports
    -- union, both additive.
Verified after rebase: ruff check + format clean, mypy --strict clean on
src/omnibase_infra/{cli,backends,enums} (the 52 remaining errors are all in a
sibling omnibase_core worktree checkout, none in this repo), and
tests/unit/cli/test_delegate_locus.py + test_cli_delegate.py -> 89 passed.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hostile Reviewer — adversarial findings (OMN-17492)

Models succeeded: glm-review
Models failed: codex
New finding threads: 1
Deduped (already posted on this PR): 0
Nit-level findings suppressed: 2

The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.

Findings not anchored to a changed file

  • [MAJOR] hostile-reviewer (glm-review)

    TOCTOU race between consumer-group liveness check and publish | The fail-closed gate verifies a STABLE consumer group exists, then publishes at some later point (after receipt setup, capture logging, and runtime construction). A group can rebalance, leave, or die between the check and the publish; the command then lands on an unattended topic, which is precisely the scenario the gate exists to prevent. The check is evidence of a past instant, not a guarantee about the publish moment, and nothing in the diff

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

  • [MINOR] hostile-reviewer (glm-review)

    AUTO resolves to DEPLOYED_LANE for existing --bus kafka callers | Any existing invocation that used --bus kafka for its documented original purpose (route this run's events through the shared broker while executing in-process) now silently becomes a remote dispatch, adds a multi-second broker interrogation, and refuses outright when KAFKA_BOOTSTRAP_SERVERS is unset or the topic check fails. This is a breaking change to the CLI's observable contract hidden behind a new default. The help text was updated, but

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

  • [MINOR] hostile-reviewer (glm-review)

    Per-group AttributeError swallowed, partial answer treated as complete | Inside the group loop, any AttributeError on group.group_id, group.state, or state.name causes the group to be silently skipped via continue. The function's own philosophy rejects partial answers for the listing level (listing.errors) but not at the group level: malformed entries are dropped and the result may report zero live groups, triggering a refusal that misattributes the cause to 'no live consumer group is bound' rather than 'a

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

  • [MINOR] hostile-reviewer (glm-review)

    Stacked timeouts outside the hard-timeout window can block ~16s | The liveness path performs AdminClient construction, list_topics(timeout=5), list_consumer_groups(request_timeout=5), and future.result(timeout=6) sequentially, all before the _hard_timeout window is entered (which is timeout + grace). Worst case, a refusing-by-hang broker delays the command by roughly 16 seconds with no alarm coverage, and the operator's --timeout no longer bounds end-to-end latency. | Evidence: admin.list_topics(timeout=tim

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

  • [MINOR] hostile-reviewer (glm-review)

    orchestrator_distribution conflates distribution name with importable module | importlib.util.find_spec(_ORCHESTRATOR_DISTRIBUTION) assumes a top-level importable module named identically to the distribution. If the omnimarket distribution ships an import package of a different name (common), find_spec returns None and every receipt reports '(unresolved)', undermining the field's stated purpose of exposing where the deciding code lives. Distribution name and import name are distinct concepts in importlib.me

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

Findings demoted from threads (anchor rejected)

  • [MAJOR] hostile-reviewer (glm-review)

    live_consumer_groups, the load-bearing gate, has no tests | Every test in test_delegate_locus.py stubs live_consumer_groups. The actual broker-interrogation logic, metadata-first ordering, listing.errors handling, STABLE-state filtering, suffix matching, and the empty-listing-on-unreachable-broker hazard documented at length in the code comments, ships with zero coverage. The comment 'Observed live 2026-08-31 against a closed port' describes a manual observation standing in for a regression test. A refactor

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

⚠️ Hostile Reviewer — DEGRADED (informational)

Critical findings: 0
Major findings: 2
Total findings: 8
Models succeeded: glm-review

Note: Fewer than 2 reviewer models succeeded. Degraded results are informational (OMN-8468/OMN-8524) and do not block merge. Error: cli_review exit 2 (fewer than 2 models succeeded — partial/total outage)


Semantics (OMN-17492 — the model finds, thread resolution gates)

Surface Meaning Blocks merge?
Review threads Per-finding, posted by the reviewer No (informational)
Hostile Review Thread Gate Deterministic: unresolved hostile-reviewer threads exist Fails until resolved (not yet a required context)
degraded verdict Fewer than 2 models succeeded (infra) No

Powered by omniintelligence.review_pairing.cli_review — multi-model adversarial review: qwen3-review, qwen3-review-b, glm-review (OMN-8468/OMN-8524/OMN-17492)

@jonahgabriel
jonahgabriel merged commit 1e51458 into dev Sep 9, 2026
306 of 320 checks passed
@jonahgabriel
jonahgabriel deleted the jonah/omn-17304-delegate-lane-locus branch September 9, 2026 15:16
@onexbot-occ-writer

Copy link
Copy Markdown
Contributor

No OCC evidence companion was minted for this PR.

this PR is already bound to OCC#8792; its evidence companion exists and nothing needs authoring

To clear this: Nothing to do — the companion already exists.

Reported by node_occ_companion_effect (decline code already_bound, OMN-16665). This decision was made against the PR's LIVE state at compute time, not at publish time — a green publisher job only means the command reached the broker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant