Skip to content

fix(OMN-18296): declare the delegation completion bound and make the client wait for it - #2510

Merged
jonahgabriel merged 3 commits into
devfrom
jonah/omn-18296-delegation-completion-bound
Sep 13, 2026
Merged

jonahgabriel merged 3 commits into
devfrom
jonah/omn-18296-delegation-completion-bound

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Closes OMN-18296 (Urgent, child of OMN-18168). Paired with OmniNode-ai/omnibase_infra#3479, which enforces the bound this PR declares.

The defect

Cloud delegation 16eafedc-199c-44c2-a2cb-b9e535839bb9 was submitted to the lab lane at 2026-09-13T09:55:04Z. At 09:57:40Z the omninode-runtime-effects pod was recreated with that delegation's inference command in flight. Read-only probe of the lab:

probe reading
gateway_workflows status='published', completed_at NULL, correlation a2fe0848-4b4b-462e-b633-c5f9559afee5
delegation_workflow_state state='ROUTED', in_flight=t, pending_emissions empty, last advanced 09:55:04.27Z
delegation-inference-request.v1 12 records, this correlation present
inference-response.v1 11 records, this correlation absent
effect consumer group offset 12 of 12, lag 0

The inference call was lost with the process and its offset was already committed, so the command was never redelivered and no response of either kind will ever arrive.

Nothing bounded the resulting wedge, and the two numbers that governed the run were unrelated to each other and neither was a contract: the runtime's give-up TTL was an environment-variable default in omnibase_infra (900s) and this client's patience was a hardcoded 300s CLI default. A caller therefore abandoned a delegation the platform was still willing to finish, and could not distinguish that from one the platform had silently stopped working on.

What changed

The contract declares the bound (node_delegation_orchestrator/contract.yaml). completion_bound carries max_wall_seconds, the restart policy, and the failure class and code the terminal must name. One declaration, read by the runtime that enforces it and by the client that waits for it, so the two cannot drift.

A typed failure class for the cause. EnumDelegationFailureClass.RUNTIME_RESTART_DURING_DELEGATION. Deliberately distinct from TIMEOUT, which is a provider that took too long to answer a call we can still see; here no call is outstanding at all.

The codec builds the terminal. StateIoCodec.build_abandoned_terminal turns an abandoned delegation_workflow_state row into a ModelDelegationFailed payload. omnibase_infra owns the bus, the envelope id and the topic; this method owns the only thing infra cannot know, which is this node's business shape.

The values are the honest ones. No content, so content is empty; no verdict, so quality_passed is false and the score is 0.0; no answer from any provider, so no tokens are claimed. latency_ms is the real elapsed wall time, because how long the customer waited is true and useful even when nothing came back. model_used reports "none" where routing never selected one rather than attributing a call that was never made. A row whose payload does not decode is returned as None and left alone rather than closed out on a guess.

terminal_failure_reason is composed from the contract's own vocabulary token into the shape the gateway's attribution grammar actually parses, so runtime_restart_during_delegation reaches gateway_workflows.terminal_failure_class as RuntimeRestartDuringDelegationError with its code, instead of arriving as an unexplained failure.

The client reads the same bound. --timeout now defaults to the contract's max_wall_seconds rather than 300. poll_until_terminal takes a deadline_source, and the typed TIMEOUT_EXCEEDED error distinguishes the two cases: a caller-chosen budget running out means wait longer, and the platform's own declared bound running out means the runtime owed a terminal and did not deliver one. The customer's next move differs, so the error must not merge them. A caller who passes --timeout still gets the previous wording unchanged.

read_declared_completion_bound refuses rather than falling back to a built-in number. A silent fallback is how this client came to be waiting 300 seconds for a 900 second platform in the first place.

dod_evidence

RED first. tests/unit/delegation/test_omn18296_completion_bound.py run against unmodified src/:

FAILED test_the_contract_declares_the_bound_the_runtime_enforces
FAILED test_the_codec_builds_a_typed_restart_terminal_from_an_abandoned_row
FAILED test_an_undecodable_row_is_left_alone_rather_than_closed_on_a_guess
FAILED test_the_client_waits_for_the_declared_bound_not_a_number_of_its_own
FAILED test_the_timeout_error_says_whose_bound_was_spent
  -> TypeError: poll_until_terminal() got an unexpected keyword argument 'deadline_source'
5 failed

GREEN.

tests/unit/delegation/test_omn18296_completion_bound.py
tests/unit/cloud/test_transport_cloud_delegation.py
tests/unit/cli/test_cli_cloud.py
  -> 66 passed

test_cli_cloud.py gains test_an_unset_timeout_waits_for_the_contract_declared_bound, which drives the real CLI end to end and asserts the poll deadline equals the contract's declared value and that the deadline names its source. Its fake transport gained the new keyword; no assertion in any existing test was changed.

The codec was additionally validated against the real abandoned row, read out of the lab's delegation_workflow_state and fed through build_abandoned_terminal unmodified. It produced a valid ModelDelegationFailed carrying task_type=summarization, the routed model, the real elapsed latency, and terminal_failure_reason that the gateway's grammar parses into RuntimeRestartDuringDelegationError / ONEX_MARKET_DELEGATION_RUNTIME_RESTART. The committed fixture is a synthetic row of the same shape, so no customer prompt text enters the repository.

ruff format, ruff check and mypy --strict clean on every file touched.

Honest gap. AC1's lab proof — delete the runtime pod mid-flight and read a terminal back within the bound — needs both this PR and the omnibase_infra one live on the lane, so it cannot be obtained from either PR alone. 16eafedc is still published and will not self-heal; closing it out requires the deployed runtime to carry the enforcement half.

Evidence-Ticket: OMN-18296
Evidence-Source: OCC#9332

@github-actions

Copy link
Copy Markdown

✅ Hostile Reviewer — PASSED

Blocking findings (critical/error): 0
Total findings: 0
Models succeeded: qwen3-review,qwen3-review-b


Gate semantics

Verdict Meaning Blocks merge?
passed >=2 models succeeded, no critical/error findings No
blocked CRITICAL or ERROR findings found Yes
degraded Fewer than 2 models succeeded (infra unavailable/timeout) Yes (OMN-15110)

Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8524)

onexbot-occ-writer Bot added a commit to OmniNode-ai/onex_change_control that referenced this pull request Sep 13, 2026
…nimarket#2510

The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited.

Ticket: OMN-16859 (AC3b)
@onexbot

onexbot Bot commented Sep 13, 2026

Copy link
Copy Markdown

No OCC evidence companion was minted for this PR.

this PR is already bound to OCC#9332; its evidence companion exists and nothing needs authoring

To clear this: Nothing to do — the companion already exists.

Reported by node_occ_companion_effect (decline code already_bound, OMN-16665). This decision was made against the PR's LIVE state at compute time, not at publish time — a green publisher job only means the command reached the broker.

@github-actions

Copy link
Copy Markdown

✅ Architectural Review — PASSED

Errors: 0
Warnings: 0

What this checks

Rule Description
ARCH-TOPIC-001 No hardcoded Kafka topic strings in handler code
ARCH-DI-001 No event_bus=None bypass
ARCH-DI-002 No direct Handler instantiation outside workflow_runner/adapters
ARCH-DI-003 No reinvented DI containers
ARCH-TOPIC-002 contract.yaml topics follow `onex.{cmd

No architectural violations found.

Static architectural lint — no model inference (OMN-14176).

@jonahgabriel
jonahgabriel enabled auto-merge (squash) September 13, 2026 11:09
jonahgabriel pushed a commit to OmniNode-ai/onex_change_control that referenced this pull request Sep 13, 2026
…2510

OCC companion by node_pr_lifecycle_fix_effect (OMN-13317 F1 / OMN-13990 / OMN-14285). Product PR head 3861abc6dc7a2c99ea9701e47818d6e2b66f51f1.
jonahgabriel pushed a commit to OmniNode-ai/onex_change_control that referenced this pull request Sep 13, 2026
…nimarket#2510

The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited.

Ticket: OMN-16859 (AC3b)
jonahgabriel pushed a commit to OmniNode-ai/onex_change_control that referenced this pull request Sep 13, 2026
…imarket#2510 (#9332)

* evidence(OMN-18296): author OCC companion for OmniNode-ai/omnimarket#2510

OCC companion by node_pr_lifecycle_fix_effect (OMN-13317 F1 / OMN-13990 / OMN-14285). Product PR head 3861abc6dc7a2c99ea9701e47818d6e2b66f51f1.

* evidence(OMN-18296): self-bind OCC#9332 + rebind contract_sha256

* evidence(OMN-16859): executed test_passes receipts for OmniNode-ai/omnimarket#2510

The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited.

Ticket: OMN-16859 (AC3b)

---------

Co-authored-by: omnimarket-bot <bot@omninode.ai>
Co-authored-by: onexbot-occ-writer[bot] <onexbot-occ-writer[bot]@users.noreply.github.com>
@jonahgabriel
jonahgabriel merged commit 76ce3b5 into dev Sep 13, 2026
129 checks passed
@jonahgabriel
jonahgabriel deleted the jonah/omn-18296-delegation-completion-bound branch September 13, 2026 13:06
Patel230 pushed a commit to OmniNode-ai/onex_change_control that referenced this pull request Sep 16, 2026
…imarket#2510 (#9332)

* evidence(OMN-18296): author OCC companion for OmniNode-ai/omnimarket#2510

OCC companion by node_pr_lifecycle_fix_effect (OMN-13317 F1 / OMN-13990 / OMN-14285). Product PR head 3861abc6dc7a2c99ea9701e47818d6e2b66f51f1.

* evidence(OMN-18296): self-bind OCC#9332 + rebind contract_sha256

* evidence(OMN-16859): executed test_passes receipts for OmniNode-ai/omnimarket#2510

The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited.

Ticket: OMN-16859 (AC3b)

---------

Co-authored-by: omnimarket-bot <bot@omninode.ai>
Co-authored-by: onexbot-occ-writer[bot] <onexbot-occ-writer[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant