Skip to content

fix(OMN-19194): raise shadow-slice timeout budget and route exit 124 to infra - #2783

Merged
jonahgabriel merged 2 commits into
devfrom
jonah/omn-19194-shadow-slice-timeout-budget
Sep 22, 2026
Merged

jonahgabriel merged 2 commits into
devfrom
jonah/omn-19194-shadow-slice-timeout-budget

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

OMN-19194

market-drain measured (2026-09-22, OMN-17427 integration pass): the shadow
tests+coverage job's scoped slice (-m "unit and not kafka") takes 621-663s
of pytest time plus ~90s of collection -- 711-753s total on a normal runner.
The old --timeout 900 in run_shadow_slice.py left only 150-190s of
margin, and a ~25% slower runner (observed on PR #2777 at heads 544aa5a
and 96c7c3e, and PR #2779 at af89283, plus merged PR #2773) pushed the
same correctly-scoped slice past the wall: exit 124. This step fell through
to the generic failure branch on that code, so reason-graph and
product-readiness / evaluate reported a false PRODUCT_FAILED root for a
runner-speed timeout, not a real product defect. Both contexts are
non-required, so nothing was blocked, but the report was wrong.

Two-part fix, mirroring the OMN-18820 exit-125 precedent already in this
file:

  • raise the budget to 1200s -- 300s+ (27%+) margin over the worst-observed
    timed-out case;
  • add an exit-124 branch reporting product_conclusion=timeout and
    succeeding the step, so a genuine remaining hang still routes to the
    infra dimension instead of a false PRODUCT_FAILED root.

This is a sibling fix under the same ticket as the onex_change_control
shadow-timeout fix (onex_change_control#10828, already merged) -- that repo's
shadow job was duplicating its full, unscoped test suite and needed
narrowing; this repo's shadow job was already correctly scoped and just
needed a realistic budget plus the same infra-classification the 125 case
already gets.

Evidence-Ticket: OMN-19194
Evidence-Source: OCC#10850

…to infra

market-drain measured (2026-09-22, OMN-17427): the scoped shadow slice
(-m "unit and not kafka") takes 621-663s of pytest time plus ~90s of
collection, 711-753s total. The old --timeout 900 left only 150-190s of
margin, and a ~25% slower runner (observed on PR #2777/#2779) pushed the
same correctly-scoped slice past the wall -- exit 124 -- which this step
fell through to the generic "failure" branch on, misreporting reason-graph
and product-readiness/evaluate as PRODUCT_FAILED for a hang/timeout, not a
real product defect.

Two-part fix, matching the OMN-18820 exit-125 precedent:
- raise the budget to 1200s, giving 300s+ (27%+) margin over the
  worst-observed timed-out case -- this is runner-speed variance on an
  already-narrow slice, unlike the sibling onex_change_control finding
  under this same ticket, where the slice itself needed narrowing;
- mirror the exit-125 branch for exit 124: report product_conclusion=timeout
  and succeed the step, so a genuine remaining hang still routes to the
  infra dimension instead of a false PRODUCT_FAILED root.
@github-actions

Copy link
Copy Markdown

✅ Architectural Review — PASSED

Errors: 0
Warnings: 0

What this checks

Rule Description
ARCH-TOPIC-001 No hardcoded Kafka topic strings in handler code
ARCH-DI-001 No event_bus=None bypass
ARCH-DI-002 No direct Handler instantiation outside workflow_runner/adapters
ARCH-DI-003 No reinvented DI containers
ARCH-TOPIC-002 contract.yaml topics follow `onex.{cmd

No architectural violations found.

Static architectural lint — no model inference (OMN-14176).

@onexbot-occ-writer

Copy link
Copy Markdown
Contributor

OCC autobind did not mint a companion for this PR: no changed-file candidate could be proven RED against the merge base, and emitting a PR-existence probe instead would be non-falsifiable evidence (OMN-15247). Hand-authored evidence is required.

@jonahgabriel
jonahgabriel enabled auto-merge (squash) September 22, 2026 21:46
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

✅ Hostile Reviewer — PASSED

Blocking findings (critical/error): 0
Total findings: 9
Models succeeded: qwen3-review,qwen3-review-b


Gate semantics

Verdict Meaning Blocks merge?
passed >=2 models succeeded, no critical/error findings No
blocked CRITICAL or ERROR findings found Yes
degraded Fewer than 2 models succeeded (infra unavailable/timeout) Yes (OMN-15110)

Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8524)

The exit-124 -> product_conclusion=timeout branch this ticket wires into
product-readiness-shadow.yml relies on product_readiness.py's fail-closed
default (an unrecognized conclusion falls through to INFRA) rather than an
explicit vocabulary entry. OMN-18820 declared runner_crashed explicitly for
exactly this reason: its own test (now mirrored here) proves the entry is
load-bearing documentation, not a functional no-op, because the fail-closed
default could be tightened later with nothing left to catch it.

Adds "timeout" to _INFRA_CONCLUSIONS with the same measured justification
already in the workflow comment, plus a focused test file mirroring
test_omn18820_shadow_runner_crash_class.py's TestConclusionVocabulary and
TestReasonGraphRoot classes, scoped to this addition only (the harness's own
EXIT_HANG=124 classification is unchanged and already covered by
test_run_shadow_slice.py).
onexbot-occ-writer Bot added a commit to OmniNode-ai/onex_change_control that referenced this pull request Sep 22, 2026
…nimarket#2783

The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited.

Ticket: OMN-16859 (AC3b)
Onex-Workflow: OCC Receipt Runner
Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/35789239764
jonahgabriel added a commit to OmniNode-ai/onex_change_control that referenced this pull request Sep 22, 2026
…0853)

* evidence(OMN-19194): OCC evidence for the omnimarket#2783 sibling shadow-timeout fix

Hand-authored per OMN-15247 (the omnimarket PR carries no RED-derivable file of
its own -- a workflow timeout budget plus a classifier vocabulary entry, not
product behavior the merge base can fail against). Appends one dod_evidence
entry to the existing OMN-19194 contract (this ticket's onex_change_control
half already merged via #10828); does not touch the five pre-existing entries.

The receipt cites the same measurement independently reproduced from the
Actions API: 10/30 recent tests+coverage (shadow) runs on omnimarket's
product-readiness-shadow.yml failed, all 10 confirmed exit-124 HARD TIMEOUT via
job log, zero real test failures.

Onex-Lane: market-shadow-timeout
Onex-Session: 398d2c3490d14f559a52d04870bb8884

* evidence(OMN-18921): mint the owed OCC self-bind for PR #10853

---------

Co-authored-by: onexbot-occ-writer[bot] <onexbot-occ-writer[bot]@users.noreply.github.com>
jonahgabriel added a commit to OmniNode-ai/onex_change_control that referenced this pull request Sep 22, 2026
…imarket#2783 (#10850)

* evidence(OMN-19194): author OCC companion for OmniNode-ai/omnimarket#2783

OCC companion by node_pr_lifecycle_fix_effect (OMN-13317 F1 / OMN-13990 / OMN-14285). Product PR head 0a5aefc87ddac19d4100b0de97e5d3b504d8952d.

* evidence(OMN-19194): self-bind OCC#10850 + rebind contract_sha256

* evidence(OMN-16859): executed test_passes receipts for OmniNode-ai/omnimarket#2783

The declared checks were executed for real in the OmniNode-ai/omnimarket checkout at the PR head by the product-repo OCC receipt runner, and the results written append-only into this companion. Producer-minted PENDING receipts are superseded by a net-new record; no existing receipt was edited.

Ticket: OMN-16859 (AC3b)
Onex-Workflow: OCC Receipt Runner
Onex-Run: https://github.com/OmniNode-ai/omnimarket/actions/runs/35789239764

* evidence(OMN-19194): add contract_entry_sha256 to the contract_sha256-rebind supersession

Onex-Lane: market-shadow-timeout
Onex-Session: 398d2c3490d14f559a52d04870bb8884

---------

Co-authored-by: omnimarket-bot <bot@omninode.ai>
Co-authored-by: onexbot-occ-writer[bot] <307849072+onexbot-occ-writer[bot]@users.noreply.github.com>
Co-authored-by: jonahgabriel <jonah@omninode.ai>
@jonahgabriel
jonahgabriel merged commit 7c97ef6 into dev Sep 22, 2026
246 of 272 checks passed
@jonahgabriel
jonahgabriel deleted the jonah/omn-19194-shadow-slice-timeout-budget branch September 22, 2026 23:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant