Repository navigation
fix(OMN-14600): inline-recover in-flight-lock defer strand + re-pin core #1439 - #2309
Merged
Merged
Conversation
added 2 commits
July 13, 2026 20:58
… dead carrier arm Follow-up to OMN-14493/#2296, not a reopen. #2296 fixed the original defect (0 rows EVER reaching terminal). This closes the residual it left behind: handler_wiring.py's in-flight-lock `defer` branch (a different input arriving while a prior leg's outbox batch is committed but un-finalized) reported (1, None) -- a SUCCESS -- with the handler never run. Under a retry-storm (or simply a winner leg that crashes/stalls and never finalizes) this drops the deferred leg's own input FOREVER: no retry, no error, no DLQ, no redelivery, and the boot-time-only recovery sweep never got a second chance to clear the row within a live process. Fable-gate correction (supersedes the first fix attempt): the original design reported the defer as a conflict and relied on retry_on_optimistic_conflict exhausting and raising OptimisticConflictError to trigger a Kafka redelivery. That does NOT happen on this runtime -- the state_io stateful callback is registered as a dispatcher on MessageDispatchEngine, whose per-dispatcher invocation loop catches every exception a dispatcher raises and converts it to a returned HANDLER_ERROR status instead of re-raising. The corrected fix INLINE-RECOVERS the stalled winner's own batch (publish + finalize, from inside the deferring leg) instead of depending on redelivery. Also: the periodic recovery sweep (_ensure_stale_rows_recovered) is now interval-gated (time.monotonic(), _STATE_IO_RECOVERY_SWEEP_INTERVAL_SECONDS) instead of a one-shot boolean, so a row that goes stranded mid-process can self-heal on later traffic instead of only at the next boot/redeploy; _recover_outbox_batches isolates each row in its own try/except so one unresolvable poison row can't choke the sweep for every other row; and _legacy_delegation_envelope_unwrap heals the pre-fix ModelDelegationEventEnvelope rows still stuck in the outbox from before the OMN-14600 emit-shape fix landed (the applier's _publish_payload_for_output_event special-case tuple keeps this legacy class name for the same reason). Dropped the embedded-topic carrier-detection arm from _build_outbox_entries (and its allowed_output_topics plumbing) -- dead per the canonical two-class split executed in the companion omnimarket PR: ModelDelegationResult (and its two new subclasses) have no topic field, so there is nothing to detect. Depends on the paired omnibase_core PR (adds ModelDelegationCompleted/ ModelDelegationFailed) for the new seam tests to import; this PR carries no pyproject.toml dependency change against dev. Live-readback proof (post 3-repo fix, .201 stability-test, image ac5a9b1a0b9b): new delegation cid 86b5dcd7 reached COMPLETED/in_flight=false, terminal published, projection landed; both pre-fix stranded rows healed via the legacy-row sweep; consumer offsets +3/+3. Verify correction (non-blocking, this commit): the periodic-sweep test's single post-stall shared_cb dispatch was ALSO that closure's first-ever call, making it indistinguishable from the OLD one-shot-boolean gate (which would pass the same assertion). Added a warm-up dispatch on a third correlation before CID ever stalls, so the real probe becomes shared_cb's SECOND call -- the only dispatch that actually discriminates interval-gated re-arming from a one-shot boolean.
…ide a9f25887 Bump the omnibase-core pins (project deps + [tool.uv] override) from ==0.46.7 to ==0.46.8 and add a TEMPORARY [tool.uv.sources] git-rev override to a9f25887 -- the MERGED squash commit of core #1439 on dev (self-reports 0.46.8) -- so the terminal delegation classes (ModelDelegationCompleted/ ModelDelegationFailed) resolve before that commit is published to PyPI. uv.lock regenerated (scoped to the core source line only). Proven live on .201 stability-test (proof_class=live-readback): a fresh synthetic delegation terminalized cleanly (state=FAILED, in_flight=false, pending_emissions=NULL, terminal published to delegation-failed.v1, delegation_events projection landed). The infra defer-strand recovery (handler_wiring.py, this branch) + market emit change were verified in-image. REMOVE WHEN RELEASED: once core publishes a version containing #1439, delete the override and `uv lock`.
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 45 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (17)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OMN-14600 — state_io defer-strand recovery (infra half)
Makes the state_io outbox recovery sweep periodic (re-runs opportunistically
on the next dispatch every 30s instead of once per adapter lifetime) and
inline-recovers the in-flight-lock defer strand (a conflict-retry exhaustion
raised inside the state_io dispatcher is absorbed by the engine catch-all
before the auto-wired boundary sees it, so redelivery is dead on this runtime;
the branch self-heals inline instead). Also unwraps the legacy pre-fix
delegation carrier entries. See
handler_wiring.py.Re-pins
omnibase-core==0.46.7→==0.46.8and adds a TEMPORARY[tool.uv.sources]git-rev override toa9f25887(the MERGED core #1439squash commit on dev, self-reports 0.46.8) so the terminal delegation classes
resolve before that commit is on PyPI. REMOVE WHEN RELEASED.
dod_evidence — proof_class=live-readback (.201 stability-test)
Workspace image (dev-tip core
a9f25887+ this branch + the market 14600branch) built and the 4 core seam services recreated on .201 stability-test.
In-image verify:
_STATE_IO_RECOVERY_SWEEP_INTERVAL_SECONDS(4 refs) +_legacy_delegation_envelope_unwrap(5 refs) present at/app/src/omnibase_infra/runtime/auto_wiring/handler_wiring.py; core terminalclasses importable. A fresh synthetic delegation (cid
33724987-4586-421d-a6b0-917d1bf6013b) terminalized cleanly:delegation_workflow_state:state=FAILED, in_flight=false, pending_emissions=NULL, version=34— no strandonex.evt.omnibase-infra.delegation-failed.v1(HW 412→413)omnidash_analytics.delegation_events(1 row)Terminal was FAILED (not COMPLETED) due to the lane's z.ai LLM provider
returning HTTP 429
1310 Weekly/Monthly Limit Exhausted(external, resets2026-07-16), not this fix. Runtime booted clean on the fresh image (rc=0, no
ONEX_CORE_064).
Scope note
This branch carries the OMN-14600
handler_wiring.pydefer-strand recovery.It does NOT include the OMN-14598 reducer
.projectionsadapter work (ownedby a separate lane) — no collision.
Depends on
Evidence-Source: OCC#4153
Evidence-Ticket: OMN-14600
Evidence-Commit: ef9656e6dfda42b9084fddd98dcc3d785ff3190e
Merged-path 2nd-consumer companion (OMN-14623 bridge) — contracts/OMN-14600.yaml already
merged via OCC#4134 (core#1439); OCC#4153 refreshes the infra#2309 head binding after the receipt-honesty workflow install repair; earlier downstream receipts are in OCC#4147/OCC#4149/OCC#4151. Codex owns merge.