[jonahgabriel]

OMN-14645 — Harden the coverage-sweep-gate against cancel storms (merge-flow WS2)

Closes OMN-14645 (epic OMN-14643).

Deconfliction with #1776 (OMN-14641) — read first

This PR and omnimarket #1776 (OMN-14641, Codex's PR; OCC companion #4226 already merged) both touch scripts/ci/run_coverage_sweep_gate.py. #1776's single job is to stream the pytest --cov subprocess output to the CI job log instead of capturing it. This revision of #1777 inherits that streaming behavior directly and reconciles the file into ONE coherent hunk: streaming output + whole-process-group reaping on timeout, together.

Why this matters — the earlier PIPE-capture cut of this PR would have REGRESSED the storm. The prior revision converted subprocess.run → subprocess.Popen but re-added stdout=PIPE/stderr=PIPE (via communicate(timeout=...)), which buffers the entire ~8-minute coverage run until the child exits. That freezes the CI run's updatedAt for the whole longest-pole Coverage job — and a frozen updatedAt is exactly what trips the Codex merge controller's stale-run heuristic into cancel+rerun (the storm OMN-14645 exists to fix; the controller self-diagnosed this at the merge-controller ledger ~17:39Z). Streaming the output live is the durable root-cause fix for the stale-run heuristic: the run keeps heartbeating, so the controller never sees it as stale. Do not re-introduce PIPE/capture_output on this subprocess.

If #1776 lands first, its streaming change is already present on dev and this PR is a clean superset (streaming + reaping in one function); if this PR lands first, #1776 becomes a no-op on the same lines.

Problem (verified, not assumed)

Live omnimarket PRs #1754 / #1758 / #1774 were wedged mergeStateStatus=BLOCKED. The sole required context on dev is CI Summary. Verified from the per-attempt jobs API (not gh pr checks text):

• run 29421305846 (PR 1758) attempt 24: Coverage Sweep Gate = cancelled (started 17:29:50Z, cancelled 17:30:20Z = 30s), and the same attempt's CI Summary = failure.
• A 30s cancel is far below any timeout → not a job timeout. grep -n concurrency .github/workflows/ci.yml returned NONE → not a within-workflow concurrency cancel. Every setup step succeeds; only the long final step is cancelled.

Root cause (verified): the coverage-sweep-gate job is the longest pole (3 fresh sibling clones + uv sync --all-extras + full pytest --cov). When its output is captured/buffered, the run's updatedAt freezes for the whole job window; the Codex merge controller reads that frozen timestamp as a stale run and issues a same-head gh run rerun (cancel+rerun). Because coverage is the longest pole, a run-wide cancel almost always catches it mid-flight → cancelled. The ci-summary verdict loop treated a dependency as passing only if success OR skipped, so a cancelled coverage sweep → FAILED=true → CI Summary red → BLOCKED. The loop (freeze updatedAt → controller reruns → cancel long job → CI Summary fail → repeat) is self-perpetuating.

What this PR does (defense-in-depth, shadow-safe)

1. Stream the coverage subprocess output — do NOT capture/PIPE/buffer it (scripts/ci/run_coverage_sweep_gate.py). The pytest --cov child inherits this process's stdout/stderr and streams live to the CI job log, keeping the run heartbeating so the Codex stale-run heuristic never fires. This is the durable storm fix and deconflicts with fix(OMN-14641): stream coverage sweep subprocess output #1776 (OMN-14641). The subprocess is bounded with proc.wait(timeout=...) (streaming) rather than communicate(timeout=...) (which requires PIPE capture).
2. Whole-process-group reaping on generation timeout (retained, genuinely valuable). The child is launched with start_new_session=True; a timeout SIGTERM→SIGKILLs the entire process group (pytest-xdist workers + spawned subprocesses) via _reap_process_group, then fails loudly — it never leaves a stale artifact to be swept as clean. The streamed pytest output above the failure already shows the hang site, so no silent capture-and-hold is needed for diagnostics.
3. Single-flight per-ref concurrency, QUEUE not cancel (ci.yml: concurrency.cancel-in-progress: false, keyed by ${{ github.workflow }}-${{ github.ref }}). A newer same-ref run waits for the in-flight one to reach a terminal result instead of cancelling it. Different PRs / main / merge_group never block each other.
4. Explicit timeout-minutes: 40 on coverage-sweep-gate (was the 360-min GitHub default). Bounds the longest pole; a wedged child can no longer hold a runner slot for hours.
5. Fail-closed CI-Summary check for coverage-sweep-gate: moved out of the loose success||skipped loop into a strict success-only check (matching the OMN-14350 / OMN-14590 pattern). Closes a real fail-OPEN hole (a skipped/cancelled coverage sweep previously PASSED CI Summary). A cancelled sweep still fails, loudly.

Honest scope / what this PR does NOT do

• Deferred to follow-up (do not weaken the authoritative sweep in this PR):
  • Coverage artifact reuse — test shards emit versioned coverage artifacts keyed by head SHA/env/scope/schema; aggregate+validate once instead of a 2nd full pytest --cov, with fail-closed stale/wrong-head validation. Substantial redesign touching the test job's coverage upload and the sweep's authoritative path; scoped as a dedicated follow-up under OMN-14645/epic OMN-14643.
  • ONE global expensive-coverage slot + queue-wait SLO telemetry. A GitHub-Actions job-level concurrency global group is NOT safe as the slot: under >2 concurrent PRs GHA cancels the pending job, reintroducing the exact cancelled → CI Summary red failure this PR fixes. A real single-slot needs an external coordinator/lock (or the artifact-reuse redesign). Scoped as follow-up.

dod_evidence

• Before: generation used subprocess.Popen with stdout=PIPE/stderr=PIPE + communicate(timeout=...) (captures/buffers the whole run → frozen updatedAt → Codex stale-run cancel storm); coverage-sweep-gate had no timeout-minutes (360-min default); no workflow concurrency; CI Summary counted skipped/cancelled as pass for the coverage gate. Verified cancel→failure: run 29421305846 attempt 24 (above).
• After: generation uses subprocess.Popen(start_new_session=True) with no stdout/stderr/capture_output kwargs (streams live) + proc.wait(timeout=...) + _reap_process_group on timeout; workflow concurrency present (cancel-in-progress: false); coverage-sweep-gate.timeout-minutes = 40; CI Summary has a strict success-only coverage check.
• Focused tests: uv run pytest tests/ci/test_run_coverage_sweep_gate.py -q → 9 passed, including: test_generate_helper_invokes_pytest_cov now asserts no capture_output/stdout/stderr kwargs are passed to Popen (streaming, not captured), and test_generation_timeout_reaps_and_fails_loudly proves a timed-out generation reaps the process group and fails loudly (no stale-artifact pass).
• Gates: ruff format/ruff check clean; mypy scripts/ci/run_coverage_sweep_gate.py → no issues; pre-commit run --files ... all Passed (the topic-naming-lint hook requires OMNIBASE_INFRA_PATH to the sibling repo — an environmental prerequisite, passes once set).
• Rollback: revert this commit; it changes the subprocess to stream (from capture), retains reaping, and adds a concurrency block, one timeout-minutes, and a strict CI-Summary check — no behavior removed from the authoritative coverage requirement.
• Worktree: $OMNI_HOME/omni_worktrees/OMN-14645/omnimarket, rebased on fresh origin/dev (f6249696). Will be cleaned up after merge.

Merge policy

Codex owns the merge queue. This PR is opened/updated only — not merged, not auto-armed. omnimarket dev is squash-only, no queue, auto-merge disabled.

Evidence-Ticket: OMN-14645
Evidence-Source: OCC#4233
Evidence-Commit: 112d401f0aad0c33352f2c8216c5747c2b8cdc11
Evidence-Head: f2b3e8c

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 2 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 915e5878-a8ae-40d1-8326-405313115c5e

📥 Commits

Reviewing files that changed from the base of the PR and between 53091d0 and f2b3e8c.

📒 Files selected for processing (3)

• .github/workflows/ci.yml
• scripts/ci/run_coverage_sweep_gate.py
• tests/ci/test_run_coverage_sweep_gate.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch jonah/omn-14645-ws2-reuse-coverage-artifacts-and-serialize-the-expensive

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

✅ Architectural Review — PASSED

Errors: 0
Warnings: 0

What this checks

Rule	Description
ARCH-TOPIC-001	No hardcoded Kafka topic strings in handler code
ARCH-DI-001	No event_bus=None bypass
ARCH-DI-002	No direct Handler instantiation outside workflow_runner/adapters
ARCH-DI-003	No reinvented DI containers
ARCH-TOPIC-002	contract.yaml topics follow `onex.{cmd

---

No architectural violations found.

Static architectural lint — no model inference (OMN-14176).

[github-actions]

✅ Hostile Reviewer — PASSED

Blocking findings (critical/error): 0
Total findings: 0
Models succeeded: qwen3-review,qwen3-review-b

---

Gate semantics (pilot phase)

Verdict	Meaning	Blocks merge?
passed	No critical/error findings	No
blocked	CRITICAL or ERROR findings found	Yes
degraded	All models unavailable (infra)	No (pilot)

Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8524)