fix(pipeline): admit the gap-filler serial to Decode under memory-high

[nh13]

Draft, stacked on #764 (base = 746/nhomer/bound-consensus-memory-under-writer-stall). It fixes a deadlock that #764's tests still hit — see below. Rebase onto #764 when that lands.

The problem: #764's deadlock is not fixed, and green CI is masking it

#764's three test_pipeline_memory_backpressure tests pass in CI but reliably hang on a loaded host. The hang is a real, scheduling-sensitive deadlock in the Decode step, not flaky tests:

• The Decode step gates new work with a blanket q3_reorder_state.is_memory_high() check before popping Q2b (bam.rs, try_step_decode). It was copied from the Decompress step, whose comment argues the guard is deadlock-safe because "Q1 is FIFO, so next_seq for the reorder buffer has already been produced."
• That invariant does not hold for Decode. When a slow writer backs the pipeline up and the Q3 reorder buffer reaches its high-water mark, the guard also stops Decode from decoding the very serial the reorder buffer is waiting on. That serial never reaches Q3, Group starves on it, and the pipeline wedges with the reader gated — exactly the OOM-avoidance path fix(pipeline): enforce the queue memory budget at the Read step #764 adds.

Evidence (controlled A/B on the loaded host, same commit, only this guard toggled):

Decode Q3 is_memory_high guard	4 backpressure tests
ON (as in #764)	4/4 TIMEOUT (hang)
OFF	4/4 pass

CI's lighter scheduling doesn't trip the race, so #764 goes green — a false negative for this deadlock class.

The fix

Replace the blanket guard with the reorder buffer's own per-serial can_proceed, applied after the pop:

• It always admits the gap-filler (serial == next_seq), even over the memory limit, so the serial Group is waiting on can never be starved.
• It backpressures only future serials, once the buffer is half full.
• A future batch rejected by can_proceed is returned to Q2b so another worker can still reach the gap-filler — avoiding the "all workers hold a non-next_seq batch and nobody can produce next_seq" deadlock the code hit when can_proceed was previously wired into Decompress (see that step's comment).

On the healthy path (memory below the high-water mark) can_proceed is always true, so throughput is unchanged — the new backpressure engages only under the exact memory-high condition that used to deadlock.

Validation

• The four backpressure tests pass on the loaded reproducer across 3 repeated runs (they reliably hang without this change).
• Full workspace suite: 6675/6675 pass. cargo ci-fmt and cargo ci-lint clean.

Caveat worth raising separately

CI cannot currently catch this deadlock class (it needs constrained scheduling to reproduce). A follow-up that reproduces it under load in CI would keep it from regressing silently.

Risk: output for grouping, consensus, sort order, corrected UMIs, and metrics: none; unsafe: none, and no CLAUDE.md allowlist update; memory bound, queue capacity, and thread/backpressure policy: changed for Decode under memory-high conditions.

Fix: Decode applies per-serial admission after popping Q2b. It requeues future serials and admits the required Q3 gap-filler serial. This prevents stalls while preserving serial order.

Validation:

• Four backpressure tests pass across three loaded-reproducer runs.
• Full workspace suite passes: 6675/6675 tests.
• Formatting and lint checks pass.

[coderabbitai]

Note

Reviews paused

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 47188a32-371f-49ce-a87b-16de45cb6b20

📥 Commits

Reviewing files that changed from the base of the PR and between e69b17a and c95e22a.

📒 Files selected for processing (2)

• src/lib/unified_pipeline/bam.rs
• tests/integration/test_pipeline_memory_backpressure.rs

Included review availability: 0 reviews are currently available. Based on recent review activity, included reviews refill at 1 per hour.

---

Walkthrough

Decode now checks physical Q3 capacity before popping Q2b. It then applies per-serial admission, requeues future serials, and directly decodes when requeueing fails. Integration tests add read-batch control and validate budget-scaled backpressure.

Changes

Decode admission flow

Layer / File(s)	Summary
Physical capacity and serial admission src/lib/unified_pipeline/bam.rs	Decode separates physical capacity checks from serial admission. Future serials return to Q2b under pressure. Required gap-filling serials continue, including bounded direct decoding when requeueing fails. Unit tests cover admission, requeueing, fallback, and charge refunding.
Read-ahead budget validation tests/integration/test_pipeline_memory_backpressure.rs	The stalled-pipeline helper accepts an optional read-batch size. Tests use fine-grained reads for 1 MiB versus 3 MiB comparisons and retain default behavior in other scenarios.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to c95e2

The change allows the Decode step to admit the required gap-filler while continuing to backpressure future work under high memory. No actionable merge-blocking risk remains beyond normal checks and review.

Possibly related issues

• test: harden pipeline-backpressure FAMILIES fixture read-ahead margin against BGZF block-size drift #789 — Both changes update read-ahead memory-budget tests and scaling assertions.

Possibly related PRs

• fulcrumgenomics/fgumi#657 — Both modify decode state-machine logic in src/lib/unified_pipeline/bam.rs.

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title uses valid Conventional Commit syntax, describes the main Decode scheduling fix, and has a lowercase imperative description without a period.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[nh13]

@coderabbitai pause

[coderabbitai]

✅ Action performed

Reviews paused.

[codecov]

Codecov Report

❌ Patch coverage is 96.36364% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.36%. Comparing base (f0d6c33) to head (c95e22a).

Files with missing lines	Patch %	Lines
src/lib/unified_pipeline/bam.rs	96.36%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #787   +/-   ##
=======================================
  Coverage   94.36%   94.36%           
=======================================
  Files         186      186           
  Lines      113659   113713   +54     
=======================================
+ Hits       107251   107304   +53     
- Misses       6408     6409    +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[nh13]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[nh13]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.