[nh13]

Summary

The typed-step deadlock monitor only logged a stall warning each window and never failed the run, so a genuine wedge (no worker can make progress) hung forever until killed externally — a regression from the prior behaviour, where the watchdog records a TimedOut error and tears the pipeline down. (The reorder-stash OOM fix also removed the one accidental backstop — unbounded growth → OOM-kill — so a wedge now hangs cleanly with flat, bounded memory.)

Fix

Restore the fail-fast guarantee with a starvation guard. On a stall, sum the bytes in flight across the byte-bounded transport queues and their reorder overflow stashes (the stash is essential — items can sit in a reorder buffer waiting for a missing serial while every transport reads empty), then classify:

• nothing in flight → idle, waiting on a slow upstream (e.g. a stdin pipe); not a deadlock — reset the clock and keep watching;
• work stuck past the warn window → warn with a stats snapshot, once per window;
• work stuck past the fatal window → record PipelineError::TimedOut and cancel(), so workers observe is_done() and the run fails fast.

Escalate to fatal only after the stall persists 6× the warn window (~60s by default at --deadlock-timeout 10): a single huge dispatch (busy-locus group, large sort merge) can legitimately flatten progress for many seconds with work queued, and a real deadlock hangs forever, so waiting longer to be sure is free whereas a false kill on a slow workload would be a regression. --deadlock-timeout 0 disables the monitor entirely.

classify_stall and the in-flight probe in_flight_bytes are extracted as pure functions and unit-tested across the progressing / starving / watching / stalled / wedged cases.

Known limitation (documented in code)

The in-flight probe covers byte-bounded transport edges only; CountBounded/Unbounded transports are not registered as queue handles, so they don't contribute. The only current CountBounded user is the sort spill→merge path, whose SortMerge is deadlock-free by construction (the cooperative slot-refill that fixed #330 BUG #3), so a wedge there is not reachable and the monitor correctly stays quiet during a slow merge. Extending the probe to count-based queues for defence-in-depth is filed as #408.

Stack context

Follow-up fix for the issue #330 unified-pipeline landing onto feat-runall (one commit per PR). Dual-reviewed before push (CLI: no findings; skill: traced the teardown/error-propagation, false-positive guard, and classify_stall boundaries — sound; surfaced the byte-bounded-only probe scope, now documented + filed as #408).

[codecov]

Codecov Report

❌ Patch coverage is 88.93443% with 27 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (feat-runall@a7f9a95). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/lib/pipeline/core/builder.rs	89.45%	25 Missing ⚠️
src/lib/pipeline/core/runtime/fused.rs	0.00%	2 Missing ⚠️

❌ Your patch check has failed because the patch coverage (88.93%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@              Coverage Diff               @@
##             feat-runall     #407   +/-   ##
==============================================
  Coverage               ?   93.27%           
==============================================
  Files                  ?      138           
  Lines                  ?    54910           
  Branches               ?        0           
==============================================
  Hits                   ?    51220           
  Misses                 ?     3690           
  Partials               ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.