Summary
test_pipeline_memory_backpressure::read_ahead_under_a_stalled_writer_shrinks_with_the_queue_memory_budget (in tests/integration/test_pipeline_memory_backpressure.rs) is intermittently failing. It failed 1 of 3 local reps and has failed in CI, independent of any particular change (it reproduces on main).
Symptom
read-ahead was 3169724 bytes under a 2097152-byte budget and 4578134 bytes under a 4194304-byte one; the gap is smaller than one tight budget, so the budget is not bounding the queues
The tight-budget arm is stable (~3169724 bytes across runs); the generous-budget arm varies with scheduler timing (observed 4578134 and 4929843 bytes). When the generous arm reads less far ahead, the tight-vs-generous gap dips below the assertion threshold:
assert!(
generous.saturating_sub(tight) >= TEST_BUDGET_BYTES, // TEST_BUDGET_BYTES = 2 MiB
...
);Why it's flaky
The assertion requires the read-ahead gap between the two budgets to span at least one tight budget (2 MiB). How far the reader runs ahead of a stalled writer depends on scheduler timing and block boundaries, so the generous arm's read-ahead is not deterministic. On a slow/contended run it can land close enough to the tight arm that the gap falls under 2 MiB even though the budget IS bounding the queues.
Suggested fix
Stabilize the discriminating assertion so it tolerates scheduler jitter without weakening what it proves, e.g.:
- average each arm's read-ahead over a few reps before comparing, or
- tie the required gap margin to the observed rep-to-rep variance rather than a fixed 2 MiB, or
- assert a proportional relationship (generous meaningfully exceeds tight) with a margin sized from measured spread.
Notes
Surfaced while rebasing #772 onto main; the PR does not touch this test or the BAM pipeline code it exercises.
Summary
test_pipeline_memory_backpressure::read_ahead_under_a_stalled_writer_shrinks_with_the_queue_memory_budget(intests/integration/test_pipeline_memory_backpressure.rs) is intermittently failing. It failed 1 of 3 local reps and has failed in CI, independent of any particular change (it reproduces onmain).Symptom
The tight-budget arm is stable (~3169724 bytes across runs); the generous-budget arm varies with scheduler timing (observed 4578134 and 4929843 bytes). When the generous arm reads less far ahead, the tight-vs-generous gap dips below the assertion threshold:
Why it's flaky
The assertion requires the read-ahead gap between the two budgets to span at least one tight budget (2 MiB). How far the reader runs ahead of a stalled writer depends on scheduler timing and block boundaries, so the generous arm's read-ahead is not deterministic. On a slow/contended run it can land close enough to the tight arm that the gap falls under 2 MiB even though the budget IS bounding the queues.
Suggested fix
Stabilize the discriminating assertion so it tolerates scheduler jitter without weakening what it proves, e.g.:
Notes
Surfaced while rebasing #772 onto
main; the PR does not touch this test or the BAM pipeline code it exercises.