Skip to content

docs(pipeline): pin the deliberate Step2 reorder-stage asymmetry - #441

Merged
nh13 merged 1 commit into
feat-runallfrom
nh/step2-reorder-asymmetry
Jun 20, 2026
Merged

nh13 merged 1 commit into
feat-runallfrom
nh/step2-reorder-asymmetry

Conversation

@nh13

@nh13 nh13 commented Jun 19, 2026 •

Copy link
Copy Markdown
Member

Summary

Resolves a review question on TypedStep2::build_output_set (raised against the fgumi-pipeline-core extraction, #440): why doesn't it apply the Serial/Exclusive reorder-collapse that the single-input TypedStep::build_output_set has?

Because applying it would be a correctness bug. This PR documents the deliberate asymmetry and adds a test that pins it.

Why the asymmetry is intentional

The single-input collapse rests on a universal property: a Serial single-input step consumes one already-ordered stream, so any input ordinal it propagates onto an output is emitted in push order — the reorder stage is redundant, and collapsing ByItemOrdinal → None is a free optimization.

A two-input merge has no such guarantee. It interleaves two branches, so a ByItemOrdinal output that propagates an input's ordinal can be pushed out of ordinal order even under serial (mutex-serialized, single-dispatcher) execution — e.g. when branch B's next-needed ordinal hasn't arrived but branch A's later one has. For such a step the reorder stage is load-bearing; dropping it would silently deliver records out of order.

Today's production Step2s happen not to need it — PairRawFastq declares BranchOrdering::None, and ZipperMergeStep assigns fresh sequential output ordinals (push order already equals ordinal order) — but the framework cannot assume that for an arbitrary merge, so it conservatively keeps the reorder for every ordered Step2 output.

Changes

  • A doc comment on TypedStep2::build_output_set explaining the deliberate difference from the single-input path.
  • step2_serial_byitemordinal_output_is_reordered_not_collapsed: pushes a Serial Step2's ByItemOrdinal output out of ordinal order (2, 0, 1) and asserts the consumer receives it in ordinal order (0, 1, 2). It fails if the single-input collapse is ever added to the Step2 path (verified by temporarily applying the collapse).

Note

Stacked on #440 (the fgumi-pipeline-core extraction) so it lives in the new crate location; targets that branch and rebases onto feat-runall once #440 merges. No behavior change — documentation + a pinning test.

Summary by CodeRabbit

  • Tests
    • Added regression test to ensure ordered output delivery is properly reordered rather than collapsed during serial execution, improving reliability and correctness validation.

@nh13
nh13 temporarily deployed to github-actions June 19, 2026 21:07 — with GitHub Actions Inactive
@coderabbitai

coderabbitai Bot commented Jun 19, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: acf9c8f8-ffe5-4b9f-86a2-8baa4e93027a

📥 Commits

Reviewing files that changed from the base of the PR and between a5c77b3 and a5dcaa1.

📒 Files selected for processing (1)
  • crates/fgumi-pipeline-core/src/erased.rs

Walkthrough

Adds a comment block to TypedStep2::build_output_set explaining why the reorder stage cannot be collapsed for ordered two-input steps under Serial/Exclusive execution, unlike the single-input TypedStep path. Adds a regression test that pushes items out of ordinal order through a serial Step2 and asserts ordinal-order delivery.

Changes

Step2 reorder-stage comment and regression test

Layer / File(s) Summary
build_output_set comment + regression test
crates/fgumi-pipeline-core/src/erased.rs
Comment documents the TypedStep vs TypedStep2 collapse asymmetry and why the reorder stage is load-bearing for ByItemOrdinal two-input merges. New test step2_serial_byitemordinal_output_is_reordered_not_collapsed pushes out-of-order items and asserts ordinal delivery.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately identifies the main change: documenting a deliberate asymmetry in Step2's reorder-stage behavior that differs from single-input steps.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch nh/step2-reorder-asymmetry

Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov

codecov Bot commented Jun 19, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (feat-runall@a5c77b3). Learn more about missing BASE report.

Additional details and impacted files
@@              Coverage Diff               @@
##             feat-runall     #441   +/-   ##
==============================================
  Coverage               ?   93.81%           
==============================================
  Files                  ?      119           
  Lines                  ?    50555           
  Branches               ?        0           
==============================================
  Hits                   ?    47429           
  Misses                 ?     3126           
  Partials               ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@nh13
nh13 force-pushed the nh/extract-pipeline-core branch from 9e62c27 to eff004f Compare June 19, 2026 21:38
@nh13
nh13 force-pushed the nh/step2-reorder-asymmetry branch from ee82d46 to 84c299e Compare June 19, 2026 21:41
@nh13
nh13 temporarily deployed to github-actions June 19, 2026 21:41 — with GitHub Actions Inactive
Base automatically changed from nh/extract-pipeline-core to feat-runall June 19, 2026 22:50
A review of `TypedStep2::build_output_set` asked why it doesn't apply the
Serial/Exclusive reorder-collapse that `TypedStep::build_output_set` has.
Document that the asymmetry is intentional and add a test that pins it.

The single-input collapse is sound because of a universal property: a Serial
single-input step consumes one already-ordered stream, so any input ordinal it
propagates onto an output is emitted in push order, making the reorder stage
redundant. A two-input MERGE has no such guarantee — it interleaves two branches
and can push a `ByItemOrdinal` output out of ordinal order even under serial
execution, so its reorder stage is load-bearing. Collapsing it would silently
deliver records out of order. Today's production Step2s happen not to need it
(`PairRawFastq` uses `BranchOrdering::None`; `ZipperMergeStep` assigns fresh
sequential ordinals), but the framework cannot assume that for an arbitrary
merge.

Adds `step2_serial_byitemordinal_output_is_reordered_not_collapsed`, which
pushes a Serial Step2's ordered output out of ordinal order and asserts the
consumer receives it in ordinal order — it fails if the single-input collapse is
ever added to the Step2 path (verified).

Stacked on the fgumi-pipeline-core extraction (#440); rebases onto feat-runall
once that merges.
@nh13
nh13 force-pushed the nh/step2-reorder-asymmetry branch from 84c299e to a5dcaa1 Compare June 19, 2026 22:55
@nh13
nh13 temporarily deployed to github-actions June 19, 2026 22:55 — with GitHub Actions Inactive
@nh13

nh13 commented Jun 20, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai resume

@coderabbitai

coderabbitai Bot commented Jun 20, 2026

Copy link
Copy Markdown
✅ Action performed

Reviews resumed.

@nh13

nh13 commented Jun 20, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai resume

@nh13

nh13 commented Jun 20, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 20, 2026

Copy link
Copy Markdown
✅ Action performed

Reviews resumed.

@coderabbitai

coderabbitai Bot commented Jun 20, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@nh13
nh13 merged commit e60d3f3 into feat-runall Jun 20, 2026
10 checks passed
@nh13
nh13 deleted the nh/step2-reorder-asymmetry branch June 20, 2026 03:08

This branch was previously deployed

1 inactive deployment
github-actions — a5dcaa1f Deployed Jun 19, 2026 by nh13 via coverage #1708
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant