Skip to content

fix(group): add reverse-orientation edges to the parallel paired assigner (GRP3-01) - #525

Merged
nh13 merged 1 commit into
mainfrom
nh/fix-group-paired-parallel-assigner
Jul 16, 2026
Merged

nh13 merged 1 commit into
mainfrom
nh/fix-group-paired-parallel-assigner

Conversation

@nh13

@nh13 nh13 commented Jul 9, 2026 •

Copy link
Copy Markdown
Member

Stacked on #510 (base branch nh/fix-group-unequal-umi-length) — both touch crates/fgumi-umi/src/assigner.rs and src/lib/umi/parallel_assigner.rs. Merge #510 first.

Fixes the audit's second remaining S1, GRP3-01.

The bug

The parallel ParallelPairedAssigner discovered adjacency edges by forward Hamming distance on canonical (lexically smaller of A-B / B-A) forms only, on the claim that "canonicalization already handles reverse-orientation matching." It does not: a single mismatch can flip which half is lexically smaller, so two reads of one molecule canonicalize to forms that are far apart forward yet within threshold when one is reversed.

Minimal case (--edits 1): CAAA-GTTT and ATTT-CAAA are the same molecule (reverse of CAAA-GTTT is GTTT-CAAA, Hamming-1 from ATTT-CAAA), but their canonical forms differ at every base forward. The sequential PairedUmiAssigner groups them via its within(l, r) OR within(reverse(l), r) check; the parallel path split them:

sequential: [PairedB(0), PairedA(0)]   # one molecule, opposite strands
parallel:   [PairedA(1), PairedA(0)]   # two molecules (over-split), both strand A

The multi-threaded paired/duplex path therefore over-split molecules versus fgbio and versus fgumi's own sequential assigner. Reachable at the Paired Auto parallel threshold of 128 templates/position, and forced by --allow-unmapped.

The fix

  • Union the forward edges with a reverse-orientation edge pass (discover_paired_reverse_edges), mirroring the sequential matches_paired. The generic forward discover_edges_parallel_k is shared with the non-paired assigners and is left untouched.
  • A read reached via a reverse-orientation edge is on the OPPOSITE strand, so /A /B can no longer be derived from "raw == its own canonical". Strand is now assigned relative to each molecule's root, mirroring the sequential assigner — including its palindrome insert-order quirk (a UMI whose halves are identical is its own reverse; a palindromic root is labeled B, a palindromic non-root child A).

Tests

  • test_parallel_paired_groups_reverse_orientation_edge — the minimal CAAA-GTTT / ATTT-CAAA case (RED before the fix).
  • GRP3-T2 — a parallel-vs-sequential PAIRED parity proptest that builds pools from random molecules emitted in both orientations with single-base mutations, asserting identical base-molecule AND strand partitions at threads {1, 4, 16} (which also pins thread-determinism). It surfaced the palindrome strand cases, now covered by committed regression seeds. Stressed to 3000 cases.

cargo ci-fmt && ci-lint && ci-test green (2221 tests); no regression in the existing paired / group / determinism suites.

Notes

  • NEEDS-RESIGN: 1Password signing was unavailable at commit time (biometric); the commit is currently unsigned. Will re-sign and force-push once signing recovers.
  • The staged real-data agilent-hs2 repro (group --strategy paired --threads 16 vs the hardened compare bams) could not be run — the scratch SSD is not mounted right now. The programmatic RED test + proptest fully reproduce the divergence and verify parity; the agilent comparison can be run for extra confidence when the SSD is back.
  • coderabbit --agent flagged one minor item in SimpleErrorUmiAssigner (the edit assigner, assigner.rs:989-991) about fix(group): reject UMIs of differing length in all assigners (GRP-01) #510's length-guard ordering on invalid UMIs — outside this diff and unrelated to the paired assigner, so left for fix(group): reject UMIs of differing length in all assigners (GRP-01) #510 rather than mixed in here.

Summary by CodeRabbit

  • Bug Fixes

    • Improved paired UMI grouping for reverse-orientation adjacency (GRP3-01), including deterministic clustering.
    • Fixed split-point collision handling to match sequential paired behavior.
    • Corrected paired strand labeling (including palindromic cases) and prevented reverse-orientation invalid UMIs from being merged.
  • Testing

    • Strengthened assignment equivalence checks (base partition + exact PairedA/PairedB labeling).
    • Expanded regression and property-based coverage for reverse invalids, GRP3-01 grouping, and multi-thread consistency using oracle comparisons (including repeated runs to detect nondeterminism).
    • Refreshed saved regression failures with newly captured cases.

@nh13
nh13 temporarily deployed to github-actions July 9, 2026 23:11 — with GitHub Actions Inactive
@coderabbitai

coderabbitai Bot commented Jul 9, 2026 •

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Parallel paired assignment now discovers reverse-orientation UMI edges, handles split-point collisions through sequential fallback, tracks cluster roots, computes strand labels, and preserves distinct invalid UMIs. Tests validate fgbio parity, sequential equivalence, and deterministic results across thread counts.

Changes

Paired UMI orientation handling

Layer / File(s) Summary
Reverse-edge discovery and strand computation
src/lib/umi/parallel_assigner.rs
Adds reverse-orientation adjacency discovery and root-relative /A versus /B strand computation, including palindrome handling.
Parallel assignment integration
src/lib/umi/parallel_assigner.rs
Delegates asymmetric-halves collisions to the sequential assigner, combines and sorts adjacency edges, records cluster roots, maps strand labels, and keys invalid fallbacks by raw uppercase UMI.
Sequential parity and regression coverage
src/lib/umi/parallel_assigner.rs, proptest-regressions/lib/umi/parallel_assigner.txt
Adds strand-sensitive equivalence checks, targeted regressions, fgbio-oracle coverage, multi-thread property tests, and saved regression cases.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Reads
  participant ParallelPairedAssigner
  participant discover_paired_reverse_edges
  participant paired_canonical_strands
  Reads->>ParallelPairedAssigner: submit paired UMI reads
  ParallelPairedAssigner->>ParallelPairedAssigner: detect split-point collision
  ParallelPairedAssigner->>discover_paired_reverse_edges: discover reverse-orientation edges
  discover_paired_reverse_edges-->>ParallelPairedAssigner: combined adjacency edges
  ParallelPairedAssigner->>paired_canonical_strands: compute root-relative strands
  paired_canonical_strands-->>ParallelPairedAssigner: PairedA/PairedB labels
  ParallelPairedAssigner-->>Reads: molecule assignments
Loading

Possibly related issues

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main fix: reverse-orientation edges in the parallel paired assigner.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch nh/fix-group-paired-parallel-assigner

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Jul 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.37838% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.84%. Comparing base (808930d) to head (855db1b).
⚠️ Report is 15 commits behind head on main.

Files with missing lines Patch % Lines
src/lib/umi/parallel_assigner.rs 98.37% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #525      +/-   ##
==========================================
+ Coverage   92.65%   92.84%   +0.19%     
==========================================
  Files         166      166              
  Lines      100136   102064    +1928     
==========================================
+ Hits        92776    94765    +1989     
+ Misses       7360     7299      -61     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@nh13
nh13 force-pushed the nh/fix-group-paired-parallel-assigner branch from e199117 to 742fa85 Compare July 9, 2026 23:30
@nh13
nh13 temporarily deployed to github-actions July 9, 2026 23:30 — with GitHub Actions Inactive
@nh13
nh13 force-pushed the nh/fix-group-paired-parallel-assigner branch from 742fa85 to dcbbed5 Compare July 9, 2026 23:46
@nh13
nh13 temporarily deployed to github-actions July 9, 2026 23:46 — with GitHub Actions Inactive
@nh13

nh13 commented Jul 10, 2026

Copy link
Copy Markdown
Member Author

Real-data confirmation (resolves the deferred agilent check). Ran on the full 53,139,936-record agilent-hs2 paired dataset with the parallel path forced on every ≥128-template position group (the natural GRP3-01 condition):

  • NEW parallel vs NEW sequential → MI bijection mismatches: 0 / EQUIVALENT. On real data, the fixed parallel assigner reproduces the (always-correct) sequential grouping exactly.
  • No regression: the OLD and NEW binaries produce the identical grouping at the default threshold, and both differ from the fgbio baseline by the same count when fed a re-sorted proxy input (a position-group-boundary artifact of the proxy sort order, independent of the assigner).

This complements the GRP3-T2 proptest (parallel ≡ sequential across threads {1,4,16}, 3000 cases). The NEEDS-RESIGN note is also resolved — the commit is now signed.

@nh13
nh13 force-pushed the nh/fix-group-unequal-umi-length branch from 3745dc7 to 718460d Compare July 11, 2026 21:17
@nh13
nh13 force-pushed the nh/fix-group-paired-parallel-assigner branch from dcbbed5 to e84180f Compare July 11, 2026 22:17
@nh13
nh13 temporarily deployed to github-actions July 11, 2026 22:17 — with GitHub Actions Inactive
@nh13
nh13 force-pushed the nh/fix-group-unequal-umi-length branch from 718460d to 032beae Compare July 12, 2026 15:11
@nh13
nh13 force-pushed the nh/fix-group-paired-parallel-assigner branch from e84180f to 89f556d Compare July 12, 2026 16:45
@nh13
nh13 temporarily deployed to github-actions July 12, 2026 16:45 — with GitHub Actions Inactive
@nh13
nh13 force-pushed the nh/fix-group-unequal-umi-length branch 2 times, most recently from 6396833 to bd1f81c Compare July 13, 2026 03:21
@nh13
nh13 force-pushed the nh/fix-group-paired-parallel-assigner branch from 89f556d to 81c8d8c Compare July 13, 2026 03:33
@nh13
nh13 temporarily deployed to github-actions July 13, 2026 03:33 — with GitHub Actions Inactive
@nh13
nh13 force-pushed the nh/fix-group-unequal-umi-length branch from bd1f81c to d03a568 Compare July 13, 2026 18:45
@nh13
nh13 force-pushed the nh/fix-group-paired-parallel-assigner branch from 81c8d8c to 9555fd6 Compare July 13, 2026 19:23
@nh13
nh13 temporarily deployed to github-actions July 13, 2026 19:23 — with GitHub Actions Inactive
@nh13
nh13 force-pushed the nh/fix-group-unequal-umi-length branch from d03a568 to 8b6838b Compare July 13, 2026 22:00
@nh13
nh13 force-pushed the nh/fix-group-paired-parallel-assigner branch 2 times, most recently from b2859e1 to 2fc5c0a Compare July 13, 2026 22:06
@nh13
nh13 temporarily deployed to github-actions July 13, 2026 22:06 — with GitHub Actions Inactive
@nh13
nh13 force-pushed the nh/fix-group-paired-parallel-assigner branch from 2fc5c0a to 015573d Compare July 14, 2026 01:35
@nh13
nh13 temporarily deployed to github-actions July 14, 2026 01:35 — with GitHub Actions Inactive
Base automatically changed from nh/fix-group-unequal-umi-length to main July 14, 2026 02:05
@coderabbitai

coderabbitai Bot commented Jul 16, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/umi/parallel_assigner.rs`:
- Around line 280-281: Update the key construction in the forward-index map used
by the parallel paired assigner so it preserves the canonical pair’s split point
instead of relying on BitEnc alone. Apply the same split-aware key consistently
wherever forward assignments are looked up, and add a regression covering paired
UMIs with identical concatenated bases but different half lengths.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 8f8cf27a-d9b9-448e-ba68-37b1ba7b573e

📥 Commits

Reviewing files that changed from the base of the PR and between 852747b and 9518d65.

📒 Files selected for processing (2)
  • proptest-regressions/lib/umi/parallel_assigner.txt
  • src/lib/umi/parallel_assigner.rs

Comment thread src/lib/umi/parallel_assigner.rs
@nh13

nh13 commented Jul 16, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 16, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@nh13

nh13 commented Jul 16, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 16, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/umi/parallel_assigner.rs`:
- Around line 275-304: Update paired assignment around
discover_paired_reverse_edges and its callers so every asymmetric paired pool
falls back to the sequential assigner, not only pools with encoding collisions.
Detect differing forward/reverse half lengths and delegate those pools before
parallel reverse-edge discovery; otherwise make reverse matching split-aware
while preserving the sequential assigner’s threshold behavior and output.
- Around line 1920-2032: Replace the hand-derived expectations in
test_paired_matches_fgbio_oracle with a checked-in fixture generated
programmatically by a pinned fgbio invocation, including the command/version and
reproducible input data. Load the fixture in assert_matches_fgbio_oracle and
validate both PairedUmiAssigner and ParallelPairedAssigner against the generated
fgbio base-molecule and strand results, preserving coverage for the listed
reverse, palindrome, tie-break, and threshold cases.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 87bf8561-f7bb-4e53-84ef-136d74f7b5ab

📥 Commits

Reviewing files that changed from the base of the PR and between 9518d65 and 452ffb9.

📒 Files selected for processing (2)
  • proptest-regressions/lib/umi/parallel_assigner.txt
  • src/lib/umi/parallel_assigner.rs

Comment thread src/lib/umi/parallel_assigner.rs
Comment on lines +1920 to +2032
/// Assert `actual` matches a hand-derived fgbio oracle.
///
/// `expected[i] = (group, strand)`, where `group` is an arbitrary label shared by
/// every read fgbio places in one *base* molecule and `strand` is fgbio's absolute
/// duplex suffix (`'A'` for `/A` / [`MoleculeId::PairedA`], `'B'` for `/B` /
/// [`MoleculeId::PairedB`]). This checks the base-molecule partition and each read's
/// absolute strand, but not the concrete numeric molecule id (a relabeling).
///
/// The oracle values are derived by hand from fgbio's `PairedUmiAssigner`
/// (`GroupReadsByUmi.assignIdsToNodes`): the canonical (lexically-smaller-half-first)
/// spelling of a cluster's root maps to `/A` and its reverse to `/B`; a descendant
/// takes `/A` for the orientation closer to the root and `/B` for the other; and
/// fgbio's `Map` last-write-wins gives the palindrome quirks (a palindrome root
/// canonical spelling resolves to `/B`, a palindrome descendant to `/A`). Because the
/// oracle is independent of BOTH fgumi assigners, a shared sequential/parallel defect
/// cannot satisfy it.
fn assert_matches_fgbio_oracle(actual: &[MoleculeId], expected: &[(u32, char)]) {
assert_eq!(
actual.len(),
expected.len(),
"length mismatch: actual={actual:?} expected={expected:?}"
);
// Base-molecule partition: reads i and j share a base id IFF the oracle groups match.
for i in 0..actual.len() {
for j in 0..actual.len() {
let actual_same = actual[i].base_id_string() == actual[j].base_id_string();
let expected_same = expected[i].0 == expected[j].0;
assert_eq!(
actual_same, expected_same,
"base-partition mismatch at ({i},{j}); actual={actual:?} expected={expected:?}"
);
}
}
// Absolute strand per read.
for (i, (id, &(_, strand))) in actual.iter().zip(expected).enumerate() {
let actual_strand = match id {
MoleculeId::PairedA(_) => 'A',
MoleculeId::PairedB(_) => 'B',
other => panic!("read {i}: expected a paired strand, got {other:?}"),
};
assert_eq!(
actual_strand, strand,
"strand mismatch at read {i}; actual={actual:?} expected={expected:?}"
);
}
}

/// GRP3-01 fgbio oracle: pin the fgbio-derived base-molecule partition AND absolute
/// `/A` `/B` strand for the reverse-aware paired cases, and assert BOTH the sequential
/// `PairedUmiAssigner` and the parallel `ParallelPairedAssigner` reproduce it.
///
/// The other reverse-orientation suites compare parallel against the sequential
/// assigner only, so a defect shared by both would pass. This table's expectations
/// are hand-derived from fgbio's `PairedUmiAssigner` (see
/// [`assert_matches_fgbio_oracle`]), giving an oracle independent of either fgumi
/// implementation. Covers reverse edges, palindrome roots, palindrome children,
/// equal-count tie-breaking, and the child-count threshold boundary (inclusion and
/// exclusion), per the grouping path's fgbio-parity requirement.
#[rstest]
// Reverse-orientation edge that canonicalization alone misses; equal counts, so the
// root is the lexically-smaller canonical "ATTT-CAAA" (=> /A) and "CAAA-GTTT" the
// reversed descendant (=> /B). The tie-break choosing the root is what fixes strand
// here: rooting on the other member would invert both labels.
#[case::reverse_edge_equal_count_tiebreak(
&["CAAA-GTTT", "ATTT-CAAA"],
&[(0, 'B'), (0, 'A')],
)]
// Palindrome root (halves equal => umi == reverse(umi)): fgbio maps the root canonical
// spelling then its (identical) reverse, so last-write-wins lands the palindrome root
// on /B. Its non-palindrome descendant "ACGT-ACGA", reached in the reverse spelling,
// takes /A.
#[case::palindrome_root(
&["ACGT-ACGT", "ACGT-ACGT", "ACGT-ACGA"],
&[(0, 'B'), (0, 'B'), (0, 'A')],
)]
// Palindrome descendant "AAAA-AAAA" under a non-palindrome root "AAAA-AAAC": fgbio's
// else-branch maps the palindrome child to /B then its identical reverse to /A, so
// last-write-wins lands the palindrome child on /A. The reverse spelling of the root
// ("AAAC-AAAA", index 2) is the strand-/B counterpart, so not every read is /A.
#[case::palindrome_child(
&["AAAA-AAAC", "AAAA-AAAC", "AAAC-AAAA", "AAAA-AAAA"],
&[(0, 'A'), (0, 'A'), (0, 'B'), (0, 'A')],
)]
// Threshold boundary, INCLUDED: descendant count 3 == root_count(5)/2 + 1 == 3, so the
// "AAAA-CCCG" node joins the "AAAA-CCCC" molecule (one group).
#[case::threshold_boundary_included(
&[
"AAAA-CCCC", "AAAA-CCCC", "AAAA-CCCC", "AAAA-CCCC", "AAAA-CCCC",
"AAAA-CCCG", "AAAA-CCCG", "AAAA-CCCG",
],
&[(0, 'A'), (0, 'A'), (0, 'A'), (0, 'A'), (0, 'A'), (0, 'A'), (0, 'A'), (0, 'A')],
)]
// Threshold boundary, EXCLUDED: descendant count 4 > root_count(5)/2 + 1 == 3, so the
// "AAAA-CCCG" node splits off into its own molecule (group 1), each a /A root.
#[case::threshold_boundary_excluded(
&[
"AAAA-CCCC", "AAAA-CCCC", "AAAA-CCCC", "AAAA-CCCC", "AAAA-CCCC",
"AAAA-CCCG", "AAAA-CCCG", "AAAA-CCCG", "AAAA-CCCG",
],
&[(0, 'A'), (0, 'A'), (0, 'A'), (0, 'A'), (0, 'A'), (1, 'A'), (1, 'A'), (1, 'A'), (1, 'A')],
)]
fn test_paired_matches_fgbio_oracle(#[case] umis: &[&str], #[case] expected: &[(u32, char)]) {
let umis: Vec<Umi> = umis.iter().map(|s| (*s).to_string()).collect();

// Both the sequential assigner AND the parallel assigner (at multiple thread
// counts) must reproduce the independent fgbio oracle.
let sequential = crate::umi::PairedUmiAssigner::new(1).assign(&umis);
assert_matches_fgbio_oracle(&sequential, expected);
for threads in [1usize, 4, 16] {
let parallel = ParallelPairedAssigner::new(1, threads).assign(&umis);
assert_matches_fgbio_oracle(&parallel, expected);
}
}

@coderabbitai coderabbitai Bot Jul 16, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift

Generate the fgbio oracle programmatically.

The expectations are documented as hand-derived, so this still does not satisfy the required reproducible fgbio-baseline coverage. Generate a checked-in fixture from a pinned fgbio invocation and validate both assigners against it.

As per path instructions, output-changing grouping requires parity coverage against “the sequential code path and the fgbio baseline, generated programmatically.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/umi/parallel_assigner.rs` around lines 1920 - 2032, Replace the
hand-derived expectations in test_paired_matches_fgbio_oracle with a checked-in
fixture generated programmatically by a pinned fgbio invocation, including the
command/version and reproducible input data. Load the fixture in
assert_matches_fgbio_oracle and validate both PairedUmiAssigner and
ParallelPairedAssigner against the generated fgbio base-molecule and strand
results, preserving coverage for the listed reverse, palindrome, tie-break, and
threshold cases.

Source: Path instructions

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deferring this one, intentionally leaving the thread open for a maintainer call rather than acting on it in this PR.

The test_paired_matches_fgbio_oracle expectations are already independent of both fgumi implementations — they are hand-derived from fgbio's PairedUmiAssigner.assignIdsToNodes algorithm (documented case-by-case in the table: reverse edges, palindrome roots/children via fgbio's last-write-wins Map, equal-count tie-breaking, and the child-count threshold boundary on both sides). Because the oracle is derived independently of both the sequential and parallel fgumi assigners, a defect shared by both cannot satisfy it — which is the coverage the path instruction is after.

Replacing it with a checked-in fixture generated by a pinned fgbio invocation runs against two repo conventions: fgumi's test suite is pure Rust with no JVM/fgbio runtime dependency, and this project generates test data programmatically rather than committing fixture files. A programmatic alternative (a Rust reimplementation of fgbio's paired assignment to generate expectations at test time) would just re-implement the logic under test, adding a second copy to keep in sync without adding independent signal.

Leaving this open so a maintainer can weigh in on whether to invest in a fgbio-provenanced fixture harness (better as its own infra task than folded into this focused reverse-edge fix). Happy to file a tracking issue if that's the preferred direction.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nh13 Risk remains: the hand-derived oracle is independent, but it is not reproducible, pinned fgbio-baseline coverage as required by the path instruction.

Keeping this open for maintainer scope judgment is reasonable. If you confirm, I can create a follow-up GitHub issue for a pinned, generated fgbio fixture/harness—explicitly preserving the Rust-only default test path and avoiding a JVM runtime requirement in normal CI.


✏️ Learnings added
Learnt from: nh13
Repo: fulcrumgenomics/fgumi PR: 525
File: src/lib/umi/parallel_assigner.rs:1963-2075
Timestamp: 2026-07-16T15:17:40.377Z
Learning: In `src/lib/umi/parallel_assigner.rs`, fgumi’s normal test suite is pure Rust and does not depend on a JVM or fgbio runtime; the repository convention favors programmatic test-data generation over committed fixture files. Any future fgbio-baseline harness for paired UMI parity should preserve a Rust-only default test path.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

@nh13
nh13 force-pushed the nh/fix-group-paired-parallel-assigner branch from 452ffb9 to 9cb3ecc Compare July 16, 2026 15:08
@nh13
nh13 temporarily deployed to github-actions July 16, 2026 15:08 — with GitHub Actions Inactive
@nh13

nh13 commented Jul 16, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 16, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@nh13

nh13 commented Jul 16, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 16, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@nh13

nh13 commented Jul 16, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 16, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@nh13

nh13 commented Jul 16, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 16, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/umi/parallel_assigner.rs`:
- Around line 2064-2073: Update test_paired_matches_fgbio_oracle to parameterize
max_mismatches and pass that threshold to both PairedUmiAssigner and
ParallelPairedAssigner. Extend the oracle cases with a reverse-only distance-2
input using max_mismatches = 2, while retaining the existing k=1 cases and
fgbio-oracle assertions.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 6fedf953-c547-4029-b5ed-4e8a81e4ef6b

📥 Commits

Reviewing files that changed from the base of the PR and between 9518d65 and 9cb3ecc.

📒 Files selected for processing (2)
  • proptest-regressions/lib/umi/parallel_assigner.txt
  • src/lib/umi/parallel_assigner.rs

Comment thread src/lib/umi/parallel_assigner.rs Outdated
…gner (GRP3-01)

The parallel `ParallelPairedAssigner` discovered adjacency edges by forward
Hamming distance on canonical (lexically smaller of A-B / B-A) forms only,
claiming "canonicalization already handles reverse-orientation matching". It does
not: a single mismatch can flip which half is lexically smaller, so two reads of
one molecule canonicalize to forms that are far apart forward yet within threshold
when one is reversed. The multi-threaded paired/duplex path therefore over-split
molecules versus fgbio and versus fgumi's own sequential `PairedUmiAssigner`
(reachable at the Paired Auto parallel threshold of 128 templates/position, and
forced by `--allow-unmapped`).

The sequential assigner matches on `within(l, r) OR within(reverse(l), r)`. This
change unions the forward edges with a reverse-orientation edge pass
(`discover_paired_reverse_edges`), so the parallel base-molecule partition now
matches the sequential one.

Reverse-orientation edges also mean a read reached via such an edge is on the
OPPOSITE strand, so strand (/A /B) can no longer be derived from "raw == its own
canonical". Strand is now assigned relative to each molecule's root (highest-count
member), mirroring the sequential assigner — including its palindrome insert-order
quirk (a UMI whose halves are identical is its own reverse; a palindromic root is
labeled B and a palindromic non-root child A, matching the sequential's last-write
insert order).

Adds `test_parallel_paired_groups_reverse_orientation_edge` (the minimal
"CAAA-GTTT" / "ATTT-CAAA" case) and GRP3-T2, a parallel-vs-sequential PAIRED parity
proptest that builds pools from random molecules in both orientations with
single-base mutations and asserts identical base-molecule AND strand partitions at
threads {1, 4, 16} (which also pins thread-determinism). The proptest surfaced the
palindrome strand cases now covered by the committed regression seeds.

Also key the parallel main-path invalid-UMI fallback by the raw uppercase string
instead of the canonical form. Two invalid (non-encodable) UMIs that are reverses
of each other (e.g. "ACGN-TTTT" / "TTTT-ACGN") canonicalize to the same form, so
the canonical keying merged them into one Single molecule whenever a valid UMI was
also present (the main path, not the all-invalid fast path). The sequential
assigner and the all-invalid branch both key by the raw uppercase string and keep
such reverses distinct, so this was a --threads-dependent divergence. Now all three
paths agree (one molecule per distinct raw UMI string), covered by a new
paired_reversed_invalid case in the cross-assigner parity harness.

Also guard the pre-existing `BitEnc`-drops-dash limitation in the parallel paired
path. `BitEnc::from_umi_str` discards the `-`, so the parallel forward AND reverse
edit distances are dash-blind, whereas fgbio and the sequential `PairedUmiAssigner`
compare the dash-delimited string byte-for-byte in both orientations. The two agree
exactly when every UMI has symmetric halves (the dash sits at a fixed position); with
asymmetric halves (read-1 UMI length != read-2 UMI length) the dash-blind path
diverges in two ways: distinct forms with the same concatenated bases but different
splits collide to one encoding (`AC-GTA` / `ACG-TA`), and the halves-swapped reverse
encoding can forge a within-threshold reverse edge the dash-sensitive reference never
draws (`A-AC` reverse `ACA` is Hamming-1 from `A-CT`'s `ACT`). Until the full
split-aware rework lands (tracked in #586), delegate any pool containing an
asymmetric-halves UMI to the sequential assigner, which is byte-for-byte faithful to
fgbio; symmetric pools take the fast parallel path unchanged. Adds
`test_parallel_paired_mixed_half_length_collision_matches_sequential` (forward
collision) and `test_parallel_paired_asymmetric_noncollision_matches_sequential` (the
reverse-edge case with no forward collision), both of which diverged before the guard.
@nh13
nh13 force-pushed the nh/fix-group-paired-parallel-assigner branch from 9cb3ecc to 855db1b Compare July 16, 2026 19:27
@nh13
nh13 temporarily deployed to github-actions July 16, 2026 19:28 — with GitHub Actions Inactive
@nh13
nh13 merged commit f55ac4b into main Jul 16, 2026
8 checks passed
@nh13
nh13 deleted the nh/fix-group-paired-parallel-assigner branch July 16, 2026 19:31
@nh13 nh13 mentioned this pull request Jul 16, 2026
nh13 added a commit that referenced this pull request Aug 22, 2026
…halves

The parallel paired UMI assigner encoded canonical paired UMIs with
`BitEnc::from_umi_str`, which drops the `-`. With asymmetric halves the split
point varies, so distinct canonical forms collide (`AC-GTA` / `ACG-TA`) and a
halves-swapped reverse encoding can forge within-threshold edges the
dash-sensitive reference never draws. PR #525 worked around this by delegating
any asymmetric-halves pool to the single-threaded sequential `PairedUmiAssigner`.

Replace that fallback with a native dash-aware path. For pools that contain an
asymmetric-halves UMI, discover edges by comparing the full dash-delimited
canonical strings position-for-position -- exactly the sequential assigner's
`matches_paired` relation -- in parallel over ordered `(parent, child)` pairs.
The relation is directional (with asymmetric halves `reverse()` moves the dash,
so `matches_paired` is not symmetric), and the sequential adjacency graph is a
dynamic directed BFS in which a node pulled into a cluster can then absorb a
lower-indexed node. So the asymmetric branch feeds a directed adjacency to the
existing count-gated BFS, reproducing the sequential traversal exactly. The
common symmetric pool keeps the sub-quadratic `BitEnc` neighbour-generation path
unchanged.

Add an asymmetric-halves parity proptest across threads {1,4,16} and pinned
regression cases for the two subtle failure modes (directional reverse edge; a
child absorbed by a higher-indexed node). Closes #586.

This branch was previously deployed

1 inactive deployment
github-actions — 855db1b5 Deployed Jul 16, 2026 by nh13 via coverage #2599
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant