Skip to content

feat(group): accept --index-threshold always|never, reject unsatisfiable requests - #632

Merged
nh13 merged 2 commits into
mainfrom
nh/fix-index-threshold-doc
Jul 26, 2026
Merged

nh13 merged 2 commits into
mainfrom
nh/fix-index-threshold-doc

Conversation

@nh13

@nh13 nh13 commented Jul 22, 2026 •

Copy link
Copy Markdown
Member

What this does

--index-threshold was a bare usize meaning "index once a position group holds at least this many distinct UMIs". That left no way to turn the index off: suppressing it needed a value larger than any group, i.e. literally --index-threshold 18446744073709551615.

This makes the option a keyword-or-number, matching --parallel-group-min-templates one screen away in the same file:

--index-threshold always     index every position group
--index-threshold never      always scan all UMI pairs
--index-threshold <N>        index groups of N or more (default 100)

never is the new capability. always is sugar for 0, which already meant this but read as its opposite — the trap that produced the wrong help text in the first place. Every existing integer invocation keeps working unchanged, 0 included, so no command line silently changes meaning.

Keeping this in one option rather than adding a --no-index flag makes the contradictory state unrepresentable: two knobs controlling one behaviour would need a precedence rule for --no-index --index-threshold 50.

always is an assertion, so it can fail

A configuration that can never index is now a command-line error rather than a flag that is quietly ignored:

$ fgumi group --strategy identity --index-threshold always
Error: --index-threshold always cannot be honoured with --strategy identity:
the identity strategy never uses the UMI index. Drop --index-threshold to leave
indexing to the default threshold, or pass --index-threshold never to state that
a linear scan is intended.

$ fgumi group --strategy adjacency --edits 2 --index-threshold always
Error: --index-threshold always cannot be honoured with --strategy adjacency
--edits 2: the adjacency strategy only indexes at --edits 1. ...

That second case surfaces an asymmetry the help had left implicit: build_adjacency_graph_bitenc gates on max_mismatches == 1 and SimpleErrorUmiAssigner::assign gates its defer/index branch the same way, so the adjacency and edit strategies both silently ignore the index at any other --edits, while paired indexes at all of them. Strategy::can_use_index now states it, and the check runs against the effective strategy and edits so --no-umi (which forces identity) is caught too.

A bare integer is deliberately not checked. It is a tuning knob allowed to end up inert — otherwise the default 100 could not coexist with --strategy identity — and that includes 0, even though 0 admits every group exactly as always does.

dedup carries its own copy of the option, so the validation lives in commands::common and both call it. Assigner constructors take T: Into<IndexThreshold>, leaving every existing integer call site untouched.

Integrating with edit's own index (#645)

Edit grew an index of its own in #645, gated on its own measured crossover (EDIT_INDEX_THRESHOLD, 200) rather than the shared default. IndexThreshold::floored_at expresses that floor: a numeric threshold is raised to the crossover, while the keywords pass through untouched, so always still means every group — the escape hatch that isolates the index in a profile — and never still means none. Every existing numeric invocation keeps the max(flag, 200) behaviour it had.

What Strategy::can_use_index reports for Edit is edits == 1, the same as adjacency. It is tempting to say otherwise: components_via_index hands max_mismatches straight to NgramIndex::new, which partitions each UMI into max_mismatches + 1 pieces and pigeonholes over them, so the index is capable at any distance — a new brute-force test pins that across five UMI lengths and four distances, including the lengths max_mismatches + 1 does not divide.

But assign never reaches it elsewhere. #645 gated the defer/index branch on one mismatch deliberately, because EDIT_INDEX_THRESHOLD's crossover was measured there and nowhere else, and past it the index loses: the partitions get too short to be selective (4 bases at k=1, 2 at k=2 for an 8-base UMI) while single linkage collapses the group into one component, which makes the set-merge's early-exiting scan cheaper rather than dearer. Widening the gate is a benchmarking question — benches/umi_assigner_threshold.rs sweeps one mismatch only.

So both edits == 1 gates now read from the assigner that enforces them, SimpleErrorUmiAssigner::indexes_at_edit_distance and its adjacency twin, rather than being restated in can_use_index, in the Index threshold: startup line, and in the parity test's rationale. Restating them is how they came to disagree in the first place: the index's capability at k>1 was read as evidence that assign used it there. The parity test stays at one mismatch — at any other distance both sides run the identical set-merge, so sweeping wider would compare the scan against itself.

The gate has a name now

Rather than only rewording the help, the first commit extracts the condition so it can be tested rather than restated:

  • AdjacencyUmiAssigner::uses_index(num_umis) — the threshold and the max_mismatches == 1 restriction; build_adjacency_graph_bitenc calls it.
  • PairedUmiAssigner::uses_index(num_umis) — the threshold only, reading the same index_admits gate build_adjacency_graph runs rather than a parallel copy of it.
  • SimpleErrorUmiAssigner::uses_index(num_umis) — the threshold only, added here alongside edit's integration.

Tests

  • index_threshold.rs — parsing (both keywords, mixed case, integers, rejected input naming the accepted forms), admits, demands_indexing, floored_at, and Display round-trips.
  • test_strategy_can_use_index — the strategy/edits matrix that decides whether always is honourable.
  • test_edit_index_is_built_at_every_edit_distance — pinned against a real NgramIndex build at 0–3 mismatches, not against the flag.
  • test_edit_index_matches_scan — always vs never produce identical molecule ids, swept across 1–3 mismatches. Naming the two sides as keywords also removes the old failure mode where a pair of numbers could silently compare the scan against itself.
  • test_index_threshold_always_rejected_when_index_unreachable / ..._accepted_when_satisfiable on group, and the equivalent end-to-end pair on dedup.

Full suite green: 6709 passed, plus ci-fmt, ci-lint, and docs with RUSTDOCFLAGS=-D warnings.

Not addressed here

--edits > 1 has no index on the adjacency path. The BK-tree branch that would serve k>1 exists but is only reachable from the paired strategy's generic builder, so --edits 2 inherits the O(u²) behaviour on large position groups. --index-threshold always now reports this as an error rather than ignoring it, which is the point — but wiring the BK-tree into the BitEnc path is a separate change and wants a benchmark first.

Summary by CodeRabbit

  • New Features
    • Introduced an IndexThreshold configuration for --index-threshold (always, never, or minimum UMI-group size) with consistent indexing eligibility across strategies and edit distances.
  • Bug Fixes
    • Invalid --index-threshold always combinations are now rejected with clear, strategy-specific error messages instead of being ignored.
    • Startup messaging now accurately reflects when indexing can be used (e.g., edit-indexing only at --edits 1 where applicable).
  • Tests
    • Expanded parsing, strategy eligibility, and command-level validation for --index-threshold semantics, including adjacency constraints and Never handling.

@nh13
nh13 temporarily deployed to github-actions July 22, 2026 06:32 — with GitHub Actions Inactive
@coderabbitai

coderabbitai Bot commented Jul 22, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: ce5042d2-dc95-4ac1-8d4c-e474aceb14a3

📥 Commits

Reviewing files that changed from the base of the PR and between cd4df0c and e6405f6.

📒 Files selected for processing (7)
  • crates/fgumi-umi/src/assigner.rs
  • crates/fgumi-umi/src/index_threshold.rs
  • crates/fgumi-umi/src/lib.rs
  • src/lib/commands/common.rs
  • src/lib/commands/dedup.rs
  • src/lib/commands/group.rs
  • tests/integration/test_dedup_command.rs

Walkthrough

The PR replaces numeric index thresholds with IndexThreshold, applies strategy-specific index eligibility across UMI assigners, and rejects CLI configurations requiring unavailable indexing.

Changes

Index threshold configuration and enforcement

Layer / File(s) Summary
IndexThreshold contract
crates/fgumi-umi/src/index_threshold.rs, crates/fgumi-umi/src/lib.rs
Adds Always, Never, and MinUmis semantics with parsing, formatting, admission logic, tests, and public re-exports.
Assigner indexing integration
crates/fgumi-umi/src/assigner.rs
Propagates typed thresholds and applies strategy-specific eligibility for edit, adjacency, and paired index construction.
CLI propagation and validation
src/lib/commands/common.rs, src/lib/commands/dedup.rs, src/lib/commands/group.rs, tests/integration/test_dedup_command.rs
Updates CLI wiring, logs effective behavior, validates unsatisfiable always configurations, and tests accepted and rejected modes.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Command
  participant Strategy
  participant Validator
  participant Assigner
  Command->>Strategy: resolve strategy and edits
  Command->>Validator: validate IndexThreshold
  Validator->>Strategy: check can_use_index(edits)
  Validator-->>Command: accept or reject configuration
  Command->>Assigner: construct with IndexThreshold
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: typed index-threshold handling with acceptance of always/never and rejection of impossible requests.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch nh/fix-index-threshold-doc

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Jul 22, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.35484% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 93.66%. Comparing base (ef0f6f8) to head (e6405f6).
⚠️ Report is 5 commits behind head on main.

Files with missing lines Patch % Lines
crates/fgumi-umi/src/assigner.rs 98.59% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main     #632    +/-   ##
========================================
  Coverage   93.65%   93.66%            
========================================
  Files         175      176     +1     
  Lines      107601   107772   +171     
========================================
+ Hits       100774   100943   +169     
- Misses       6827     6829     +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@nh13
nh13 force-pushed the nh/fix-index-threshold-doc branch from 4d70bdd to b27997d Compare July 23, 2026 23:05
@nh13
nh13 temporarily deployed to github-actions July 23, 2026 23:05 — with GitHub Actions Inactive
@nh13
nh13 force-pushed the nh/fix-index-threshold-doc branch from b27997d to dc5e8ba Compare July 24, 2026 00:39
@nh13
nh13 temporarily deployed to github-actions July 24, 2026 00:39 — with GitHub Actions Inactive
@nh13

nh13 commented Jul 24, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 24, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@nh13

nh13 commented Jul 24, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 24, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@nh13

nh13 commented Jul 24, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 24, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@nh13

nh13 commented Jul 24, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 24, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@nh13
nh13 temporarily deployed to github-actions July 24, 2026 16:30 — with GitHub Actions Inactive
@nh13

nh13 commented Jul 25, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 25, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/commands/dedup.rs`:
- Around line 1123-1129: The --index-threshold help text incorrectly equates 0
with always beyond gating. Update the documentation on the index_threshold
option in src/lib/commands/dedup.rs lines 1123-1129 and
src/lib/commands/group.rs lines 861-867 to clarify that 0 and always agree only
for gating, while only always asserts and errors when the strategy cannot index;
leave the option behavior unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 50a4b0da-08e7-4c20-9b33-7f5f221d7815

📥 Commits

Reviewing files that changed from the base of the PR and between 50372a6 and 1bbb9fa.

📒 Files selected for processing (7)
  • crates/fgumi-umi/src/assigner.rs
  • crates/fgumi-umi/src/index_threshold.rs
  • crates/fgumi-umi/src/lib.rs
  • src/lib/commands/common.rs
  • src/lib/commands/dedup.rs
  • src/lib/commands/group.rs
  • tests/integration/test_dedup_command.rs

Comment thread src/lib/commands/dedup.rs
The `--index-threshold` help was silent about a second gate.
`build_adjacency_graph_bitenc` requires `self.max_mismatches == 1`, so the
Adjacency strategy ignores `--index-threshold` entirely at any other `--edits`
and falls back to the O(u^2) scan. The Paired strategy has no such restriction:
its generic `build_adjacency_graph` indexes at every edit distance, N-gram for
k=1 and BK-tree for k>1.

Rather than only rewording the help, extract the gate into
`AdjacencyUmiAssigner::uses_index` and `PairedUmiAssigner::uses_index` and call
the former from `build_adjacency_graph_bitenc`. The condition now has a name, a
doc comment, and three rstest tables covering the threshold boundary, `--edits`
sensitivity, and the paired strategy's wider coverage -- so the help is checked
against behaviour instead of restating it.

The help also still read as though the threshold were a switch. It is a
minimum: the gate is `distinct >= threshold`, so `0` indexes every position
group rather than disabling the index, and no small value turns the index off --
that takes a threshold larger than any group. Anyone reaching for `0` to force a
linear scan, to isolate the index in a profile or to sidestep a suspected index
bug, gets maximum indexing instead. The help now says so outright.

Behaviour is unchanged; this is documentation plus the extraction needed to test
it.
@nh13
nh13 force-pushed the nh/fix-index-threshold-doc branch from 1bbb9fa to d6d655e Compare July 25, 2026 18:51
@nh13
nh13 temporarily deployed to github-actions July 25, 2026 18:52 — with GitHub Actions Inactive
@nh13 nh13 changed the title docs(group): correct inverted --index-threshold help, and pin its semantics feat(group): accept --index-threshold always|never, reject unsatisfiable requests Jul 25, 2026
@nh13
nh13 force-pushed the nh/fix-index-threshold-doc branch from d6d655e to b31d8e2 Compare July 25, 2026 19:11
@nh13
nh13 temporarily deployed to github-actions July 25, 2026 19:11 — with GitHub Actions Inactive
@nh13

nh13 commented Jul 25, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 25, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@nh13
nh13 force-pushed the nh/fix-index-threshold-doc branch from b31d8e2 to 6521aa1 Compare July 25, 2026 19:56
@nh13
nh13 temporarily deployed to github-actions July 25, 2026 19:56 — with GitHub Actions Inactive
@nh13

nh13 commented Jul 26, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 26, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@nh13

nh13 commented Jul 26, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 26, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (3)
src/lib/commands/group.rs (1)

482-501: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Parallel assigners ignore index_threshold entirely.

ParallelEditAssigner/ParallelAdjacencyAssigner/ParallelPairedAssigner take no threshold, so under --threads N with a parallel-eligible group --index-threshold always passes validation and is then discarded. validate_index_threshold can't see the per-group use_parallel decision, so at minimum say so in the --index-threshold help (Lines 861-868) — currently it reads as unconditional.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/commands/group.rs` around lines 482 - 501, The parallel branch of
create_umi_assigner drops index_threshold when constructing
ParallelEditAssigner, ParallelAdjacencyAssigner, and ParallelPairedAssigner.
Update the --index-threshold help text near its definition to state that the
setting is ignored for groups using parallel assigners, while preserving
existing validation and assignment behavior.
src/lib/commands/common.rs (1)

135-175: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Edit's log line will lie at --edits != 1 if the can_index gate stays.

The Adjacency arm correctly reports "not used" outside --edits 1; Edit unconditionally prints a floored number even though SimpleErrorUmiAssigner::assign never reaches the index at --edits != 1 (crates/fgumi-umi/src/assigner.rs Line 1290). Downstream of that root cause — no change needed here if the gate is dropped.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/commands/common.rs` around lines 135 - 175, Remove the `can_index`
gate that prevents the Edit strategy from reaching the index when effective
edits differ from 1, so `SimpleErrorUmiAssigner::assign` uses the index
consistently with `index_threshold_log_message`. Preserve the existing floored
threshold reporting and Adjacency-specific “not used” behavior.
crates/fgumi-umi/src/assigner.rs (1)

1285-1296: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

One root cause: SimpleErrorUmiAssigner::assign gates the whole defer/index branch on can_index = max_mismatches == 1, while everything else this PR adds says Edit indexes at every --edits value. Consequence: --index-threshold always --strategy edit --edits 2 passes validate_index_threshold and then silently scans — the quietly-ignored-flag failure IndexThreshold was introduced to eliminate. Pick one contract (drop the gate, or narrow Edit to edits == 1) and make these five sites agree.

  • crates/fgumi-umi/src/assigner.rs#L1285-L1296: drop can_index = self.max_mismatches == 1 (let it be true) so the index is reachable at every distance — NgramIndex::new already declines inputs it cannot partition — or keep it and narrow everything below.
  • crates/fgumi-umi/src/assigner.rs#L600-L617: if the gate stays, change Strategy::Edit | Strategy::Paired => true to give Edit its own edits == 1 arm.
  • crates/fgumi-umi/src/assigner.rs#L4852-L4884: drive test_edit_index_is_built_at_every_edit_distance through assign() instead of calling components_via_index directly, so the claim is actually exercised; likewise make test_edit_index_matches_scan's edits=2/3 cases non-vacuous.
  • src/lib/commands/common.rs#L135-L175: if the gate stays, give the Strategy::Edit arm the same "not used" branch the Adjacency arm has for effective_edits != 1, and update the pinned edit_two_edits/edit_zero_mismatches cases at Lines 1300-1311.
  • src/lib/commands/dedup.rs#L1123-L1132: qualify "Edit, Adjacency and Paired index" if Edit ends up gated to --edits 1.
  • src/lib/commands/group.rs#L861-L868: apply the identical help-text change (same string).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/fgumi-umi/src/assigner.rs` around lines 1285 - 1296, Make Edit
indexing reachable for every edit distance by removing the max_mismatches == 1
gate in SimpleErrorUmiAssigner::assign at
crates/fgumi-umi/src/assigner.rs:1285-1296; retain NgramIndex::new as the
capability check. Update the Strategy handling at
crates/fgumi-umi/src/assigner.rs:600-617 to preserve Edit indexing for all edit
values, and revise the tests at crates/fgumi-umi/src/assigner.rs:4852-4884 to
exercise assign() and make edits=2/3 cases non-vacuous. No direct changes are
required at src/lib/commands/common.rs:135-175,
src/lib/commands/dedup.rs:1123-1132, or src/lib/commands/group.rs:861-868
because the all-edit-distance Edit contract remains unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/test_dedup_command.rs`:
- Around line 531-533: The dedup integration test only verifies that an output
file exists, not that the never-threshold output preserves the default-threshold
result. Update the test around cmd.execute("fgumi dedup") to read the output BAM
and assert all 6 records with their expected MI grouping, or compare it against
a default-threshold run using the same input.

---

Outside diff comments:
In `@crates/fgumi-umi/src/assigner.rs`:
- Around line 1285-1296: Make Edit indexing reachable for every edit distance by
removing the max_mismatches == 1 gate in SimpleErrorUmiAssigner::assign at
crates/fgumi-umi/src/assigner.rs:1285-1296; retain NgramIndex::new as the
capability check. Update the Strategy handling at
crates/fgumi-umi/src/assigner.rs:600-617 to preserve Edit indexing for all edit
values, and revise the tests at crates/fgumi-umi/src/assigner.rs:4852-4884 to
exercise assign() and make edits=2/3 cases non-vacuous. No direct changes are
required at src/lib/commands/common.rs:135-175,
src/lib/commands/dedup.rs:1123-1132, or src/lib/commands/group.rs:861-868
because the all-edit-distance Edit contract remains unchanged.

In `@src/lib/commands/common.rs`:
- Around line 135-175: Remove the `can_index` gate that prevents the Edit
strategy from reaching the index when effective edits differ from 1, so
`SimpleErrorUmiAssigner::assign` uses the index consistently with
`index_threshold_log_message`. Preserve the existing floored threshold reporting
and Adjacency-specific “not used” behavior.

In `@src/lib/commands/group.rs`:
- Around line 482-501: The parallel branch of create_umi_assigner drops
index_threshold when constructing ParallelEditAssigner,
ParallelAdjacencyAssigner, and ParallelPairedAssigner. Update the
--index-threshold help text near its definition to state that the setting is
ignored for groups using parallel assigners, while preserving existing
validation and assignment behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 77fc4353-2d1b-4e94-9442-8292f699205f

📥 Commits

Reviewing files that changed from the base of the PR and between 1bbb9fa and cd4df0c.

📒 Files selected for processing (7)
  • crates/fgumi-umi/src/assigner.rs
  • crates/fgumi-umi/src/index_threshold.rs
  • crates/fgumi-umi/src/lib.rs
  • src/lib/commands/common.rs
  • src/lib/commands/dedup.rs
  • src/lib/commands/group.rs
  • tests/integration/test_dedup_command.rs

Comment thread tests/integration/test_dedup_command.rs Outdated
…ble requests (#636)

`--index-threshold` was a bare `usize` meaning "index once a position
group holds at least this many distinct UMIs". That left no way to turn
the index off: suppressing it needed a value larger than any group, i.e.
literally `--index-threshold 18446744073709551615`. The previous commit
spelled out that `0` enables the index for every group rather than
disabling it — but saying so plainly only made the missing capability
more obvious.

Make the option a keyword-or-number, matching `--parallel-group-min-templates`
one screen away in the same file:

    --index-threshold always     index every position group
    --index-threshold never      always scan all UMI pairs
    --index-threshold <N>        index groups of N or more (default 100)

`never` is the new capability. `always` is sugar for `0`, which already
meant this but read as its opposite — the trap that produced the wrong
docs in the first place. Every existing integer invocation keeps working
unchanged, `0` included, so no command line silently changes meaning.

Keeping this in one option rather than adding a `--no-index` flag makes
the contradictory state unrepresentable: two knobs controlling one
behaviour would need a precedence rule for `--no-index --index-threshold 50`.

`always` asserts that indexing will happen, so a configuration that can
never index is now a command-line error rather than a flag that is quietly
ignored:

    $ fgumi group --strategy identity --index-threshold always
    Error: --index-threshold always cannot be honoured with --strategy
    identity: the identity strategy never uses the UMI index. ...

    $ fgumi group --strategy adjacency --edits 2 --index-threshold always
    Error: --index-threshold always cannot be honoured with --strategy
    adjacency --edits 2: the adjacency strategy only indexes at --edits 1. ...

That second case surfaces an asymmetry the help had left implicit: the edit
and adjacency strategies silently ignore the index at any edit distance other
than 1, while paired indexes at all of them. `Strategy::can_use_index` now
states it, and the check runs against the EFFECTIVE strategy and edits so
`--no-umi` (which forces identity) is caught too.

A bare integer is deliberately not checked. It is a tuning knob allowed to
end up inert — otherwise the default `100` could not coexist with
`--strategy identity` — and that includes `0`, even though `0` admits every
group exactly as `always` does.

`Edit` grew an index of its own in #645, gated on its own measured crossover
(`EDIT_INDEX_THRESHOLD`, 200) rather than the shared default.
`IndexThreshold::floored_at` expresses that floor: a numeric threshold is
raised to the crossover, while the keywords pass through untouched, so
`always` still means every group — the escape hatch that isolates the index
in a profile — and `never` still means none. Every existing numeric
invocation keeps the `max(flag, 200)` behaviour it had.

What `Strategy::can_use_index` reports for `Edit` is `edits == 1`, the same as
adjacency. It is tempting to say otherwise: `components_via_index` hands
`max_mismatches` straight to `NgramIndex::new`, which partitions each UMI into
`max_mismatches + 1` pieces and pigeonholes over them, so the index is
*capable* at any distance — a new test pins that, brute-force, across five UMI
lengths and four distances. But `assign` never *reaches* it elsewhere: #645
gated the defer/index branch on one mismatch deliberately, because
`EDIT_INDEX_THRESHOLD`'s crossover was measured there and nowhere else. Past
it the index loses — the partitions get too short to be selective (4 bases at
k=1, 2 at k=2 for an 8-base UMI) while single linkage collapses the group into
one component, making the set-merge's early-exiting scan cheaper rather than
dearer. Widening the gate is a benchmarking question;
`benches/umi_assigner_threshold.rs` sweeps one mismatch only.

So both `edits == 1` gates now read from the assigner that enforces them —
`SimpleErrorUmiAssigner::indexes_at_edit_distance` and its adjacency twin —
rather than being restated in `can_use_index`, in the `Index threshold:`
startup line, and in the parity test's rationale. Restating them is how they
came to disagree in the first place, and the parity test is back to sweeping
one mismatch: at any other distance both sides run the identical set-merge, so
sweeping wider compared the scan against itself.

`dedup` carries its own copy of the option, so the validation lives in
`commands::common` and both call it. Assigner constructors take
`T: Into<IndexThreshold>`, leaving every existing integer call site
untouched.
@nh13
nh13 force-pushed the nh/fix-index-threshold-doc branch from cd4df0c to e6405f6 Compare July 26, 2026 02:49
@nh13
nh13 temporarily deployed to github-actions July 26, 2026 02:49 — with GitHub Actions Inactive
@nh13

nh13 commented Jul 26, 2026

Copy link
Copy Markdown
Member Author

All four findings from the last review are addressed in e6405f64. The three outside-diff ones have no thread to resolve, so recording them here.

crates/fgumi-umi/src/assigner.rs:1285-1296 — 🟠 the can_index gate contradicts everything this PR added about Edit. Confirmed, and fixed by narrowing the contract, not by dropping the gate.

You were right that five sites disagreed with one gate. Picking which side to keep took some digging, and the gate wins:

  • The gate is deliberate. perf(umi): index the edit assigner's neighbour search #645 was explicitly scoped to one mismatch: EDIT_INDEX_THRESHOLD's crossover table is labelled "index vs. scan at one mismatch", and the plan was to widen it only after adding a -e 2/-e 3 benchmark arm. That arm still does not exist — benches/umi_assigner_threshold.rs names its group umi_assigner/edit_1 and hardcodes 1 in every arm.
  • main already said so; this PR un-said it. Before the last force-push, index_threshold_log_message had a Strategy::Edit if effective_edits == 1 branch and an explicit "not used (edit indexes only at --edits 1)"; SimpleErrorUmiAssigner::uses_index ended in && self.max_mismatches == 1; the module doc said "At one mismatch (--edits 1)…". This PR replaced all three with the inverse. The reasoning error was reading NgramIndex's capability at k>1 as evidence that assign reaches it there.
  • Dropping the gate would be a performance regression, not just an unmeasured risk. At k≥2 on 8-base UMIs two effects compound against the index: partition_len collapses 4 → 2, so a bucket holds n/16 instead of n/256; and single linkage collapses the group to one component, so merge_umi_into_sets' set.iter().any(...) early-exits sooner and the scan gets cheaper (≈3,870 → 962 → 197 comparisons/UMI at k=1/2/3 for 12,967 distinct). Modelled index-to-scan work ratio: 0.03× at k=1, 2.4× at k=2, 15× at k=3. It stays a win at L ≥ 12, so the right shape is length-dependent — which is the argument against making it unconditional. PairedUmiAssigner already reflects this, switching to BkTree above k=1.

You also noted this is not a correctness question, and that is right: find_within re-verifies each candidate's Hamming distance, and pigeonhole gives no false negatives. That fact is now pinned rather than asserted — test_ngram_index_matches_brute_force compares the index's pair set against a brute-force one across five UMI lengths and four distances, including the lengths max_mismatches + 1 does not divide (partition_len is a floor, so an 8-base UMI at k=2 leaves the last two bases uncovered). Verified load-bearing: restricting find_within to one partition kills it at every k≥1. The lengths deliberately include ones where random UMIs have no neighbours, so neighbours at exactly distance k are planted and their absence is asserted — otherwise the recall half would pass while checking nothing.

So both edits == 1 gates now read from the assigner that enforces them — SimpleErrorUmiAssigner::indexes_at_edit_distance and its adjacency twin — instead of being restated in can_use_index, in the startup log line, and in a test rationale. Restating them is how they drifted; assign's can_index is now derived from the same predicate the CLI validates against.

Beyond the five sites you listed, four more needed the same correction. One is a bug in its own right: validate_index_threshold's reason match sent Strategy::Edit into _ => "the edit strategy never uses the UMI index", which is false — edit does index, just at --edits 1. The Adjacency arm is widened, so --strategy edit --edits 2 --index-threshold always now says:

--index-threshold always cannot be honoured with --strategy edit --edits 2: the edit strategy only indexes at --edits 1. Drop --index-threshold to leave indexing to the default threshold, or pass --index-threshold never to state that a linear scan is intended.

The others: the NgramIndex::new guard comment (under the gate, partition_len > 16 is unreachable in-crate — BitEnc caps at 32 bases so half is always ≤ 16 — while == 0 is live for a 1-base UMI); test_strategy_can_use_index's edit_zero_edits/edit_two_edits expectations; and a group.rs case that moved from "accepted" to "rejected". Both commit message and PR body are corrected too — they carried the same inverted claim.

src/lib/commands/common.rs:135-175 — 🟡 the Edit log line lies at --edits != 1. Fixed. Downstream of the above, as you said; since the gate stays, the Edit arm gets the "not used" branch Adjacency has, and the pinned edit_two_edits/edit_zero_mismatches cases now expect it. Added a case for never at two mismatches, where the edit-distance condition outranks the keyword.

src/lib/commands/group.rs:482-501 — 🟡 parallel assigners ignore index_threshold. Fixed in the help text. Confirmed: none of ParallelEditAssigner/ParallelAdjacencyAssigner/ParallelPairedAssigner consults an index at all — all three go through discover_edges_parallel_k, which enumerates neighbours and hash-looks-them-up. Two details worth naming, since both would have made the disclosure wrong:

  • It is conditional on --threads > 1. create_umi_assigner falls back to the sequential assigner at one thread regardless of use_parallel, so --threads 1 --allow-unmapped does honour the threshold.
  • Only group has a parallel path. dedup always builds via new_assigner_full, so its help gets the Edit-gate correction but not this caveat.

validate_index_threshold still cannot see the per-group use_parallel decision, so this stays documentation rather than validation — as you suggested.

@nh13

nh13 commented Jul 26, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 26, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@nh13

nh13 commented Jul 26, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 26, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@nh13

nh13 commented Jul 26, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 26, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@nh13

nh13 commented Jul 26, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 26, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@nh13

nh13 commented Jul 26, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 26, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@nh13

nh13 commented Jul 26, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 26, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@nh13

nh13 commented Jul 26, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 26, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@nh13

nh13 commented Jul 26, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 26, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@nh13
nh13 merged commit e10785b into main Jul 26, 2026
14 checks passed
@nh13
nh13 deleted the nh/fix-index-threshold-doc branch July 26, 2026 16:35

This branch was previously deployed

1 inactive deployment
github-actions — e6405f64 Deployed Jul 26, 2026 by nh13 via coverage #3094
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant