Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
166 changes: 166 additions & 0 deletions CHANGELOG.md

Large diffs are not rendered by default.

356 changes: 179 additions & 177 deletions Cargo.lock

Large diffs are not rendered by default.

34 changes: 17 additions & 17 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ members = [".", "crates/fgumi-raw-bam", "crates/fgumi-dna", "crates/fgumi-bgzf",
resolver = "2"

[workspace.package]
version = "0.7.0"
version = "0.8.0"
edition = "2024"
rust-version = "1.93.0"
repository = "https://github.com/fulcrumgenomics/fgumi"
Expand All @@ -20,22 +20,22 @@ license = "MIT"
flate2 = { version = "1.1", features = ["zlib-rs"] }

# Internal crates. Versions are bumped in lockstep by release-plz.
fgumi-bam-io = { version = "0.7.0", path = "crates/fgumi-bam-io" }
fgumi-bgzf = { version = "0.7.0", path = "crates/fgumi-bgzf" }
fgumi-cli-common = { version = "0.7.0", path = "crates/fgumi-cli-common" }
fgumi-cli-macros = { version = "0.7.0", path = "crates/fgumi-cli-macros" }
fgumi-fmt = { version = "0.7.0", path = "crates/fgumi-fmt" }
fgumi-consensus = { version = "0.7.0", path = "crates/fgumi-consensus", default-features = false }
fgumi-dna = { version = "0.7.0", path = "crates/fgumi-dna" }
fgumi-metrics = { version = "0.7.0", path = "crates/fgumi-metrics" }
fgumi-pipeline-core = { version = "0.7.0", path = "crates/fgumi-pipeline-core" }
fgumi-pipeline-io = { version = "0.7.0", path = "crates/fgumi-pipeline-io" }
fgumi-raw-bam = { version = "0.7.0", path = "crates/fgumi-raw-bam" }
fgumi-sam = { version = "0.7.0", path = "crates/fgumi-sam" }
fgumi-simd-fastq = { version = "0.7.0", path = "crates/fgumi-simd-fastq" }
fgumi-sort = { version = "0.7.0", path = "crates/fgumi-sort" }
fgumi-tag = { version = "0.7.0", path = "crates/fgumi-tag" }
fgumi-umi = { version = "0.7.0", path = "crates/fgumi-umi" }
fgumi-bam-io = { version = "0.8.0", path = "crates/fgumi-bam-io" }
fgumi-bgzf = { version = "0.8.0", path = "crates/fgumi-bgzf" }
fgumi-cli-common = { version = "0.8.0", path = "crates/fgumi-cli-common" }
fgumi-cli-macros = { version = "0.8.0", path = "crates/fgumi-cli-macros" }
fgumi-fmt = { version = "0.8.0", path = "crates/fgumi-fmt" }
fgumi-consensus = { version = "0.8.0", path = "crates/fgumi-consensus", default-features = false }
fgumi-dna = { version = "0.8.0", path = "crates/fgumi-dna" }
fgumi-metrics = { version = "0.8.0", path = "crates/fgumi-metrics" }
fgumi-pipeline-core = { version = "0.8.0", path = "crates/fgumi-pipeline-core" }
fgumi-pipeline-io = { version = "0.8.0", path = "crates/fgumi-pipeline-io" }
fgumi-raw-bam = { version = "0.8.0", path = "crates/fgumi-raw-bam" }
fgumi-sam = { version = "0.8.0", path = "crates/fgumi-sam" }
fgumi-simd-fastq = { version = "0.8.0", path = "crates/fgumi-simd-fastq" }
fgumi-sort = { version = "0.8.0", path = "crates/fgumi-sort" }
fgumi-tag = { version = "0.8.0", path = "crates/fgumi-tag" }
fgumi-umi = { version = "0.8.0", path = "crates/fgumi-umi" }

# Third-party crates shared by two or more members. Declared once here so a member can
# never drift onto a different version: two versions of a crate whose types cross crate
Expand Down
27 changes: 27 additions & 0 deletions crates/fgumi-bam-io/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,33 @@ All notable changes to this project will be documented in this file.

## [Unreleased]

## [0.8.0] - 2026-10-08

### Bug Fixes

- Point empty stdin/pipe input at the upstream stage ([#866](https://github.com/fulcrumgenomics/fgumi/pull/866))
- Return an error instead of panicking on >65,535 libraries ([#956](https://github.com/fulcrumgenomics/fgumi/pull/956))
- Add one @PG per command instead of one per chain leaf ([#1023](https://github.com/fulcrumgenomics/fgumi/pull/1023))
- Propagate errors writing the final BGZF EOF block ([#1046](https://github.com/fulcrumgenomics/fgumi/pull/1046))

### Features

- Typed-step pipeline foundation ([#870](https://github.com/fulcrumgenomics/fgumi/pull/870))
- Inline BAI indexer on the arena sink ([#883](https://github.com/fulcrumgenomics/fgumi/pull/883))
- Restore --read-streams on the arena chain via a concurrent scatter reader ([#889](https://github.com/fulcrumgenomics/fgumi/pull/889))
- Add the bam-roundtrip subcommand ([#936](https://github.com/fulcrumgenomics/fgumi/pull/936))

### Performance

- Drop the discarded decode key and add skippable input CRC ([#960](https://github.com/fulcrumgenomics/fgumi/pull/960))
- Fold UMI-position cache into the group-key aux scan ([#976](https://github.com/fulcrumgenomics/fgumi/pull/976))

### Refactor

- [**breaking**] Remove the legacy unified_pipeline engine ([#947](https://github.com/fulcrumgenomics/fgumi/pull/947))

<!-- generated by git-cliff -->

## [0.7.0] - 2026-08-24

### Bug Fixes
Expand Down
13 changes: 13 additions & 0 deletions crates/fgumi-bgzf/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,19 @@

All notable changes to this project will be documented in this file.

## [0.8.0] - 2026-10-08

### Features

- Typed-step pipeline foundation ([#870](https://github.com/fulcrumgenomics/fgumi/pull/870))
- Add --check-crc / --no-check-crc with the file-vs-stdin default ([#933](https://github.com/fulcrumgenomics/fgumi/pull/933))

### Refactor

- [**breaking**] Remove the legacy unified_pipeline engine ([#947](https://github.com/fulcrumgenomics/fgumi/pull/947))

<!-- generated by git-cliff -->

## [0.7.0] - 2026-08-24

### Features
Expand Down
7 changes: 7 additions & 0 deletions crates/fgumi-cli-common/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Changelog

All notable changes to this project will be documented in this file.

## [0.8.0] - 2026-10-08

<!-- generated by git-cliff -->
7 changes: 7 additions & 0 deletions crates/fgumi-cli-macros/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Changelog

All notable changes to this project will be documented in this file.

## [0.8.0] - 2026-10-08

<!-- generated by git-cliff -->
31 changes: 31 additions & 0 deletions crates/fgumi-consensus/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,37 @@

All notable changes to this project will be documented in this file.

## [0.8.0] - 2026-10-08

### Bug Fixes

- [**breaking**] Reject duplex/codec flags the callers never honor ([#997](https://github.com/fulcrumgenomics/fgumi/pull/997))
- `fgumi duplex --min-consensus-base-quality`, `fgumi codec --min-consensus-base-quality`, `fgumi codec --trim` and their `runall` forms are removed. `ConsensusCallingOptions` loses `trim` and `min_consensus_base_quality` (now in `QualityTrimOptions` / `MinConsensusBaseQualityOptions`); `DuplexOptions` loses `min_consensus_base_quality`; `CodecOptions` and
- [**breaking**] Give each methylation consensus caller one SEQ convention and make filters, simulate and docs follow it ([#1010](https://github.com/fulcrumgenomics/fgumi/pull/1010))
- methylation::query_to_ref_positions takes the CIGAR as aligned (with its clips) and no longer takes the simplified CIGAR. * fix(consensus)!: emit observed bases from simplex methylation consensus Simplex EM-seq/TAPs consensus rewrote converted bases in the source reads (T->C / A->G at reference cytosines) before calling, so the emitted SEQ reported every site as unconverted. SEQ-based methylation callers and bisulfite-aware aligners read methylation from SEQ and saw every cytosine as methylated. The rewrite also hid real disagreements: all reads of a single-strand family copy one converted strand, so a C/T split is an error (or a UMI collision) and must lower quality like any other. Consensus is now always called on the observed bases, and simplex emits them as they are, with the evidence in cu/ct. MM/ML are no longer emitted for simplex: MM can only describe bases present in SEQ, and a converted cytosine is not one. A simplex consensus also cannot tell a converted cytosine from a C>T mutation, so restoring it would assert a base the molecule may not have. The shared MM builder now writes the SAM-spec '?' flag (unlisted bases are unknown), takes the conversion pattern from the read type (R1 C->T, R2 G->A) instead of the alignment strand, encodes each ML probability p as floor(256 p) capped at 255 (the SAM spec's [N/256, (N+1)/256) bins; it used floor(255 p), which put 1/2 and 3/4 in the bin below), and MN is available as a tag constant.
- Keep MM/ML in step with hard-clipped and unmapped reads ([#1012](https://github.com/fulcrumgenomics/fgumi/pull/1012))
- Classify dovetail FR pairs with coincident 5' ends as FR ([#1022](https://github.com/fulcrumgenomics/fgumi/pull/1022))
- Decide read-level rejection before masking, and short-circuit templates like fgbio ([#1028](https://github.com/fulcrumgenomics/fgumi/pull/1028))
- Handle NaN likelihoods like fgbio ([#1029](https://github.com/fulcrumgenomics/fgumi/pull/1029))
- Truncate source-read cigars before converting hard clips, like fgbio ([#1042](https://github.com/fulcrumgenomics/fgumi/pull/1042))

### Features

- Typed-step pipeline foundation ([#870](https://github.com/fulcrumgenomics/fgumi/pull/870))

### Performance

- Use fixed-seed ahash on hot per-item maps to remove RandomState global-counter contention ([#865](https://github.com/fulcrumgenomics/fgumi/pull/865))
- Single-pass scalar consensus-tag extraction ([#969](https://github.com/fulcrumgenomics/fgumi/pull/969))
- Gated multi-base fast path for simplex consensus ([#973](https://github.com/fulcrumgenomics/fgumi/pull/973))

### Testing

- Speed up the slow-test tail without weakening invariants ([#881](https://github.com/fulcrumgenomics/fgumi/pull/881))
- Pin read-order independence of alignment grouping ([#1037](https://github.com/fulcrumgenomics/fgumi/pull/1037))

<!-- generated by git-cliff -->

## [0.7.0] - 2026-08-24

### Bug Fixes
Expand Down
8 changes: 8 additions & 0 deletions crates/fgumi-dna/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@

All notable changes to this project will be documented in this file.

## [0.8.0] - 2026-10-08

### Performance

- Use fixed-seed ahash on hot per-item maps to remove RandomState global-counter contention ([#865](https://github.com/fulcrumgenomics/fgumi/pull/865))

<!-- generated by git-cliff -->

## [0.7.0] - 2026-08-24

<!-- generated by git-cliff -->
Expand Down
7 changes: 7 additions & 0 deletions crates/fgumi-fmt/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Changelog

All notable changes to this project will be documented in this file.

## [0.8.0] - 2026-10-08

<!-- generated by git-cliff -->
19 changes: 19 additions & 0 deletions crates/fgumi-metrics/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,25 @@

All notable changes to this project will be documented in this file.

## [0.8.0] - 2026-10-08

### Documentation

- Stop calling the duplication ladder a saturation curve ([#980](https://github.com/fulcrumgenomics/fgumi/pull/980))

### Features

- Produce --metrics on the multi-threaded chain path ([#915](https://github.com/fulcrumgenomics/fgumi/pull/915))
- Compute consensus QC metrics inline in fused runall and standalone consensus ([#948](https://github.com/fulcrumgenomics/fgumi/pull/948))
- [**breaking**] Publish metric column manifest; headered filter --stats ([#979](https://github.com/fulcrumgenomics/fgumi/pull/979))
- `fgumi filter --stats` now writes a headered fgbio-Metric TSV (a total_reads/passed_reads/failed_reads/pass_rate header row plus one data row) instead of the previous headerless two-column key/value layout. Column names and semantics are unchanged; downstream parsers of the old vertical format must be updated. * test(metrics): round-trip conformance tests for all emitters * feat(metrics): publish metric column manifest with contract tests Adds crates/fgumi-metrics/metric_columns.json, the ordered column contract for every metric file fgumi emits, keyed <namespace>.<file> because group, simplex, duplex and dedup each write a differently shaped *.family_sizes.txt. tests/integration/test_metric_contract.rs parses the workspace with syn and fails in three cases: the manifest drifts from the live structs; a struct deriving serde's Serialize is neither listed nor explicitly allowlisted; or a serialized f64 field lacks the fgbio float encoding (Infinity/NaN rather than inf). downsample's --histogram-kept/--histogram-rejected were hand-formatted with writeln!. They now serialize a DownsampleHistogramMetric through the shared writer, so they are covered by the contract. The output is byte-identical. * fix(metrics): write headers for empty group and dedup histogram outputs group (--family-size-histogram, and the --metrics family- and position-group-size histograms) and dedup (--metrics, --family-size-histogram, --duplication-ladder) wrote through a bare DelimFile, which emits the header lazily. An empty histogram or ladder was therefore a 0-byte file that fgbio's Metric.read rejects ("No header found"). Route them through the shared write_metrics, which always writes the header. * docs(changelog): note filter --stats format break and metrics fixes * fix(metrics): honor .gz, special-file and symlink destinations in write_metrics write_metrics wrote rows to an extension-less temp file and renamed it into place. That had three problems. A .gz destination received plain text under a .gz name, which the metrics reader then failed to decompress. A non-regular destination such as /dev/stdout, a FIFO or a >(...) process substitution failed at rename time, after the whole run. A symlinked destination was replaced by a regular file. The temp file now carries the destination's extension so compression follows it. Non-regular destinations are written in place. Symlinks are resolved before the atomic rename. Empty outputs derive their header from a plain scratch file, so the gzip path keeps its header too.

### Testing

- Fix inconsistent mate-strand flags in duplicate-group fixtures ([#908](https://github.com/fulcrumgenomics/fgumi/pull/908))

<!-- generated by git-cliff -->

## [0.7.0] - 2026-08-24

### Bug Fixes
Expand Down
26 changes: 26 additions & 0 deletions crates/fgumi-pipeline-core/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Changelog

All notable changes to this project will be documented in this file.

## [0.8.0] - 2026-10-08

### Features

- Wire --pipeline-trace through the chain builder ([#937](https://github.com/fulcrumgenomics/fgumi/pull/937))
- Thread-utilization and per-step bandwidth telemetry for the chain-builder engine ([#951](https://github.com/fulcrumgenomics/fgumi/pull/951))
- Park idle pool workers to fix the thread-oversubscription cliff ([#955](https://github.com/fulcrumgenomics/fgumi/pull/955))
- Add a refill-aware drain-first scheduler ([#986](https://github.com/fulcrumgenomics/fgumi/pull/986))

### Performance

- Don't fuse single-thread chains that declare Detached steps ([#884](https://github.com/fulcrumgenomics/fgumi/pull/884))

### Refactor

- [**breaking**] Remove the legacy unified_pipeline engine ([#947](https://github.com/fulcrumgenomics/fgumi/pull/947))

### Testing

- Read drained state once in pair-summer test steps ([#981](https://github.com/fulcrumgenomics/fgumi/pull/981))

<!-- generated by git-cliff -->
39 changes: 39 additions & 0 deletions crates/fgumi-pipeline-io/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Changelog

All notable changes to this project will be documented in this file.

## [0.8.0] - 2026-10-08

### Bug Fixes

- [**breaking**] Enforce --max-temp-files on the arena spill path ([#993](https://github.com/fulcrumgenomics/fgumi/pull/993))
- fgumi-pipeline-io's SpillBlockEvent::Block gains a required key_kind field, and SpillWrite now emits SpillReady for its surviving runs at AllAnnounced (with slot_count set to their number) instead of as each run closes. RunMergerDyn::step takes a byte budget.
- Leave no output files behind when a chain fails to build ([#1006](https://github.com/fulcrumgenomics/fgumi/pull/1006))

### Features

- Typed-step pipeline foundation ([#870](https://github.com/fulcrumgenomics/fgumi/pull/870))
- Inline BAI indexer on the arena sink ([#883](https://github.com/fulcrumgenomics/fgumi/pull/883))
- Route the sort command onto the declarative chain builder ([#885](https://github.com/fulcrumgenomics/fgumi/pull/885))
- Restore --read-streams on the arena chain via a concurrent scatter reader ([#889](https://github.com/fulcrumgenomics/fgumi/pull/889))
- Thread --sort-stats diagnostics through the chain ([#913](https://github.com/fulcrumgenomics/fgumi/pull/913))
- Add --check-crc / --no-check-crc with the file-vs-stdin default ([#933](https://github.com/fulcrumgenomics/fgumi/pull/933))
- Expose --block-batch and --file-granularity as hidden flags ([#943](https://github.com/fulcrumgenomics/fgumi/pull/943))
- Thread-utilization and per-step bandwidth telemetry for the chain-builder engine ([#951](https://github.com/fulcrumgenomics/fgumi/pull/951))

### Performance

- Don't fuse single-thread chains that declare Detached steps ([#884](https://github.com/fulcrumgenomics/fgumi/pull/884))
- Extend runs in the arena spill path (t16 parity) ([#886](https://github.com/fulcrumgenomics/fgumi/pull/886))
- Parallel gather for the in-memory sort fast path ([#966](https://github.com/fulcrumgenomics/fgumi/pull/966))

### Refactor

- [**breaking**] Remove the legacy unified_pipeline engine ([#947](https://github.com/fulcrumgenomics/fgumi/pull/947))
- Stop configuring the chain sort through RawExternalSorter ([#999](https://github.com/fulcrumgenomics/fgumi/pull/999))

### Testing

- Speed up the slow-test tail without weakening invariants ([#881](https://github.com/fulcrumgenomics/fgumi/pull/881))

<!-- generated by git-cliff -->
23 changes: 23 additions & 0 deletions crates/fgumi-raw-bam/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,29 @@

All notable changes to this project will be documented in this file.

## [0.8.0] - 2026-10-08

### Bug Fixes

- Replace deprecated wide swizzle_relaxed with shuffle ([#992](https://github.com/fulcrumgenomics/fgumi/pull/992))
- [**breaking**] Give each methylation consensus caller one SEQ convention and make filters, simulate and docs follow it ([#1010](https://github.com/fulcrumgenomics/fgumi/pull/1010))
- methylation::query_to_ref_positions takes the CIGAR as aligned (with its clips) and no longer takes the simplified CIGAR. * fix(consensus)!: emit observed bases from simplex methylation consensus Simplex EM-seq/TAPs consensus rewrote converted bases in the source reads (T->C / A->G at reference cytosines) before calling, so the emitted SEQ reported every site as unconverted. SEQ-based methylation callers and bisulfite-aware aligners read methylation from SEQ and saw every cytosine as methylated. The rewrite also hid real disagreements: all reads of a single-strand family copy one converted strand, so a C/T split is an error (or a UMI collision) and must lower quality like any other. Consensus is now always called on the observed bases, and simplex emits them as they are, with the evidence in cu/ct. MM/ML are no longer emitted for simplex: MM can only describe bases present in SEQ, and a converted cytosine is not one. A simplex consensus also cannot tell a converted cytosine from a C>T mutation, so restoring it would assert a base the molecule may not have. The shared MM builder now writes the SAM-spec '?' flag (unlisted bases are unknown), takes the conversion pattern from the read type (R1 C->T, R2 G->A) instead of the alignment strand, encodes each ML probability p as floor(256 p) capped at 255 (the SAM spec's [N/256, (N+1)/256) bins; it used floor(255 p), which put 1/2 and 3/4 in the bin below), and MN is available as a tag constant.
- Classify dovetail FR pairs with coincident 5' ends as FR ([#1022](https://github.com/fulcrumgenomics/fgumi/pull/1022))
- Truncate source-read cigars before converting hard clips, like fgbio ([#1042](https://github.com/fulcrumgenomics/fgumi/pull/1042))

### Features

- Typed-step pipeline foundation ([#870](https://github.com/fulcrumgenomics/fgumi/pull/870))
- Declarative chain-builder layer + group --threads pilot ([#872](https://github.com/fulcrumgenomics/fgumi/pull/872))

### Performance

- Single-pass aux-tag rebuild (RawTagsEditor::rebuild_with) for zipper merge and NM/UQ/MD strip ([#962](https://github.com/fulcrumgenomics/fgumi/pull/962))
- Fold per-record PG/AS/XS aux scans into single-pass walks ([#964](https://github.com/fulcrumgenomics/fgumi/pull/964))
- Reduce serial-merge CPU and allocations at 1-2 threads ([#971](https://github.com/fulcrumgenomics/fgumi/pull/971))

<!-- generated by git-cliff -->

## [0.7.0] - 2026-08-24

### Bug Fixes
Expand Down
Loading
Loading