Repository navigation
docs(zipper): recommend uncompressed BAM input for streaming pipelines - #362
Conversation
`zipper` accepts uncompressed BAM on stdin via the existing raw-byte path, which is faster end-to-end than piping SAM text: it skips ASCII formatting in the aligner and SAM parsing in zipper. Aligners like `bwa-mem3` support emitting uncompressed BAM directly (`--bam=0`) without going through a SAM-text intermediate. Update the `--input` CLI help and `long_about` to recommend uncompressed BAM for streaming pipelines while keeping a SAM example for aligners that can't emit BAM. Replace the stdin-BAM `warn!` with a neutral `info!` mirroring the existing stdin-SAM info log, and drop the file-BAM warn entirely (the SAM-file path doesn't log either, so this restores symmetry). Apply the same recommendation flip to the getting-started, best-practices, performance-tuning, and migration-from-fgbio guides, and remove the integration-test assertion on the dropped warn text. No behavior change.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (7)
💤 Files with no reviewable changes (1)
📝 WalkthroughWalkthroughThis PR updates 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #362 +/- ##
==========================================
- Coverage 90.97% 90.97% -0.01%
==========================================
Files 78 78
Lines 51017 51016 -1
==========================================
- Hits 46412 46410 -2
- Misses 4605 4606 +1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Summary
fgumi zipperaccepts uncompressed BAM on stdin via the existing raw-byte path, which is faster end-to-end than piping SAM text — it skips ASCII formatting in the aligner and SAM parsing in zipper. The prior recommendation in the CLI help,long_about, twowarn!calls, and four guide pages predates aligners that can emit uncompressed BAM directly.bwa-mem3supports--bam[=N]withN=0(uncompressed BGZF, no DEFLATE) and buildsbam1_trecords directly frommem_aln_twithout going through a SAM-text intermediate, so the recommended pipeline shape now is uncompressed BAM end-to-end.This PR flips the recommendation:
--inputCLI help and thelong_aboutexample block to recommendbwa-mem3 mem --bam=0for streaming pipelines, keeping a SAM example for aligners that can't emit BAM.warn!with a neutralinfo!("Reading BAM from stdin")mirroring the existing stdin-SAM info log; drops the file-BAMwarn!entirely (the SAM-file path doesn't log either, so this restores symmetry).docs/src/guide/{getting-started,best-practices,performance-tuning,migration-from-fgbio}.md.## [Unreleased]### DocumentationCHANGELOG entry.No behavior change.
Why uncompressed BAM beats SAM text on a pipe
Both formats end up at roughly the same on-pipe byte count (BGZF level 0 adds ~26 bytes of framing per ~64 KiB block — header 18 + trailer 8). The win is on the CPU side at both ends:
bwa-mem3buildsbam1_tdirectly (mem_aln_to_bam, no SAM-text intermediate).TemplateIterator(MappedReader::StdinBam→RawBamReader). The SAM path goes through noodlesRecordBufdecode +RecordBufEncoderre-encode (record_bufs_to_templatesinsrc/lib/commands/zipper.rs), which is meaningful CPU per record.Compressed BAM on a pipe (
--bam=1+) is still discouraged in the new copy — it burns DEFLATE CPU on the writer and inflate on the reader for data the downstream sort step recompresses anyway.Test plan
cargo ci-fmtcargo ci-lintcargo nextest run -E 'test(zipper)'— 75/75 zipper tests pass, includingtest_zipper_bam_mapped_input(file BAM input) andtest_zipper_bam_stdin_input(stdin BAM auto-detect)cargo run -- zipper --helpNotes for reviewers
fgumi fastq'slong_aboutexamples on plainbwa mem— those are aligner-agnostic illustrations and inverting them feels out of scope for a zipper-docs PR. Happy to do a follow-up if the team wants consistency.long_aboutblock is slightly longer than sibling commands' help text (e.g.fastq,sort) — included a three-sentence rationale paragraph because zipper is the most pipeline-shape-sensitive command. Will trim if reviewers prefer a terser feel.