Speed up read compression and reduce disk load

[fungs]

Hi,

I see that you have already incorporated pigz, if available. You might also consider compressing reads while they are calculated using a pipe. That will also save you disk space and double disk I/O (saving plain reads, reading plain reads, saving compressed reads).

Best,
Johannes

[fungs]

Basically all final is uncompressed as well. For light intermediate compression which does not cost CPU but saves disk I/O you can, for instance, use LZ4.

[fungs]

LZ4 and snappy are still good candidates and it would be great to write huge text-based output files in compressed (de-facto standard) format.

[asl]

Well, the standard de-facto is just plain text FASTA files. The majority of tools will not even accept GZIP-compressed FASTA's, so this would create obvious problems for downstream analysis, as the results will still be needed to be decompressed.

[fungs]

You can say different but 95% of software processing large FASTA files I have used in the past 10 years either support gzipped files directly or support reading from stdin, which allows decompressing on-the-fly. Raw reads are also typically compressed and gzip compression is already built into formats such as SAM.

You don't have to make output compression mandatory, if you want to support software which does not support it.

The overall reason to make use of compression, be it for intermediate (LZ4, snappy) or output (gzip), is to reduce the required disk storage to an estimated size of 10 to 20%. Depending on which system you run the software on, this can make a huge difference.

[asl]

Well, intermediates are already either compressed by default (reads after error correction) or binary (information between different assembly steps). Unfortunately, assembly results cannot be streamed to stdout simply because there are multiple outputs and there is no single unified output. But we might consider optional compression of the output, yes.