sketching files containing many small sequences: manysketch is astonishingly fast

[ctb]

I'm trying to sketch the RVDB, the Reference Viral Genome Database. The clustered file is ~600 MB.

sourmash scripts manysketch C-RVDBvCurrent.manysketch.csv -o C-RVDBvCurrent.manysketch.zip -p dna,k=21,scaled=1000 --singleton

took about 5 minutes.

sourmash sketch dna -p k=21 C-RVDBvCurrent.fasta.gz -o C-RVDBvCurrent.sig.zip --singleton

didn't finish in 24 hours.

what's the reason!? By my understanding manysketch isn't multithreaded when reading single FASTA files, so it's not multithreading. Presumably just the Python for loop penalty and/or using screed!? Wow.

On a mostly unrelated note, the sig.zip file is larger than the FASTA file. So that sucks.

[ctb]

and on a further somewhat unrelated note, fastgather took even less time than sketching.

[ctb]

and even more so, to add a sketch it is faster to

• add a sequence to the fa gz file
• rerun manysketch

than it is to run sig cat to combine the old database with a new sketch 😭

[lxsteiner]

Hi @ctb
How does this compare with #2537 ?

In case of a multifasta file, to work with manysketch, we'd first have to break down the entries into individual files, and compile a .csv list for the input, or what would the recommendation be?
Thanks.

[ctb]

Try using --singleton - that should do what you want :). See: link

[ctb]

singlesketch in sourmash_plugin_branchwater is even faster now ;)