Very slow download of dehydrated .zip file

[Jtrachsel]

Hello,

I am downloading a large collection of bacterial genomes using the datasets download genome accession --dehydrated ... technique.
The initial download of the dehydrated .zip file is proceeding very slowly ~ 3.0kB/s. Is this rate expected? Is there anything I can do to speed this process up?

Once the .zip file is downloaded the rehydration proceeds very quickly.

Thanks

[ericcox1]

Hi @Jtrachsel,

Thanks for opening this issue.

We are investigating the slow download speed and I will comment on this thread with any updates.

Best,
Eric

[peterlaurin]

Was this ever resolved? I'm experiencing the same issue, but in addition it's throwing out the following error:

datasets download genome accession $genome_accession --dehydrated --annotated --assembly-source 'RefSeq'

> Collecting 186 genome records [================================================] 100% 186/186
> Downloading: ncbi_dataset.zip    18.9kB 170B/s
> Error: Download error: stream error: stream ID 5; INTERNAL_ERROR; received from peer
>
> Use datasets download genome accession <command> --help for detailed help about a command.

Thanks for your help!

[ericcox1]

Thank you for your comment, @peterlaurin. We have identified the problem and are working on a fix.

-Eric

[valery-shap]

I have the same issue.

[ericcox1]

Hi @valery-shap,

Thanks for your report. We are still working on a fix and hope to get this out soon.

Best,
Eric

[ericcox1]

We believe that the slow download reported here as well as the errors reported in #455 both stem from the same problem related to slow retrieval of md5 checksums. We are continuing to work on a fix and hope to have this resolved sometime next week.

Best,
Eric