This repository has been archived by the owner on May 12, 2021. It is now read-only.

Consider multi-threading for some steps #64

Open

jpmckinney opened this issue Oct 30, 2020 · 0 comments

Labels

storage

Member

jpmckinney commented Oct 30, 2020

I think maybe only LZ4 compression, to compress different crawls on different cores.

I think the S3 step is network-bound, so won't benefit from multi-threading.

The first step of selecting crawls is I/O bound, so won't benefit from multi-threading.

jpmckinney added the storage label

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.