Ivan/http transport upgrades #1732
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
feat: add multistream http download * Add a feature to split http download into multiple chunks. Chunks are downloaded into temporary files, which then get merged into one; * Downnloads over libp2p are always done in one chunk only; * Add NChunks configuration parameter; * Fix tests to handle not just open-ended range requests.
* add configuration option to disable http download from private IPs * don't follow http redirects --------- Co-authored-by: Anton Evangelatov <[email protected]>
fix: add multistream performance test verifies that multistream download is faster than single stream by introducing artificial tcp-level latency
Add control file * Control file captures the number of chunks that download has been started with and ensures that its the same across restarts
dirkmc
approved these changes
Oct 3, 2023
| // this test runs a tcp proxy in front of the http web server that adds a latency to each tcp packet. | ||
| // This is done to simulate network latency and ensure that multistream transfer is reasonably faster than singlestream. | ||
| // rawSize and latency need to be big enough so that data transfer takes a reasonable proportion of time comparing to writing to the disk. | ||
| ctx := context.Background() |
There was a problem hiding this comment.
I'd suggest we add t.Skip() at the top of this test as we will only use it occasionally.
There was a problem hiding this comment.
My idea was to have that test running for each build so that we know that nothing occasionally broke concurrent transfers. For example if the code gets changed to download chunks serially, then all existing tests would just pass while the functionality would be broken. WDYT?
There was a problem hiding this comment.
Ok seems fair, let's leave it as-is 👍
nonsense
approved these changes
Oct 3, 2023
LexLuthr
approved these changes
Oct 3, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This branch has a series of reviewed PRs merged into it. The code introduces the following changes:
NChunksproperty that defines the number of concurrent streams for each http download with the minimum of 1, maximum of 15 and the default of 5. That will speed up how fast car files get downloaded up to the factor of theNChunksvalue. Libp2p over http is always defaulted to 1 stream only;Multistream downloads have been tested on the main net. With
NChunks=5the overall download speed averaged at 448MBps, while withNChunks=1at 169MBps (approx 2.6x improvement). It's expected that the speed-up should be proportional to theNChunksvalue if enough bandwidth / IOPs is available as the chunks are downloaded concurrently without blocking each other.After download is complete chunks need to be glued up into the main output file. In our testing that process took approx. 4 minutes to complete for a 32GB car file on HDD. We expect that to be much faster for SSDs. However, even with 4 minutes of extra overhead the performance boost of parallelising HTTP download is still significant.
Fixes #1593