How to make “as greedy as possible” client? Ow how to make high-speed streaming?

[AskAlexSharov]

Which download everything as fast as it can

[AskAlexSharov]

I found something #150 (comment)

[anacrolix]

The current way to do this would be set a very large MaxUnverifiedBytes for the Client and Torrent (experiment with this for performance reasons), and call Torrent.DownloadAll. This should cause the client to download as much as it can from all connections without overlap until endgame mode.

[AskAlexSharov]

but MaxUnverifiedBytes=MaxInt - behave same as default value MaxUnverifiedBytes=0 - means unlimited.
no?

[anacrolix]

Yes you are right. I forgot that unlimited was the default. There was an issue recently where someone had performance issues with that for very large torrents.

[AskAlexSharov]

Just FYI: we have 200gb of torrents (80 small files, and 20 large). We downloading by internet (not by private network). For now - most greedy way we found is "download files 1-by-1" sequentially. (we didn't experiment yet with io.Copy(io.Discard, t.NewReader()) and with MaxUnverifiedBytes).

[anacrolix]

You'll have to clarify what your goal is: Do you want to get all the data as quickly as possible with no care for what order it arrives in, or do you want to stream the entire torrent, file-by-file? Could you maybe explain what you're trying to do, those use-cases are quite different.

[AskAlexSharov]

Goal:

• Ethereum came back to you :-) We working on alternative to go-ethereum - project https://github.com/ledgerwatch/erigon
• Using Bittorrent to sync historical data - not new hot data, but old and immutable.
• Main goal: download everything as fast as possible (Seeders have upload ratelimiter) - and just start using files locally (and seed them). 200Gb or 500Gb - depends on network: ~80 files < 1gb, ~20 files > 5gb, maximum 40gb file.
• And use some seedboxes providers to provide cheap data availability.
• I believe streaming feature - may be great for us: to reduce amount of random read/writes - and make us more slow-disk-friendly. Because we doing some heavy data processing and seed torrents in same time - better reduce impact of torrent to disk. I already see very low RAM usage in anacrolix/torrent - great. (having less goroutines is also helpful).

Problem:

• When we download all torrents in same time - we see download rate 2Mb/s (probably because of too much concurrency and requests prioritization). When download 1-by-1 - can see >50Mb/s (depends on seeders availability). I think it's not related to torrent size, but related to amount of torrents. Can provide some pprof picture if need.
• Other observation on seedbox.io (they using rTorrent): if I have 2 seedboxes - they almost never see each other - because there is limit on peers amount (and no way to configure it). They seeding to geo-locally far peers with 300kb/s. But if stop all torrents and start 1: seedboxes see each other and can sync 100mb/s. So, amount of torrent files may lead to weird results: some files will never sync, seed speed is very low, all reads/writes are random, etc... Strategy "download 1-by-1" may also help in this case.

[anacrolix]

Goal:

• Ethereum came back to you :-) We working on alternative to go-ethereum - project https://github.com/ledgerwatch/erigon
• Using Bittorrent to sync historical data - not new hot data, but old and immutable.
• Main goal: download everything as fast as possible (Seeders have upload ratelimiter) - and just start using files locally (and seed them). 200Gb or 500Gb - depends on network: ~80 files < 1gb, ~20 files > 5gb, maximum 40gb file.

If you want to seed files you've obtained, but also use the data in its file-orientation, and you have sufficient disk space, then a file-based storage implementation would probably suit best, I expect you've already done this (it's also the default). Definitely downloading as fast as possible means minimizing overlap and having the flattest possible prioritization structure. If you want some files to finish sooner, or complete whole files before moving on, that would be a slightly different case. Similarly if you're storing on something like S3 or whatever, you can seed from that, but you will want to stream or batch the data from local storage.

• And use some seedboxes providers to provide cheap data availability.
• I believe streaming feature - may be great for us: to reduce amount of random read/writes - and make us more slow-disk-friendly. Because we doing some heavy data processing and seed torrents in same time - better reduce impact of torrent to disk. I already see very low RAM usage in anacrolix/torrent - great. (having less goroutines is also helpful).

This might be a bit different: You would want to use a memory-based storage implementation, and then stream out the data to truly mazimize disk write speed. That also makes seeding difficult though, since you'd want to seed from the files and probably not from memory as you wouldn't want to keep everything in memory. There's no write cache in anacrolix/torrent yet, that would help with ordering writes to disk. Also you would make use of a smaller "unverified bytes" window in that case, you wouldn't want to readahead more than the size of your cache so as to ensure flushes were able to write sequentially.

Problem:

• When we download all torrents in same time - we see download rate 2Mb/s (probably because of too much concurrency and requests prioritization). When download 1-by-1 - can see >50Mb/s (depends on seeders availability). I think it's not related to torrent size, but related to amount of torrents. Can provide some pprof picture if need.

More information helps! Please send it through. Have you checked CPU utilization? anacrolix/torrent should have some CPU issues with very large torrents and an unbounded max verified bytes (it will consider the entire space of possible requests to optimize requests to each peer. In the case where you're not streaming, and aren't constraining the options, the space can be huge.

• Other observation on seedbox.io (they using rTorrent): if I have 2 seedboxes - they almost never see each other - because there is limit on peers amount (and no way to configure it). They seeding to geo-locally far peers with 300kb/s. But if stop all torrents and start 1: seedboxes see each other and can sync 100mb/s. So, amount of torrent files may lead to weird results: some files will never sync, seed speed is very low, all reads/writes are random, etc... Strategy "download 1-by-1" may also help in this case.

I'm not quite sure I follow this, but you can up the number of peers allowed per Torrent. anacrolix/torrent hasn't been heavily optimized for seeding as I've not operated it in that situation in a performance critical situation. It should upload fine, but I can't make any claims as to how fast. It's possible that if you're finding uploads to distance peers to be slow, there is plenty of room for optimization.

[AskAlexSharov]

memory-based storage implementation

We can't: now in worst case on 1 machine have 400Gb of immutable torrents and also project with 7Tb database. And DB has some hot data.

file-based storage

Using mmap now, just because our db is also mmap and we have an experience. (seems it open/close file on every read? or i miss-understand the code?)

More information helps!

I will try to collect tomorrow. As I remember it used 140% on 16-core machine, but more details tomorrow.

In the case where you're not streaming, and aren't constraining the options, the space can be huge.

Yes, will try. FYI: I did just try qBittorrent client - and it has checkbox "download in sequential order" - if check it download 3 files at a time.

finding uploads to distance peers to be slow

true, on neighbour servers it show 100mb/s now. Our network is not very big now (~50 fully downloaded servers around the world - just some users of our project).

Great reply - I need get sleep with it.

[anacrolix]

Yeah there's nothing special about sequentially downloading files when you intend to download the entire torrent anyway. I expect that you're just finding it faster because the CPU load is lower if you have less pieces prioritized for download at once.

[AskAlexSharov]

Some pprof wend do .DownloadAll() on list of files (200gb of total size).
It shows now 10Mb/s (i'm not sure why not 2mb/s as I saw before).

[AskAlexSharov]

reuse-iters using sqlite instead of my pieceCompletion storage.
reuse-iters maybeUpdateRequest func is not red anymore:

[anacrolix]

Looks way better?

[AskAlexSharov]

Yes

[anacrolix]

Thank you so much for testing and persisting through this.

[AskAlexSharov]

"reuse-iters using sqlite instead of my pieceCompletion storage." - it was bug on my side

[anacrolix]

@AskAlexSharov there were massive performance improvements addressing this exact problem in 43c680e. Released in v1.54.1.