Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix simultaneous downloads #433

Merged
merged 3 commits into from
Oct 2, 2024
Merged

Fix simultaneous downloads #433

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1f94352
limit the number of simultaneous downloads
thomhayward Oct 1, 2024
ff3b7dd
set reasonable number of concurrent downloads
thomhayward Oct 1, 2024
919e5d0
don't wrap `request::Client` in an `Arc`
thomhayward Oct 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions src/download.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ use std::{
};
use tokio::sync::Semaphore;

const CONCURRENT_DOWNLOADS: usize = 6;

/// Check the given `directory`
///
/// - If there are files there that are not in `to_download` or `to_install`, they will be moved to `directory`/.old
Expand Down Expand Up @@ -113,17 +115,17 @@ pub async fn download(
.expect("Mutex poisoned")
.enable_steady_tick(Duration::from_millis(100));
let mut tasks = FuturesUnordered::new();
let semaphore = Arc::new(Semaphore::new(75));
let client = Arc::new(reqwest::Client::new());
let semaphore = Arc::new(Semaphore::new(CONCURRENT_DOWNLOADS));
let client = reqwest::Client::new();

for downloadable in to_download {
let permit = Arc::clone(&semaphore).acquire_owned().await?;
let semaphore = Arc::clone(&semaphore);
let progress_bar = Arc::clone(&progress_bar);
let client = Arc::clone(&client);
let client = client.clone();
let output_dir = output_dir.clone();

tasks.push(async move {
let _permit = permit;
let _permit = semaphore.acquire_owned().await?;
let (length, filename) = downloadable
.download(&client, &output_dir, |additional| {
progress_bar
Expand Down