feat: add HTTP timeout, retries, and concurrency env var support#5581
Conversation
Pixi uses uv as a Rust library rather than a CLI subprocess, so uv's own environment variable handling (UV_HTTP_TIMEOUT, UV_REQUEST_TIMEOUT, HTTP_TIMEOUT) was never invoked. This left users unable to override the default 30-second read timeout, causing failures when downloading large packages (e.g., Nvidia CUDA wheels). Read these environment variables in UvResolutionContext and pass the parsed timeout to BaseClientBuilder, matching the precedence order used by the uv CLI itself. https://claude.ai/code/session_0151RuGQuJDMRpyPG1uvwd1W
Add support for additional uv environment variables that were silently ignored because pixi uses uv as a library rather than a CLI subprocess: - UV_HTTP_RETRIES: configure the number of HTTP retries (default: 3) - UV_CONCURRENT_DOWNLOADS: max concurrent downloads (default: 50) - UV_CONCURRENT_BUILDS: max concurrent source builds (default: cpus) - UV_CONCURRENT_INSTALLS: max concurrent installs (default: cpus) Also fix a bug where pixi's own `concurrency.downloads` config setting was being ignored for uv operations — UvResolutionContext was always using Concurrency::default() instead of reading from pixi config. The precedence for downloads is: UV_CONCURRENT_DOWNLOADS env var > pixi concurrency.downloads config > default (50). https://claude.ai/code/session_0151RuGQuJDMRpyPG1uvwd1W
Reduce 18 individual test functions down to 3 focused tests that cover the same behavior with less repetition. https://claude.ai/code/session_0151RuGQuJDMRpyPG1uvwd1W
wolfv
changed the title
|
It could be nice to have this soon, as I am hitting a timeout when installing pytorch from As a note, things worked well a few weeks ago (so something changed in our network configuration in the meantime). I can get it installed with uv though, so I am not sure what pixi does differently. I can open an issue if you think it is useful. |
| fn with_env_vars<F, R>(vars: &[(&str, Option<&str>)], f: F) -> R | ||
| where | ||
| F: FnOnce() -> R, | ||
| { | ||
| let _lock = ENV_MUTEX.lock().unwrap(); | ||
| let originals: Vec<_> = vars | ||
| .iter() | ||
| .map(|(k, _)| (*k, std::env::var(k).ok())) | ||
| .collect(); | ||
|
|
||
| // SAFETY: We hold ENV_MUTEX to ensure no concurrent env var access. | ||
| unsafe { | ||
| for (k, v) in vars { | ||
| match v { | ||
| Some(val) => std::env::set_var(k, val), | ||
| None => std::env::remove_var(k), | ||
| } | ||
| } | ||
| } | ||
|
|
||
| let result = f(); | ||
|
|
||
| unsafe { | ||
| for (k, v) in &originals { | ||
| match v { | ||
| Some(val) => std::env::set_var(k, val), | ||
| None => std::env::remove_var(k), | ||
| } | ||
| } | ||
| } | ||
|
|
||
| result | ||
| } |
There was a problem hiding this comment.
I think we had a crate that basically already did this.
|
im also hitting an annoying af timeout and spent too long diving into the chain of a script to find it was this package not accepting timeout env's. would very much appreciate this coming in <3 |
| .extra_middleware(self.extra_middleware.clone()); | ||
|
|
||
| if let Some(timeout) = self.http_timeout { | ||
| base_client_builder = base_client_builder.timeout(timeout); |
There was a problem hiding this comment.
maybe we should double check if we should use read_timeout here? timeout is a "global" timeout (ie. deadline for the entire request) but read timeout refreshes after each "chunk" that was read.
There was a problem hiding this comment.
The uv client builder uses read_timeout for the timeout:
fn create_client(
&self,
user_agent: &str,
timeout: Duration,
ssl_cert_file_exists: bool,
security: Security,
redirect_policy: RedirectPolicy,
) -> Client {
// Configure the builder.
let client_builder = ClientBuilder::new()
.http1_title_case_headers()
.user_agent(user_agent)
.pool_max_idle_per_host(20)
.read_timeout(timeout) # <<<<<<<<<<<<<<<<<<
.tls_built_in_root_certs(self.built_in_root_certs)
.redirect(redirect_policy.reqwest_policy());
ruben-arts
left a comment
ruben-arts
left a comment
There was a problem hiding this comment.
Looks good to me, testing it, it did work as expected. I was able to set the timeout value through env vars and it did work if I made it very low. I'm not able to reproduce the situation where it goes over the default value to test if that can be solved with making this even higher but the code paths do look properly setup for that.
7 participants
Description
This PR adds support for reading HTTP timeout, retry, and concurrency configuration from environment variables in the UV resolution context, matching the behavior of the
uvCLI.New Features:
UV_HTTP_TIMEOUT,UV_REQUEST_TIMEOUT, orHTTP_TIMEOUTenvironment variables (in that order of precedence). Supports both integer seconds and floating-point values.UV_HTTP_RETRIESenvironment variable. Accepts non-negative integers.UV_CONCURRENT_DOWNLOADS,UV_CONCURRENT_BUILDS, andUV_CONCURRENT_INSTALLSenvironment variables, with fallback to pixi config and uv defaults.Implementation Details:
http_timeout: Option<Duration>andhttp_retries: Option<u32>fields toUvResolutionContextread_http_timeout_from_env(),read_http_retries_from_env(),read_usize_env(), andbuild_concurrency()How Has This Been Tested?
Added 20+ unit tests covering:
All tests use a mutex-protected helper to safely manipulate environment variables in a multi-threaded test environment.
Checklist:
https://claude.ai/code/session_0151RuGQuJDMRpyPG1uvwd1W