feat: Add support for CONDA_OVERRIDE_CUDA

[SobhanMP]

Description

Hi, I know this is terrible, not elegant, etc, but I do think that having support for something in the spirit of CONDA_OVERRIDE_CUDA is necessary on academic HPC environments. Reasons include

• only having internet access on the login node with no gpu
• launching jobs with submitit on login nodes with no gpu.

I'm not really used to writing rust, so would be greatful for some guidance.

Thanks

[baszalmstra]

This function should be pure functional without environmental side-effects.

Instead I propose we add some sort of from_env function to the appropriate virtual packages that allow users to choose whether they want this behavior. Or a or_override_with_env_var or something along those lines.

We should also debate the environment variable name. I believe conda has a different name? Im also not sure whether we should hardcode that in rattler itself.

[SobhanMP]

This function should be pure functional without environmental side-effects.

But the function is already not pure because it runs shell commands/dlopens libraries.

Instead I propose we add some sort of from_env function to the appropriate virtual packages that allow users to choose whether they want this behavior. Or a or_override_with_env_var or something along those lines.

I'm not sure I understand. where would this be? in the config?

We should also debate the environment variable name.

This is the name see here

[wolfv]

I think this would also be good for all the other virtual packages as a way to hot-fix certain scenarios that we haven't covered yet (#815 for example).

[baszalmstra]

My idea would be to add functions to the virtual package implementations. Lets take CUDA as an example, we would add these functions to the Cuda impl:

impl Cuda {
    pub fn from_env_var_name(env_var_name: &str) -> Option<Self> {
		let version_str = env::var(env_var_name)?;
		// Extract versions
		...
    }
    
    pub fn from_default_env_var() -> Option<Self> {
    	Self::from_env_var_name("CONDA_OVERRIDE_CUDA")
    }
    
    ...
}

You can then write:

let cuda = Cuda::from_default_env_var().or_else(Cuda::current);

To get the CUDA version of the system based on env var, or if not present by detecting it.

Similarly a function VirtualPackage::from_default_env_vars could be added.

I think this allows:

• Users of the library to decide whether they want to allow environment overrides.
• Users can specify custom environment variables if they want to.
• Users can use the default environment variable names.

Which I think hits the sweet spot.

And indeed as @wolfv said, it would be nice to add these for the other virtual packages as well.

[SobhanMP]

oh, yeah that looks great!

[SobhanMP]

Does this make sense as a rough draft?

[baszalmstra]

Yeah that looks a lot better! I left a few comments.

[SobhanMP]

I think there is one issue, finding the environment variable and it being empty is different than the environment variable not being present, should the type be Option<Option<T>>? first one option showing that the env was found the second one that the env was none?

[SobhanMP]

I think everything should have been handled gracefully.

[SobhanMP]

Do we want to override the glibc runtime? like CONDA_OVERRIDE_GLIBC="2@musl", same for Linux and Archspec. We could also do unix and win but all of these are not overridable by conda.

[iamthebot]

Do we want to override the glibc runtime? like CONDA_OVERRIDE_GLIBC="2@musl", same for Linux and Archspec. We could also do unix and win but all of these are not overridable by conda.

Suggestion-- could this be done in a followup PR? Landing CONDA_OVERRIDE_CUDA support is already immensely valuable and likely there will be a long tail of less common overrides needed.

[iamthebot]

Anything else needed to get this over the finish line? This is a blocker for us switching fully over to rattler so happy to dedicate to some time to any remaining blockers in getting this merged.

[baszalmstra]

This looks good! Ill take a closer look at the CI failure tomorrow!

[SobhanMP]

This should solve some of the failure, pre-commit is not ran by default

[iamthebot]

Are there any implications for pyrattler from this PR btw? Anything new that should be exposed?

[SobhanMP]

I don't think pyrattler has any implication for pixi, so I'd rather this be merged w/o py-rattler support. My understanding is that in py-rattler/src/virtual_package.rs the PyVirtualPackage needs to expose some more code, the problem is that this either requires either
1- A special classs for virtual packages that can be overriden (there is already one for arch),
2- that all virtual pacakges have override, or
3- using Cuda::from_default_env_var.unwrap_or(Cuda::current().into()).unwrap() in the VirtualPackage::current or try_detect_virtual_packages instead of Cuda::current()

But I think that someone else has probably a better idea.

[SobhanMP]

@baszalmstra It's very inconvenient that the checks require maintainer auth.

[baszalmstra]

Yes, but once your first PR lands that restriction is lifted 👍

[SobhanMP]

oh nice!

[baszalmstra]

I think it would be good to also expose this functionality to the python bindings, but we can do that in another PR.

Thanks for your first contribution @SobhanMP !

[SobhanMP]

@baszalmstra I think this first requires figuring out #818 (comment) . I think option 2 makes the most sense.

[SobhanMP]

The next thing is where this should be used. Should try_detect_virtual_packages be patched? Should it take some structure to rename overrides?