Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add relocatable installs to support concurrency-safe cached environments #5509

Merged
merged 2 commits into from
Jul 29, 2024

Conversation

charliermarsh
Copy link
Member

@charliermarsh charliermarsh commented Jul 27, 2024

Summary

The idea here is similar to what we do for wheels: we create the CachedEnvironment in the archive-v0 bucket, then symlink it to its content-addressed location. This ensures that we can always recreate these environments without concern for whether anyone else is accessing them.

Part of the challenge here is that we want the virtual environments to be relocatable, because we're now building them in one location but persisting them in another. This requires that we write relative (rather than absolute) paths to scripts and entrypoints. The main risk with relocatable virtual environments is that the scripts and entrypoints themselves are not relocatable, because they use a relative shebang. But that's fine for cached environments, which are never intended to leave the cache.

Closes #5503.

@charliermarsh charliermarsh added bug Something isn't working enhancement New feature or request preview Experimental behavior labels Jul 27, 2024
@charliermarsh charliermarsh force-pushed the charlie/safe branch 2 times, most recently from 6bcb8e1 to 67c2105 Compare July 27, 2024 17:26
@charliermarsh charliermarsh enabled auto-merge (squash) July 29, 2024 00:18
@charliermarsh charliermarsh merged commit 9af0ae2 into main Jul 29, 2024
45 checks passed
@charliermarsh charliermarsh deleted the charlie/safe branch July 29, 2024 00:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request preview Experimental behavior
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make CachedEnvironment robust to concurrent modifications
1 participant