Skip to content

feat: add run_exports cache#1060

Merged
tdejager merged 26 commits intoconda:mainfrom
nichmor:feat/partial-unpacking
Feb 18, 2025
Merged

feat: add run_exports cache#1060
tdejager merged 26 commits intoconda:mainfrom
nichmor:feat/partial-unpacking

Conversation

@nichmor
Copy link
Contributor

@nichmor nichmor commented Feb 11, 2025

Description

Add RunExportsCache entity for storing run_exports.json

This PR introduces a new RunExportsCache entity that stores run_exports.json files on disk.
This will allow rattler-build to access only the run_exports.json from archives instead of
unpacking entire packages, which resolves issues on Windows.

Implementation details:

  • Used in prefix-dev/rattler-build@main/src/run_exports.rs#L99
  • Provides URL fetching with retry policy (similar to PackageCache)
  • Focuses only on run_exports.json retrieval (unlike PackageCache)
  • Uses atomic file replacement via NamedTempFile::persist, eliminating need for filesystem locks

Future benefit: Optimizes package handling by avoiding full package extraction when only
run_exports.json is needed.

@nichmor nichmor changed the title feat: add archive cache feat: add run_exports cache Feb 12, 2025
@wolfv
Copy link
Contributor

wolfv commented Feb 13, 2025

One quick remark: I think we shoudl fall back to the md5 hash if sha256 is not available.

temp_file
.as_file()
.try_clone()
.map_err(FetchRepoDataError::IoError)?,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this happen? Just wondering :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess the clone operation can fail if https://man7.org/linux/man-pages/man2/fcntl.2.html will return something unexpected which will be converted by https://docs.rs/cvt/latest/cvt/.
don't know what are the triggers for this

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here! Use into_parts to forgo to the try_clone

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I cannot do it here.

HashingWriter::<_, Blake2b256>::new(file); take ownership of the file.
If I will do temp_file.into_parts() I will lose this file handle when creating it from_parts so later, I will not be able to create NamedTempFile again.

Copy link
Collaborator

@tdejager tdejager left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking, good just a few minor nits.

temp_file
.as_file()
.try_clone()
.map_err(FetchRepoDataError::IoError)?,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here! Use into_parts to forgo to the try_clone

@tdejager tdejager merged commit 70a2319 into conda:main Feb 18, 2025
15 checks passed
This was referenced Feb 18, 2025
@baszalmstra baszalmstra mentioned this pull request Feb 25, 2025
travishathaway pushed a commit to travishathaway/rattler that referenced this pull request Feb 26, 2025
Add `RunExportsCache` entity for storing run_exports.json

This PR introduces a new `RunExportsCache` entity that stores `run_exports.json` files on disk. 
This will allow rattler-build to access only the `run_exports.json` from archives instead of 
unpacking entire packages, which resolves issues on Windows.

Implementation details:
- Used in prefix-dev/rattler-build@main/src/run_exports.rs#L99
- Provides URL fetching with retry policy (similar to PackageCache)
- Focuses only on run_exports.json retrieval (unlike PackageCache)
- Uses atomic file replacement via NamedTempFile::persist, eliminating need for filesystem locks

Future benefit: Optimizes package handling by avoiding full package extraction when only 
run_exports.json is needed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants