Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: use the gateway to fetch repodata #1307

Merged
merged 14 commits into from
May 14, 2024

Conversation

baszalmstra
Copy link
Contributor

This branch replaces repodata fetching with the repodata gateway from conda/rattler#560 . This yields considerable time and memory savings!

@ruben-arts
Copy link
Contributor

I would be more than glad to merge this feature but there are a few questions I have.

  • Can users opt-in to this feature?
  • Can users disable it if it gives them issues?
  • What is the fallback if the website goes down?
  • How can someone request the real latest version if the shards get out of date due to server issues?

@baszalmstra
Copy link
Contributor Author

baszalmstra commented May 1, 2024

Hey @ruben-arts !

This feature is not yet done. The biggest missing piece is proper feedback of repodata fetching. I will add this in the next few days.

There are two features here:

  1. The gateway itself which replaces the old way of fetching repodata. This still uses the "old" method underneath.

  2. The use of the sharded index. To use the sharded index you currently have to use a special channel: e.g. https://fast.prefix.dev/conda-forge . If and when the CEP gets merged and the sharded index will become more widespread the gateway will have a detection mechanism similar to JLAP. This should similarly then then also be disableable through config.

@ruben-arts
Copy link
Contributor

Great, thanks for the info!

@baszalmstra baszalmstra marked this pull request as ready for review May 14, 2024 13:39
src/lock_file/update.rs Outdated Show resolved Hide resolved
@ruben-arts ruben-arts enabled auto-merge (squash) May 14, 2024 13:58
@ruben-arts ruben-arts merged commit 1ec1b90 into prefix-dev:main May 14, 2024
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants