Skip to content

stop "uncompressing" registry data#4470

Merged
KristofferC merged 2 commits intomasterfrom
pkg-registry-no-incompress
Nov 4, 2025
Merged

stop "uncompressing" registry data#4470
KristofferC merged 2 commits intomasterfrom
pkg-registry-no-incompress

Conversation

@KristofferC
Copy link
Copy Markdown
Member

@KristofferC KristofferC commented Oct 26, 2025

instead, do all operations on top of compressed data (VersionRange -> data) instead of uncompressed (VersionNumber -> data).

This avoids having to materialize dictionaries with compat data for every version.

As an example:

Pkg.activate(; temp=true)
Pkg.add("Plots")
 _, _, reg_instance = first(Pkg.Registry.REGISTRY_CACHE)[end]

# master
julia> (Base.summarysize(reg_instance) - Base.summarysize(reg_instance.in_memory_registry)) / 1024^2
29.9 # MB

# master
julia> (Base.summarysize(reg_instance) - Base.summarysize(reg_instance.in_memory_registry)) / 1024^2
66.6 # MB

(where I am removing the size of storing the raw data of the registry itself).

and


julia> Pkg.rm(["Plots"]); empty!(Pkg.Registry.REGISTRY_CACHE); @time Pkg.add(["Plots"])

# PR
  1.320097 seconds (8.84 M allocations: 587.032 MiB, 11.50% gc time)

# master
  1.397698 seconds (9.65 M allocations: 718.492 MiB, 13.56% gc time)

@github-project-automation github-project-automation Bot moved this to New in Pkg.jl Oct 26, 2025
@KristofferC KristofferC force-pushed the pkg-registry-no-incompress branch 5 times, most recently from e190b0a to 3e23b37 Compare October 28, 2025 18:33
@KristofferC
Copy link
Copy Markdown
Member Author

Test seems to pass now and updated figures for

try Pkg.rm(["Plots"]; io=devnull) catch end; empty!(Pkg.Registry.REGISTRY_CACHE); @time Pkg.add(["Plots"]; io=devnull)

are

# master
  1.117075 seconds (8.07 M allocations: 644.749 MiB, 1 lock conflict)

# PR
  0.800553 seconds (6.86 M allocations: 466.209 MiB)

and specifically the deps_graph step where we gather information from the registry is

#master
  collect deps
  0.129429 seconds (2.61 M allocations: 290.017 MiB)

# PR
  collect deps
  0.063639 seconds (1.43 M allocations: 115.601 MiB)

I still need to take a proper pass through this because there is honestly quite a bit of AI slop in here, but I just wanted a finished implementation so I could look at the numbers and see it was what I thought it would be.

@KristofferC KristofferC force-pushed the pkg-registry-no-incompress branch from 290a468 to 3cfefbc Compare October 28, 2025 19:23
KristofferC and others added 2 commits November 4, 2025 12:56
instead, do all operations on top of compressed data (`VersionRange` -> data) instead of uncompressed (`VersionNumber` -> data).

This avoids having to materialize dictionaries with compat data for every version.
@KristofferC KristofferC force-pushed the pkg-registry-no-incompress branch from 3cfefbc to 05c6e20 Compare November 4, 2025 13:16
@KristofferC KristofferC marked this pull request as ready for review November 4, 2025 13:16
@KristofferC KristofferC merged commit efe1eaf into master Nov 4, 2025
11 checks passed
@KristofferC KristofferC deleted the pkg-registry-no-incompress branch November 4, 2025 14:15
@github-project-automation github-project-automation Bot moved this from New to Done in Pkg.jl Nov 4, 2025
KristofferC added a commit that referenced this pull request Nov 7, 2025
@KristofferC KristofferC mentioned this pull request Nov 7, 2025
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant