GitInputAccessor: Speed up lookup()#10013
Merged
edolstra merged 1 commit intoNixOS:masterfrom Feb 15, 2024
Merged
Conversation
A command like rm -rf ~/.cache/nix/tarball-cache/ ~/.cache/nix/fetcher-cache-v1.sqlite*; nix flake metadata 'git+file:///home/eelco/Dev/nixpkgs?rev=9463103069725474698139ab10f17a9d125da859' was spending about 84% of its runtime in lookup(), specifically in git_tree_entry_bypath(). (The reading of blobs is less than 3%.) It appears libgit2 doesn't do a lot of caching of trees, so we now make sure that when we look up a path, we add all its parents, and all the immediate children of the parents (since we have them in memory anyway), to our own cache. This speed up the command above from 17.2s to 7.8s on my machine. Fixes (or at least should improve a lot) NixOS#9684.
github-actions
bot
added
the
fetching
This was referenced Feb 14, 2024
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
A command like
was spending about 84% of its runtime in lookup(), specifically in git_tree_entry_bypath(). (The reading of blobs is less than 3%.)
It appears libgit2 doesn't do a lot of caching of trees, so we now make sure that when we look up a path, we add all its parents, and all the immediate children of the parents (since we have them in memory anyway), to our own cache.
This speeds up the command above from 17.2s to 7.8s on my machine.
Fixes (or at least should improve a lot) #9684.
Context
Priorities and Process
Add 👍 to pull requests you find important.
The Nix maintainer team uses a GitHub project board to schedule and track reviews.