Optimize RepoMappingManifestAction #20091
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The following optimizations reduce the time spent writing the repo mapping manifest from >6s to <80ms for a test referencing each of ~4,000 repos created by a module extension: * The set of repository names appearing in runfiles paths is only constructed once rather than for each repo by moving the `build()` call out of a closure. * The relevant entries per repository are now sorted after filtering out those for repos that don't contribute runfiles. * Crucially, the relevant mapping entries per repository are now cached per instance of `RepositoryMapping#entries()`. Since extension repos all share the same instance due to interning, this reduces the complexity from quadratic to linear in the number of extension repos.
fmeum
requested review from
sdtwigg and
Wyverald
and removed request for
a team and
sdtwigg
github-actions
bot
added
awaiting-review
|
Turns out that the action itself is really fast now, but the key computation is still slow. I will see whether that can be fixed without memory overhead. |
|
@bazel-io flag |
bazel-io
added
the
potential release blocker
|
I pushed a commit that also caches the key computation. The first computation is actually still a bit slower than running the action, but subsequent evaluations are much faster (~1-2ms), leveraging both the nested set fingerprint cache as well as the new repo mapping fingerprint cache. Here is a profile of the reproducing repo (https://github.com/DavidZbarsky-at/repo-mapping-manifest-repro) with |
|
@bazel-io fork 7.0.0 |
bazel-io
removed
the
potential release blocker
Wyverald
approved these changes
Nov 8, 2023
src/main/java/com/google/devtools/build/lib/analysis/RepoMappingManifestAction.java
Outdated
Show resolved
Hide resolved
Wyverald
added
awaiting-PR-merge
fmeum
force-pushed
the
faster-repo-mapping-manifest
branch
from
7684474 to
c6e3efb
Compare
github-actions
bot
removed
the
awaiting-PR-merge
bazel-io
pushed a commit
to bazel-io/bazel
that referenced
this pull request
Nov 10, 2023
The following optimizations reduce the time spent writing the repo mapping manifest from >6s to <80ms for a test referencing each of ~4,000 repos created by a module extension: * The set of repository names appearing in runfiles paths is only constructed once rather than for each repo by moving the `build()` call out of a closure. * The relevant entries per repository are now sorted after filtering out those for repos that don't contribute runfiles. * Crucially, the relevant mapping entries per repository are now cached per instance of `RepositoryMapping#entries()`. Since extension repos all share the same instance due to interning, this reduces the complexity from quadratic to linear in the number of extension repos. Closes bazelbuild#20091. PiperOrigin-RevId: 581128978 Change-Id: I946e7788b8538e84714cf25ece89a86edd0d6948
keertk
pushed a commit
that referenced
this pull request
Nov 10, 2023
The following optimizations reduce the time spent writing the repo mapping manifest from >6s to <80ms for a test referencing each of ~4,000 repos created by a module extension: * The set of repository names appearing in runfiles paths is only constructed once rather than for each repo by moving the `build()` call out of a closure. * The relevant entries per repository are now sorted after filtering out those for repos that don't contribute runfiles. * Crucially, the relevant mapping entries per repository are now cached per instance of `RepositoryMapping#entries()`. Since extension repos all share the same instance due to interning, this reduces the complexity from quadratic to linear in the number of extension repos. Closes #20091. Commit 305ab3b PiperOrigin-RevId: 581128978 Change-Id: I946e7788b8538e84714cf25ece89a86edd0d6948 Co-authored-by: Fabian Meumertzheim <[email protected]>
|
The changes in this PR have been included in Bazel 7.0.0 RC5. Please test out the release candidate and report any issues as soon as possible. If you're using Bazelisk, you can point to the latest RC by setting USE_BAZEL_VERSION=last_rc. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The following optimizations reduce the time spent writing the repo mapping manifest from >6s to <80ms for a test referencing each of ~4,000 repos created by a module extension:
build()call out of a closure.RepositoryMapping#entries(). Since extension repos all share the same instance due to interning, this reduces the complexity from quadratic to linear in the number of extension repos.