-
Notifications
You must be signed in to change notification settings - Fork 391
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[JENKINS-64383] combined refrepo became our bottleneck, support a fanout location too #644
Open
jimklimov
wants to merge
145
commits into
jenkinsci:master
Choose a base branch
from
jimklimov:refrepo-args
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,643
−65
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jimklimov
force-pushed
the
refrepo-args
branch
2 times, most recently
from
December 8, 2020 12:34
191ca38
to
694ee90
Compare
…RL} to replace by url => funny dir subtree in filesystem
…atibleGitAPIImpl.java so its logic (expected to grow in complexity) can be shared by both JGitAPIImpl.java and CliGitAPIImpl.java
…d ref-repos in submodule checkouts (only CliGitAPIImpl.java has it)
… findParameterizedReferenceRepository()
…SHA256} reference repos
… name for the mirrored repo
…6() with real sanity checks
…(): do not bother normalizing the URL if the string is not with supported suffix
…intsToLocal*Mirror() with custom paths and bare vs workspace repos
…6() to find expected paths
…refactor getObjectPath(referencePath) to check on git dirs elsewhere later
…erenceRepository() and isParameterizedReferenceRepository() taking a File reference (not only a String) object
…ceRepository() processing to stderr
… keep original reference intact, and as indicator to recreate referencePath object once for many cases
…256_FALLBACK suffixes for using unsuffixed directory if expanded path points nowhere useful
…ith fallback modes of operation
…d GIT_SUBMODULES_FALLBACK support
jimklimov
added a commit
to jimklimov/git-client-plugin
that referenced
this pull request
Jun 14, 2023
jimklimov
added a commit
to jimklimov/git-client-plugin
that referenced
this pull request
Jun 14, 2023
… the PR jenkinsci#644 (by @MarkEWaite) and fix some
jimklimov
added a commit
to jimklimov/git-client-plugin
that referenced
this pull request
Jun 14, 2023
…MarkEWaite) and fix some Keeping an eye out for comprehensibility of the resulting source code text
Promulgates inconsistent coding style and breaks "logical" markup of messages defined on multiple lines (like "item " + javavar) which it fails to keep together, etc. Unavoidable evil I guess, but hopefully someone can configure it.
Signed-off-by: Jim Klimov <[email protected]>
…refrepo-args Signed-off-by: Evgeny Klimov <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
enhancement
Improvement or new feature
ShortTerm
Short term improvements
tests
Automated test addition or improvement
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
JENKINS-64383 - combined refrepo became our bottleneck
As detailed in the JIRA issue, our heavy use of a single combined reference repository made it more a bottleneck and cause of job timeouts than a speedup and reliability improvement which it once was. This PR explores a way to keep the single point of configuration of the reference repository directory, suffixed with some "magic variable" to substitute a path to subdirectory with a smaller-scope reference repository for a particular source Git URL. On file systems with symlinks it is possible to maintain several such names that would point to the same directory, for closely-related repositories or different URLs of the same repository.
This PoC introduces trivial support for reference repository paths ending with
/${GIT_URL}
to replace by url => funny dir subtree in filesystem. Its limitation at the moment is that the URL is pasted in verbatim - this works for Linux and Unix like systems that only forbid a 0x00 and a slash from being characters in a filename, and slash suits us as a directory subtree separator. This code likely won't run on Windows as is (colon inhttps:
and likely other chars - Microsoft has an extensive list of invalid chars).The next ideas, commented but not yet PoCed, are to either escape such characters (non-ASCII and offensive to at least one popular filesystem), or convert URLs into base64 strings or sha/md5/... hashes. Using submodules and finding a way to map several URLs to a certain submodule might be a good idea if they keep indexes separately. This all can be built on top of this PoCed code by introducing further suffixes and handling for them.
It was tested on a MultiBranch pipeline job, where an original definition of the reference repository was suffixed with the new magic string, yielding
/home/abuild/jenkins-gitcache/${GIT_URL}
(verbatim in "Advanced clone behaviours"). During the checkout into a wiped workspace, with this plugin variant installed:This completed quickly, much faster than the usual checkout with huge refrepo in original
/home/abuild/jenkins-gitcache/
, and did automatically find the "funny"/home/abuild/jenkins-gitcache/https://github.com/zeromq/czmq.git
directory prepared with the single repo's reference cache:DOCS NOTE: With 2.36.x and newer Git versions, if your reference repository maintenance script runs as a different user account than the Jenkins server (or Jenkins agent), safety checks about
safe.directory
(see https://github.blog/2022-04-18-highlights-from-git-2-36/) can be disabled by configuring each such user account: