[SPARK-10878][core] Fix race condition when multiple clients resolves artifacts at the same time #21251
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #90273 has finished for PR 21251 at commit
|
Member
Author
Member
Author
|
cc @gatorsmile |
vanzin
reviewed
May 7, 2018
| ivySettings: IvySettings, | ||
| ivyConfName: String): Unit = { | ||
| val currentResolutionFiles = Seq[File]( | ||
| new File(ivySettings.getDefaultCache, |
Contributor
There was a problem hiding this comment.
nit: you could move new File(ivySettings.getDefaultCache to the foreach loop instead.
| isTest = true) | ||
| val r = """.*org.apache.spark-spark-submit-parent-.*""".r | ||
| assert(ivySettings.getDefaultCache.listFiles.map(_.getName) | ||
| .forall { case n @ r() => false case _ => true }, |
Contributor
There was a problem hiding this comment.
nit: I think using !<list of files>.exists(r.findFirstIn(_).isDefined) would be slightly clearer than you version.
| "1.0")) | ||
|
|
||
| /** | ||
| * clear ivy resolution from current launch. The resolution file is usually at |
Contributor
There was a problem hiding this comment.
nit: start comment with capital letter.
|
Test build #90396 has finished for PR 21251 at commit
|
Member
Author
|
retest this please |
|
Test build #90405 has finished for PR 21251 at commit
|
Contributor
|
LGTM. Merging to master / 2.3. |
asfgit
pushed a commit
that referenced
this pull request
May 10, 2018
… artifacts at the same time ## What changes were proposed in this pull request? When multiple clients attempt to resolve artifacts via the `--packages` parameter, they could run into race condition when they each attempt to modify the dummy `org.apache.spark-spark-submit-parent-default.xml` file created in the default ivy cache dir. This PR changes the behavior to encode UUID in the dummy module descriptor so each client will operate on a different resolution file in the ivy cache dir. In addition, this patch changes the behavior of when and which resolution files are cleaned to prevent accumulation of resolution files in the default ivy cache dir. Since this PR is a successor of #18801, close #18801. Many codes were ported from #18801. **Many efforts were put here. I think this PR should credit to Victsm .** ## How was this patch tested? added UT into `SparkSubmitUtilsSuite` Author: Kazuaki Ishizaki <[email protected]> Closes #21251 from kiszk/SPARK-10878. (cherry picked from commit d3c426a) Signed-off-by: Marcelo Vanzin <[email protected]>
robert3005
pushed a commit
to palantir/spark
that referenced
this pull request
Jun 24, 2018
… artifacts at the same time ## What changes were proposed in this pull request? When multiple clients attempt to resolve artifacts via the `--packages` parameter, they could run into race condition when they each attempt to modify the dummy `org.apache.spark-spark-submit-parent-default.xml` file created in the default ivy cache dir. This PR changes the behavior to encode UUID in the dummy module descriptor so each client will operate on a different resolution file in the ivy cache dir. In addition, this patch changes the behavior of when and which resolution files are cleaned to prevent accumulation of resolution files in the default ivy cache dir. Since this PR is a successor of apache#18801, close apache#18801. Many codes were ported from apache#18801. **Many efforts were put here. I think this PR should credit to Victsm .** ## How was this patch tested? added UT into `SparkSubmitUtilsSuite` Author: Kazuaki Ishizaki <[email protected]> Closes apache#21251 from kiszk/SPARK-10878.
otterc
pushed a commit
to linkedin/spark
that referenced
this pull request
Mar 22, 2023
… artifacts at the same time When multiple clients attempt to resolve artifacts via the `--packages` parameter, they could run into race condition when they each attempt to modify the dummy `org.apache.spark-spark-submit-parent-default.xml` file created in the default ivy cache dir. This PR changes the behavior to encode UUID in the dummy module descriptor so each client will operate on a different resolution file in the ivy cache dir. In addition, this patch changes the behavior of when and which resolution files are cleaned to prevent accumulation of resolution files in the default ivy cache dir. Since this PR is a successor of apache#18801, close apache#18801. Many codes were ported from apache#18801. **Many efforts were put here. I think this PR should credit to Victsm .** added UT into `SparkSubmitUtilsSuite` Author: Kazuaki Ishizaki <[email protected]> Closes apache#21251 from kiszk/SPARK-10878. (cherry picked from commit d3c426a) RB=1313943 G=superfriends-reviewers R=fli,mshen,yezhou,edlu A=fli
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
When multiple clients attempt to resolve artifacts via the
--packagesparameter, they could run into race condition when they each attempt to modify the dummyorg.apache.spark-spark-submit-parent-default.xmlfile created in the default ivy cache dir.This PR changes the behavior to encode UUID in the dummy module descriptor so each client will operate on a different resolution file in the ivy cache dir. In addition, this patch changes the behavior of when and which resolution files are cleaned to prevent accumulation of resolution files in the default ivy cache dir.
Since this PR is a successor of #18801, close #18801. Many codes were ported from #18801. Many efforts were put here. I think this PR should credit to @Victsm .
How was this patch tested?
added UT into
SparkSubmitUtilsSuite