[SPARK-32556][INFRA] Fix release script to have urlencoded passwords where required. #29373
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
@zhengruifeng Hi, it might be useful for you, as you are working on 3.0.1 release. |
|
Test build #127131 has finished for PR 29373 at commit
|
``` [INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 57.801 s] [INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [01:13 min] [INFO] Spark Kinesis Integration .......................... SUCCESS [ 59.862 s] [INFO] Spark Project Examples ............................. SUCCESS [02:10 min] [INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 45.454 s] [INFO] Spark Avro ......................................... SUCCESS [01:58 min] [INFO] Spark Project External Flume Sink .................. SUCCESS [01:34 min] [INFO] Spark Project External Flume ....................... FAILURE [20:04 min] [INFO] Spark Project External Flume Assembly .............. SKIPPED [INFO] Spark Project Kinesis Assembly 2.4.7 ............... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 54:10 min [INFO] Finished at: 2020-08-06T10:10:12Z [INFO] ------------------------------------------------------------------------ [ERROR] Java heap space -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError ```
|
Test build #127139 has finished for PR 29373 at commit
|
dongjoon-hyun
approved these changes
Aug 7, 2020
asfgit
pushed a commit
that referenced
this pull request
Aug 7, 2020
…where required
### What changes were proposed in this pull request?
1. URL encode the `ASF_PASSWORD` of the release manager.
2. Update the image to install `qpdf` and `jq` dep
3. Increase the JVM HEAM memory for maven build.
### Why are the changes needed?
Release script takes hours to run, and if a single failure happens about somewhere midway, then either one has to get down to manually doing stuff or re run the entire script. (This is my understanding) So, I have made the fixes of a few failures, discovered so far.
1. If the release manager password contains a char, that is not allowed in URL, then it fails the build at the clone spark step.
`git clone "https://$ASF_USERNAME:$ASF_PASSWORD$ASF_SPARK_REPO" -b $GIT_BRANCH`
^^^ Fails with bad URL
`ASF_USERNAME` may not be URL encoded, but we need to encode `ASF_PASSWORD`.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
By running the release for branch-2.4, using both type of passwords, i.e. passwords with special chars and without it.
Closes #29373 from ScrapCodes/release-script-fix2.
Lead-authored-by: Prashant Sharma <[email protected]>
Co-authored-by: Prashant Sharma <[email protected]>
Signed-off-by: Prashant Sharma <[email protected]>
(cherry picked from commit 6c3d0a4)
Signed-off-by: Prashant Sharma <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
ASF_PASSWORDof the release manager.qpdfandjqdepWhy are the changes needed?
Release script takes hours to run, and if a single failure happens about somewhere midway, then either one has to get down to manually doing stuff or re run the entire script. (This is my understanding) So, I have made the fixes of a few failures, discovered so far.
If the release manager password contains a char, that is not allowed in URL, then it fails the build at the clone spark step.
git clone "https://$ASF_USERNAME:$ASF_PASSWORD@$ASF_SPARK_REPO" -b $GIT_BRANCHASF_USERNAMEmay not be URL encoded, but we need to encodeASF_PASSWORD.Does this PR introduce any user-facing change?
No
How was this patch tested?
By running the release for branch-2.4, using both type of passwords, i.e. passwords with special chars and without it.