Skip to content

[SPARK-31320] Fix release script for 3.0.0 #28088

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cloud-fan wants to merge 1 commit into from
Closed

[SPARK-31320] Fix release script for 3.0.0 #28088

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dev/create-release/do-release-docker.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ done

GPG_KEY_FILE="$WORKDIR/gpg.key"
fcreate_secure "$GPG_KEY_FILE"
$GPG --export-secret-key --armor "$GPG_KEY" > "$GPG_KEY_FILE"
$GPG --export-secret-key --armor --pinentry-mode loopback --passphrase "$GPG_PASSPHRASE" "$GPG_KEY" > "$GPG_KEY_FILE"
Copy link
Contributor Author

@cloud-fan cloud-fan Apr 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the modern gpg command requires users to input a password, which breaks the workflow of the script. This fixes it.


run_silent "Building spark-rm image with tag $IMGTAG..." "docker-build.log" \
docker build -t "spark-rm:$IMGTAG" --build-arg UID=$UID "$SELF/spark-rm"
Expand Down
10 changes: 7 additions & 3 deletions dev/create-release/release-util.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,10 +159,14 @@ function get_release_info {
export SPARK_PACKAGE_VERSION="$RELEASE_TAG"

# Gather some user information.
export ASF_USERNAME=$(read_config "ASF user" "$LOGNAME")
if [ -z "$ASF_USERNAME" ]; then
Copy link
Contributor Author

@cloud-fan cloud-fan Apr 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should respect these env variables if they exist

export ASF_USERNAME=$(read_config "ASF user" "$LOGNAME")
fi

GIT_NAME=$(git config user.name || echo "")
export GIT_NAME=$(read_config "Full name" "$GIT_NAME")
if [ -z "$GIT_NAME" ]; then
GIT_NAME=$(git config user.name || echo "")
export GIT_NAME=$(read_config "Full name" "$GIT_NAME")
fi

export GIT_EMAIL="[email protected]"
export GPG_KEY=$(read_config "GPG key" "$GIT_EMAIL")
Expand Down
13 changes: 5 additions & 8 deletions dev/create-release/spark-rm/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ ENV DEBCONF_NONINTERACTIVE_SEEN true
# These arguments are just for reuse and not really meant to be customized.
ARG APT_INSTALL="apt-get install --no-install-recommends -y"

ARG BASE_PIP_PKGS="setuptools wheel"
ARG PIP_PKGS="pyopenssl numpy sphinx"
ARG PIP_PKGS="sphinx==2.3.1 mkdocs==1.0.4 numpy==1.18.1"
ARG GEM_PKGS="jekyll:4.0.0 jekyll-redirect-from:0.16.0 rouge:3.15.0"
Copy link
Contributor Author

@cloud-fan cloud-fan Apr 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This follows part of #27534


# Install extra needed repos and refresh.
# - CRAN repo
Expand Down Expand Up @@ -62,12 +62,11 @@ RUN apt-get clean && apt-get update && $APT_INSTALL gnupg ca-certificates && \
curl -sL https://deb.nodesource.com/setup_11.x | bash && \
$APT_INSTALL nodejs && \
# Install needed python packages. Use pip for installing packages (for consistency).
$APT_INSTALL libpython3-dev python3-pip && \
$APT_INSTALL libpython3-dev python3-pip python3-setuptools && \
# Change default python version to python3.
update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1 && \
update-alternatives --install /usr/bin/python python /usr/bin/python3.6 2 && \
update-alternatives --set python /usr/bin/python3.6 && \
pip3 install $BASE_PIP_PKGS && \
pip3 install $PIP_PKGS && \
# Install R packages and dependencies used when building.
# R depends on pandoc*, libssl (which are installed above).
Expand All @@ -76,10 +75,8 @@ RUN apt-get clean && apt-get update && $APT_INSTALL gnupg ca-certificates && \
Rscript -e "install.packages(c('curl', 'xml2', 'httr', 'devtools', 'testthat', 'knitr', 'rmarkdown', 'roxygen2', 'e1071', 'survival'), repos='https://cloud.r-project.org/')" && \
Rscript -e "devtools::install_github('jimhester/lintr')" && \
# Install tools needed to build the documentation.
$APT_INSTALL ruby2.3 ruby2.3-dev mkdocs && \
gem install jekyll --no-rdoc --no-ri -v 3.8.6 && \
gem install jekyll-redirect-from -v 0.15.0 && \
gem install rouge
$APT_INSTALL ruby2.5 ruby2.5-dev && \
Copy link
Member

@dongjoon-hyun dongjoon-hyun Apr 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, did ruby2.5 work correctly?

Copy link
Contributor Author

@cloud-fan cloud-fan Apr 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mkdocs requires ruby 2.4+, I googled a bit and seems ubuntu 18 supports ruby 2.5 at most.

gem install --no-document $GEM_PKGS

WORKDIR /opt/spark-rm/output

Expand Down