Skip to content

ARROW-7717: [CI] Have nightly integration test for Spark's latest release #6316

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kszucs wants to merge 7 commits into from
Closed

ARROW-7717: [CI] Have nightly integration test for Spark's latest release #6316

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
bde7d10
nightly test for spark's latest release
kszucs Jan 29, 2020
bfbaea1
remove patches argument from spark installation script
kszucs Jan 29, 2020
fc0cd40
fix argument number in usage
kszucs Jan 29, 2020
598319b
conditionally export ARROW_PRE_0_15_IPC_FORMAT
kszucs Feb 12, 2020
e91d5c3
set as nightly
kszucs Feb 12, 2020
a914eea
set ARROW_PRE_0_15_IPC_FORMAT for spark 2 build
kszucs Jun 8, 2020
55d9411
remove spark 2.4
kszucs Jul 1, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions ci/docker/conda-python-spark.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,12 @@ RUN conda install -q \
# installing specific version of spark
ARG spark=master
COPY ci/scripts/install_spark.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_spark.sh ${spark} /spark /arrow/ci/etc
RUN /arrow/ci/scripts/install_spark.sh ${spark} /spark

# build cpp with tests
ENV CC=gcc \
CXX=g++ \
ARROW_PYTHON=ON \
ARROW_HDFS=ON \
ARROW_BUILD_TESTS=OFF
ARROW_BUILD_TESTS=OFF \
SPARK_VERSION=${spark}
9 changes: 4 additions & 5 deletions ci/scripts/install_spark.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,13 @@

set -e

if [ "$#" -ne 3 ]; then
echo "Usage: $0 <spark version> <target directory> <patches directory>"
if [ "$#" -ne 2 ]; then
echo "Usage: $0 <spark version> <target directory>"
exit 1
fi

spark=$1
target=$2
patches=${3}

mkdir "${target}"
wget -q -O - https://github.com/apache/spark/archive/${spark}.tar.gz | tar -xzf - --strip-components=1 -C "${target}"
git clone https://github.com/apache/spark "${target}"
git -C "${target}" checkout "${spark}"
6 changes: 6 additions & 0 deletions ci/scripts/integration_spark.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,12 @@ set -eu

source_dir=${1}
spark_dir=${2}
spark_version=${SPARK_VERSION:-master}

if [ "${SPARK_VERSION:0:2}" == "2." ]; then
# https://github.com/apache/spark/blob/master/docs/sql-pyspark-pandas-with-arrow.md#compatibility-setting-for-pyarrow--0150-and-spark-23x-24x
export ARROW_PRE_0_15_IPC_FORMAT=1
fi

pushd ${source_dir}/java
arrow_version=`mvn org.apache.maven.plugins:maven-help-plugin:2.1.1:evaluate -Dexpression=project.version | sed -n -e '/^\[.*\]/ !{ /^[0-9]/ { p; q } }'`
Expand Down
12 changes: 11 additions & 1 deletion dev/tasks/tasks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1833,12 +1833,22 @@ tasks:
HDFS: 2.9.2
run: conda-python-hdfs

test-conda-python-3.7-spark-master:
test-conda-python-3.7-spark-branch-3.0:
Copy link
Member

@BryanCutler BryanCutler Jul 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like this should be named python-3.8 also

Copy link
Member Author

@kszucs kszucs Jul 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixing it.

ci: github
template: docker-tests/github.linux.yml
params:
env:
PYTHON: 3.7
SPARK: "branch-3.0"
# use the master branch of spark, so prevent reusing any layers
run: --no-leaf-cache conda-python-spark

test-conda-python-3.8-spark-master:
ci: github
template: docker-tests/github.linux.yml
params:
env:
PYTHON: 3.8
SPARK: master
# use the master branch of spark, so prevent reusing any layers
run: --no-leaf-cache conda-python-spark
Expand Down