Skip to content

[SPARK-44544][INFRA] Deduplicate run_python_packaging_tests #42146

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zhengruifeng wants to merge 8 commits into from
Closed

[SPARK-44544][INFRA] Deduplicate run_python_packaging_tests #42146

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
37b9be4
init
zhengruifeng Jul 25, 2023
6e1e79a
del conda install
zhengruifeng Jul 25, 2023
7ee9003
add packaging test back
zhengruifeng Jul 25, 2023
e4bebaf
del exports
zhengruifeng Jul 25, 2023
d9dcf5a
refine
zhengruifeng Jul 25, 2023
134a4ff
simplify
zhengruifeng Jul 26, 2023
89d365f
fix bash
zhengruifeng Jul 26, 2023
04c271e
fix bash again
zhengruifeng Jul 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions .github/workflows/build_and_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,7 @@ jobs:
HIVE_PROFILE: ${{ matrix.hive }}
GITHUB_PREV_SHA: ${{ github.event.before }}
SPARK_LOCAL_IP: localhost
SKIP_PACKAGING: true
steps:
- name: Checkout Spark repository
uses: actions/checkout@v3
Expand Down Expand Up @@ -344,6 +345,8 @@ jobs:
java:
- ${{ inputs.java }}
modules:
- >-
pyspark-errors
- >-
pyspark-sql, pyspark-mllib, pyspark-resource, pyspark-testing
- >-
Expand All @@ -353,7 +356,7 @@ jobs:
- >-
pyspark-pandas-slow
- >-
pyspark-connect, pyspark-errors
pyspark-connect
- >-
pyspark-pandas-connect
- >-
Expand All @@ -366,6 +369,7 @@ jobs:
SPARK_LOCAL_IP: localhost
SKIP_UNIDOC: true
SKIP_MIMA: true
SKIP_PACKAGING: true
METASPACE_SIZE: 1g
steps:
- name: Checkout Spark repository
Expand Down Expand Up @@ -414,14 +418,20 @@ jobs:
python3.9 -m pip list
pypy3 -m pip list
- name: Install Conda for pip packaging test
if: ${{ matrix.modules == 'pyspark-errors' }}
run: |
curl -s https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh > miniconda.sh
bash miniconda.sh -b -p $HOME/miniconda
# Run the tests.
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
shell: 'script -q -e -c "bash {0}"'
run: |
export PATH=$PATH:$HOME/miniconda/bin
if [[ "$MODULES_TO_TEST" == "pyspark-errors" ]]; then
export PATH=$PATH:$HOME/miniconda/bin
export SKIP_PACKAGING=false
echo "Python Packaging Tests Enabled!"
fi
./dev/run-tests --parallelism 1 --modules "$MODULES_TO_TEST"
- name: Upload coverage to Codecov
if: fromJSON(inputs.envs).PYSPARK_CODECOV == 'true'
Expand Down Expand Up @@ -457,6 +467,7 @@ jobs:
GITHUB_PREV_SHA: ${{ github.event.before }}
SPARK_LOCAL_IP: localhost
SKIP_MIMA: true
SKIP_PACKAGING: true
steps:
- name: Checkout Spark repository
uses: actions/checkout@v3
Expand Down Expand Up @@ -911,6 +922,7 @@ jobs:
SPARK_LOCAL_IP: localhost
ORACLE_DOCKER_IMAGE_NAME: gvenzl/oracle-xe:21.3.0
SKIP_MIMA: true
SKIP_PACKAGING: true
steps:
- name: Checkout Spark repository
uses: actions/checkout@v3
Expand Down
2 changes: 1 addition & 1 deletion dev/run-tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -395,7 +395,7 @@ def run_python_tests(test_modules, parallelism, with_coverage=False):


def run_python_packaging_tests():
if not os.environ.get("SPARK_JENKINS"):
if not os.environ.get("SPARK_JENKINS") and os.environ.get("SKIP_PACKAGING", "false") != "true":
set_title_and_block("Running PySpark packaging tests", "BLOCK_PYSPARK_PIP_TESTS")
command = [os.path.join(SPARK_HOME, "dev", "run-pip-tests")]
run_cmd(command)
Expand Down