Skip to content

[SPARK-44524][BUILD] Balancing pyspark-pandas-connect and pyspark-pandas-slow-connect GA testing time #42115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
panbingkun wants to merge 22 commits into from
Closed

[SPARK-44524][BUILD] Balancing pyspark-pandas-connect and pyspark-pandas-slow-connect GA testing time #42115

Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
e759b1d
[TEST] free disk space
panbingkun Jul 24, 2023
87e4b9e
[TEST] free disk space
panbingkun Jul 24, 2023
49588d9
[TEST] free disk space
panbingkun Jul 24, 2023
76b1de7
[TEST] free disk space
panbingkun Jul 24, 2023
78d9019
[TEST] free disk space
panbingkun Jul 24, 2023
9b293c1
[TEST] free disk space
panbingkun Jul 24, 2023
90b8207
[TEST] free disk space
panbingkun Jul 24, 2023
58fd90c
[TEST] free disk space
panbingkun Jul 24, 2023
ded1def
[SPARK-44524][BUILD] Add a new test group for pyspark-pandas-slow-con…
panbingkun Jul 24, 2023
ac255f6
[SPARK-44524][BUILD] Add a new test group for pyspark-pandas-slow-con…
panbingkun Jul 24, 2023
095c808
[SPARK-44524][BUILD] Add a new test group for pyspark-pandas-slow-con…
panbingkun Jul 25, 2023
45436d0
Merge branch 'master' into free_disk_space
panbingkun Jul 25, 2023
5fd4cba
[SPARK-44524][BUILD] Add a new test group for pyspark-pandas-slow-con…
panbingkun Jul 25, 2023
3f472e0
[SPARK-44524][BUILD] Add a new test group for pyspark-pandas-slow-con…
panbingkun Jul 25, 2023
2a13db4
[SPARK-44524][BUILD] Add a new test group for pyspark-pandas-slow-con…
panbingkun Jul 25, 2023
b892841
Merge branch 'master' into free_disk_space
panbingkun Jul 26, 2023
5f93b32
rebalancing pyspark-pandas-connect and pyspark-pandas-slow-connect
panbingkun Jul 26, 2023
b927af0
rebalancing pyspark-pandas-connect and pyspark-pandas-slow-connect
panbingkun Jul 27, 2023
e1e85fa
Merge branch 'master' into free_disk_space
panbingkun Jul 27, 2023
4266e89
Merge branch 'master' into free_disk_space
panbingkun Jul 27, 2023
53fe574
Apply suggestions from code review
panbingkun Jul 28, 2023
7846ae2
[SPARK-44524][BUILD] Balancing pyspark-pandas-connect and pyspark-pan…
panbingkun Jul 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 17 additions & 18 deletions dev/sparktestsupport/modules.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -890,7 +890,7 @@ def __hash__(self):

pyspark_pandas_connect = Module(
name="pyspark-pandas-connect",
dependencies=[pyspark_connect, pyspark_pandas],
dependencies=[pyspark_connect, pyspark_pandas, pyspark_pandas_slow],
source_file_regexes=[
"python/pyspark/pandas",
],
Expand Down Expand Up @@ -948,23 +948,6 @@ def __hash__(self):
"pyspark.pandas.tests.connect.test_parity_utils",
"pyspark.pandas.tests.connect.test_parity_window",
"pyspark.pandas.tests.connect.indexes.test_parity_base",
],
excluded_python_implementations=[
"PyPy" # Skip these tests under PyPy since they require numpy, pandas, and pyarrow and
# they aren't available there
],
)


# This module should contain the same test list with 'pyspark_pandas_slow' for maintenance.
pyspark_pandas_slow_connect = Module(
name="pyspark-pandas-slow-connect",
dependencies=[pyspark_connect, pyspark_pandas_slow],
source_file_regexes=[
"python/pyspark/pandas",
],
python_test_goals=[
# pandas-on-Spark unittests
"pyspark.pandas.tests.connect.indexes.test_parity_datetime",
"pyspark.pandas.tests.connect.indexes.test_parity_align",
"pyspark.pandas.tests.connect.indexes.test_parity_indexing",
Expand All @@ -984,6 +967,22 @@ def __hash__(self):
"pyspark.pandas.tests.connect.computation.test_parity_melt",
"pyspark.pandas.tests.connect.computation.test_parity_missing_data",
"pyspark.pandas.tests.connect.computation.test_parity_pivot",
],
excluded_python_implementations=[
"PyPy" # Skip these tests under PyPy since they require numpy, pandas, and pyarrow and
# they aren't available there
],
)


pyspark_pandas_slow_connect = Module(
name="pyspark-pandas-slow-connect",
dependencies=[pyspark_connect, pyspark_pandas, pyspark_pandas_slow],
Copy link
Contributor

@zhengruifeng zhengruifeng Jul 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should also set dependencies=[pyspark_connect, pyspark_pandas, pyspark_pandas_slow], in the above pyspark_pandas_connect module

Copy link
Contributor Author

@panbingkun panbingkun Jul 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

source_file_regexes=[
"python/pyspark/pandas",
],
python_test_goals=[
# pandas-on-Spark unittests
"pyspark.pandas.tests.connect.frame.test_parity_attrs",
"pyspark.pandas.tests.connect.frame.test_parity_constructor",
"pyspark.pandas.tests.connect.frame.test_parity_conversion",
Expand Down