[SPARK-44524][BUILD] Balancing pyspark-pandas-connect and pyspark-pandas-slow-connect GA testing time #42115

panbingkun · 2023-07-24T01:34:16Z

What changes were proposed in this pull request?

The pr aims to balancing pyspark-pandas-connect and pyspark-pandas-slow-connect GA testing time.

Why are the changes needed?

After pr: #42146, the difference in testing time between pyspark-pandas-connect and pyspark-pandas-slow-connect is a bit significant, which affects the overall running time. In order to make GA operation more efficient and stable.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GA.
Manually monitor GA.

LuciferYang · 2023-07-24T08:39:29Z

It seems that splitting the test task is useful. Do we have to split it into 3?

LuciferYang · 2023-07-24T08:40:03Z

cc @HyukjinKwon @zhengruifeng FYI

…nect module

zhengruifeng

LGTM if CI pass

panbingkun · 2023-07-25T01:13:38Z

Maybe splitting into three is a stable choice. Let's try splitting into two first. Yesterday, splitting into three was successful.

LuciferYang · 2023-07-25T03:24:13Z

If dividing into two groups is still easy to fail, I do not oppose splitting into three groups :)

zhengruifeng · 2023-07-25T03:26:25Z

If dividing into two groups is still easy to fail, I do not oppose splitting into three groups :)

If so, I suggest we'd better find better names for the groups (let's also take existing pyspark_pandas_connect into account), maybe pyspark_pandas_connect_part_xxx

Yeah, let's probably go with #42146 way.

panbingkun · 2023-07-25T05:33:46Z

If so, I suggest we'd better find better names for the groups (let's also take existing pyspark_pandas_connect into account), maybe pyspark_pandas_connect_part_xxx

Ok, let me try again.

…nect module

dismiss

zhengruifeng · 2023-07-25T07:58:56Z

I think it is fine to split pyspark-pandas-slow-connect which is slow.

not sure whether they are due to the same cause, but I also see pyspark-sql, pyspark-pandas-slow failed for some reason (e.g. https://github.com/apache/spark/actions/runs/5651977344/job/15310946651)

It's kind of dangerous, since the GA recognized them as skip and made CI succeeded.

LuciferYang · 2023-07-25T08:00:54Z

It's kind of dangerous, since the GA recognized them as skip and made CI succeeded.

...

zhengruifeng · 2023-07-26T00:49:11Z

@panbingkun yes, it seems the python packaging test is the blocker, please hold on this PR, let's fix it in #42146 first.

zhengruifeng · 2023-07-26T07:59:05Z

@panbingkun after we disable packaging tests in pyspark-pandas-connect and pyspark-pandas-slow-connect, in https://github.com/zhengruifeng/spark/actions/runs/5664781463/job/15348941550 , the pyspark-pandas-connect took 1h, and pyspark-pandas-slow-connect took 2h.
I think it is not worthwhile to split pyspark-pandas-slow-connect any more, instead I suggest rebalancing them by moving some tests from pyspark-pandas-slow-connect to pyspark-pandas-connect.

panbingkun · 2023-07-26T08:04:51Z

@panbingkun after we disable packaging tests in pyspark-pandas-connect and pyspark-pandas-slow-connect, in https://github.com/zhengruifeng/spark/actions/runs/5664781463/job/15348941550 , the pyspark-pandas-connect took 1h, and pyspark-pandas-slow-connect took 2h. I think it is not worthwhile to split pyspark-pandas-slow-connect any more, instead I suggest rebalancing them by moving some tests from pyspark-pandas-slow-connect to pyspark-pandas-connect.

Ok, let me turn this PR into the logic of rebalancing? 😄

zhengruifeng · 2023-07-26T08:05:39Z

@panbingkun after we disable packaging tests in pyspark-pandas-connect and pyspark-pandas-slow-connect, in https://github.com/zhengruifeng/spark/actions/runs/5664781463/job/15348941550 , the pyspark-pandas-connect took 1h, and pyspark-pandas-slow-connect took 2h. I think it is not worthwhile to split pyspark-pandas-slow-connect any more, instead I suggest rebalancing them by moving some tests from pyspark-pandas-slow-connect to pyspark-pandas-connect.

Ok, let me turn this PR into the logic of rebalancing? 😄

yeah, many thanks!

panbingkun · 2023-07-26T08:11:44Z

@HyukjinKwon @zhengruifeng @LuciferYang
In addition, there may be a potential fix point needed in run_python_packaging_tests testing, as it may affect the final release as 2023-09-26 time approaches.
Are we submitting a PR separately? Or did you just make it on this PR?
I prefer a separate PR because it may require backport to branch-3.3 & branch-3.4.

spark/python/setup.cfg

Line 22 in 748eaff

description-file = README.md

zhengruifeng · 2023-07-26T08:16:37Z

@panbingkun I see, what about fixing it after 3.5 is released?

panbingkun · 2023-07-26T08:23:44Z

@panbingkun I see, what about fixing it after 3.5 is released?

Okay, fine to me.

zhengruifeng · 2023-07-27T08:32:15Z

please update this PR to latest master, since we just enabled > 100 test cases in 9b40c0c and 185a0a5

panbingkun · 2023-07-27T08:54:03Z

please update this PR to latest master, since we just enabled > 100 test cases in 9b40c0c and 185a0a5

Okay, done.

panbingkun · 2023-07-27T13:49:41Z

@zhengruifeng
After this modification, the testing time of pyspark-pandas-connect and pyspark-pandas-slow-connect has been basically balanced.

1.pyspark-pandas-connect

2.pyspark-pandas-slow-connect

dev/sparktestsupport/modules.py

Co-authored-by: Ruifeng Zheng <[email protected]>

zhengruifeng · 2023-07-28T09:23:02Z

dev/sparktestsupport/modules.py

+
+pyspark_pandas_slow_connect = Module(
+    name="pyspark-pandas-slow-connect",
+    dependencies=[pyspark_connect, pyspark_pandas, pyspark_pandas_slow],


should also set dependencies=[pyspark_connect, pyspark_pandas, pyspark_pandas_slow], in the above pyspark_pandas_connect module

panbingkun · 2023-07-28T10:32:46Z

Let me double check the testing time this time, it looks quite balanced:
1.pyspark-pandas-connect

2.pyspark-pandas-slow-connect

…das-slow-connect GA testing time

panbingkun · 2023-07-29T00:13:06Z

Third run cost:
1.pyspark-pandas-connect

2.pyspark-pandas-slow-connect

HyukjinKwon

I am good with this. cc @itholic, @ueshin and @xinrong-meng

zhengruifeng · 2023-07-31T01:13:51Z

thanks, merged to master

[TEST] free disk space

e759b1d

github-actions bot added BUILD INFRA labels Jul 24, 2023

panbingkun added 7 commits July 24, 2023 09:47

[TEST] free disk space

87e4b9e

[TEST] free disk space

49588d9

[TEST] free disk space

76b1de7

[TEST] free disk space

78d9019

[TEST] free disk space

9b293c1

[TEST] free disk space

90b8207

[TEST] free disk space

58fd90c

[SPARK-44524][BUILD] Add a new test group for pyspark-pandas-slow-con…

ded1def

…nect module

panbingkun changed the title ~~[DO NOT Merge][TEST] free disk space~~ [SPARK-44524][BUILD] Add a new test group for pyspark-pandas-slow-connect module Jul 24, 2023

HyukjinKwon previously approved these changes Jul 24, 2023

View reviewed changes

[SPARK-44524][BUILD] Add a new test group for pyspark-pandas-slow-con…

ac255f6

…nect module

LuciferYang mentioned this pull request Jul 24, 2023

[SPARK-44522][BUILD] Upgrade scala-xml to 2.2.0 #42119

Closed

zhengruifeng previously approved these changes Jul 25, 2023

View reviewed changes

panbingkun added 4 commits July 25, 2023 13:48

[SPARK-44524][BUILD] Add a new test group for pyspark-pandas-slow-con…

095c808

…nect module

Merge branch 'master' into free_disk_space

45436d0

[SPARK-44524][BUILD] Add a new test group for pyspark-pandas-slow-con…

5fd4cba

…nect module

[SPARK-44524][BUILD] Add a new test group for pyspark-pandas-slow-con…

3f472e0

…nect module

panbingkun added 2 commits July 26, 2023 19:15

Merge branch 'master' into free_disk_space

b892841

rebalancing pyspark-pandas-connect and pyspark-pandas-slow-connect

5f93b32

github-actions bot removed the INFRA label Jul 26, 2023

panbingkun marked this pull request as draft July 26, 2023 11:36

panbingkun changed the title ~~[SPARK-44524][BUILD] Add a new test group for pyspark-pandas-slow-connect module~~ [SPARK-44524][BUILD] Balancing pyspark-pandas-connect and pyspark-pandas-slow-connect GA testing time Jul 26, 2023

panbingkun added 2 commits July 27, 2023 16:24

rebalancing pyspark-pandas-connect and pyspark-pandas-slow-connect

b927af0

Merge branch 'master' into free_disk_space

e1e85fa

Merge branch 'master' into free_disk_space

4266e89

panbingkun marked this pull request as ready for review July 27, 2023 13:49

zhengruifeng reviewed Jul 28, 2023

View reviewed changes

dev/sparktestsupport/modules.py Outdated Show resolved Hide resolved

dev/sparktestsupport/modules.py Outdated Show resolved Hide resolved

Apply suggestions from code review

53fe574

Co-authored-by: Ruifeng Zheng <[email protected]>

zhengruifeng reviewed Jul 28, 2023

View reviewed changes

[SPARK-44524][BUILD] Balancing pyspark-pandas-connect and pyspark-pan…

7846ae2

…das-slow-connect GA testing time

HyukjinKwon approved these changes Jul 30, 2023

View reviewed changes

itholic approved these changes Jul 31, 2023

View reviewed changes

zhengruifeng closed this in be95298 Jul 31, 2023

[SPARK-44524][BUILD] Balancing pyspark-pandas-connect and pyspark-pandas-slow-connect GA testing time #42115

[SPARK-44524][BUILD] Balancing pyspark-pandas-connect and pyspark-pandas-slow-connect GA testing time #42115

Uh oh!

Conversation

panbingkun commented Jul 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

LuciferYang commented Jul 24, 2023

Uh oh!

LuciferYang commented Jul 24, 2023

Uh oh!

zhengruifeng left a comment

Choose a reason for hiding this comment

Uh oh!

panbingkun commented Jul 25, 2023

Uh oh!

LuciferYang commented Jul 25, 2023

Uh oh!

zhengruifeng commented Jul 25, 2023

Uh oh!

panbingkun commented Jul 25, 2023

Uh oh!

zhengruifeng commented Jul 25, 2023

Uh oh!

LuciferYang commented Jul 25, 2023

Uh oh!

zhengruifeng commented Jul 26, 2023

Uh oh!

zhengruifeng commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

panbingkun commented Jul 26, 2023

Uh oh!

zhengruifeng commented Jul 26, 2023

Uh oh!

panbingkun commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhengruifeng commented Jul 26, 2023

Uh oh!

panbingkun commented Jul 26, 2023

Uh oh!

zhengruifeng commented Jul 27, 2023

Uh oh!

panbingkun commented Jul 27, 2023

Uh oh!

panbingkun commented Jul 27, 2023

Uh oh!

Uh oh!

Uh oh!

zhengruifeng Jul 28, 2023

Choose a reason for hiding this comment

Uh oh!

panbingkun Jul 28, 2023

Choose a reason for hiding this comment

Uh oh!

panbingkun commented Jul 28, 2023

Uh oh!

panbingkun commented Jul 29, 2023

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

zhengruifeng commented Jul 31, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

panbingkun commented Jul 24, 2023 •

edited

Loading

zhengruifeng commented Jul 26, 2023 •

edited

Loading

panbingkun commented Jul 26, 2023 •

edited

Loading