[SPARK-44544][INFRA] Deduplicate `run_python_packaging_tests` #42146

zhengruifeng · 2023-07-25T03:37:24Z

What changes were proposed in this pull request?

it seems that run_python_packaging_tests requires some disk space and cause some pyspark modules fail, this PR is to make run_python_packaging_tests only enabled within pyspark-errors (which is the smallest pyspark test module)

Why are the changes needed?

1, it seems it is the run_python_packaging_tests that cause the No space left error;
2, the run_python_packaging_tests is tested in all pyspark-* test modules, should be deduplicated;

Does this PR introduce any user-facing change?

no, infra-only

How was this patch tested?

updated CI

HyukjinKwon · 2023-07-25T03:47:12Z

dev/run-tests.py

Yeah, we should probably deduplicate because it runs for every python module test. another way is just to add an env variable, and enable it in only one split.

I'm just checking whether disabling this test works.
will do so if this is the root cause.

From this workflow: https://github.com/panbingkun/spark/actions/runs/5653268655/job/15314215187 Look, it seems to be a problem with run_python_packaging_tests.

Because all the tests in pyspark-pandas-connect-part-3 have ended running, run_python_packaging_tests caused an error

I am not 100% sure about this, since in https://github.com/zhengruifeng/spark/actions/runs/5652363908/job/15311889590 I also disabled all the packaging tests. (you can check that there is no packaging tests in pyspark-core),
but pyspark-sql and pyspark-pandas-slow-connect still failed ...

I just rebase the PR to re-test whether disabling works

and I see packaging test in pyspark-core successed in https://github.com/apache/spark/actions/runs/5652170312/job/15311427056, so maybe packaging test itself is fine?

TBH, I don't figure out what is happening

From the test results, after disable run_python_packaging_tests and splitting, it was successful.
https://github.com/panbingkun/spark/actions/runs/5654907063/job/15318981268

zhengruifeng · 2023-07-25T12:35:13Z

It seems disabling run_python_packaging_tests works, let me add it back in a separate job

https://github.com/zhengruifeng/spark/actions/runs/5654026436/job/15316814903

https://github.com/zhengruifeng/spark/actions/runs/5654706264/job/15324666018

zhengruifeng · 2023-07-25T13:08:35Z

also cc @Yikun

HyukjinKwon · 2023-07-26T00:36:14Z

let me take a quick look at #42159 too. I think we can just remove some directory before running pip test.

panbingkun · 2023-07-26T01:03:21Z

@zhengruifeng @HyukjinKwon
Perhaps we can disable 'run_python_packaging_tests' first to make GA to quickly recover, and then handle this matter later on.

HyukjinKwon · 2023-07-26T01:58:42Z

yeah. let me take a quick look at #42159

fix bash again

zhengruifeng · 2023-07-26T06:20:44Z

@HyukjinKwon I think this PR is orthogonal to #42159:

this PR now aims to only trigger run_python_packaging_tests in one pyspark test module, it seem ok to resolve the CI failure;
[SPARK-44557][INFRA] Clean up untracked/ignored files before running pip packaging test in GitHub Actions #42159 is to reduce the disk requirement of run_python_packaging_tests, which can make the CI more stable, IIUC

shall we go with this PR first (after all tests pass)?

zhengruifeng · 2023-07-26T07:33:49Z

also cc @LuciferYang @dongjoon-hyun

### What changes were proposed in this pull request? it seems that `run_python_packaging_tests` requires some disk space and cause some pyspark modules fail, this PR is to make `run_python_packaging_tests` only enabled within `pyspark-errors` (which is the smallest pyspark test module) ![image](https://github.com/apache/spark/assets/7322292/2d37c141-15b8-4d9f-bfbd-4dd7782ab62e) ### Why are the changes needed? 1, it seems it is the `run_python_packaging_tests` that cause the `No space left` error; 2, the `run_python_packaging_tests` is tested in all `pyspark-*` test modules, should be deduplicated; ### Does this PR introduce _any_ user-facing change? no, infra-only ### How was this patch tested? updated CI Closes #42146 from zhengruifeng/infra_skip_py_packing_tests. Authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]> (cherry picked from commit 748eaff) Signed-off-by: Ruifeng Zheng <[email protected]>

zhengruifeng · 2023-07-26T07:53:41Z

all python tests passed, merged to master/branch-3.5

LuciferYang

late LGTM

panbingkun · 2023-07-26T08:29:54Z

all python tests passed, merged to master/branch-3.5

@zhengruifeng Maybe branch-3.4 also need it?
https://github.com/panbingkun/spark/actions/runs/5665345967/job/15350140605

zhengruifeng · 2023-07-26T12:04:22Z

@panbingkun good catch! Let me backport it to 3.4

### What changes were proposed in this pull request? cherry-pick #42146 to 3.4 ### Why are the changes needed? can not cherry-pick clearly, so make this PR ### Does this PR introduce _any_ user-facing change? no, infra-only ### How was this patch tested? updated CI Closes #42172 from zhengruifeng/cp_fix. Authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

### What changes were proposed in this pull request? run `run_python_packaging_tests` when there are any changes in PySpark ### Why are the changes needed? #42146 make CI run `run_python_packaging_tests` only within `pyspark-errors` (see https://github.com/apache/spark/actions/runs/5666118302/job/15359190468 and https://github.com/apache/spark/actions/runs/5668071930/job/15358091003) ![image](https://github.com/apache/spark/assets/7322292/aef5cd4c-87ee-4b52-add3-e19ca131cdf1) but I ignored that `pyspark-errors` maybe skipped (because no related source changes), so the `run_python_packaging_tests` maybe also skipped unexpectedly (see https://github.com/apache/spark/actions/runs/5666523657/job/15353485731) ![image](https://github.com/apache/spark/assets/7322292/c2517d39-efcf-4a95-8562-1507dad35794) this PR is to run `run_python_packaging_tests` even if `pyspark-errors` is skipped ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? updated CI Closes #42173 from zhengruifeng/infra_followup. Lead-authored-by: Ruifeng Zheng <[email protected]> Co-authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>

### What changes were proposed in this pull request? run `run_python_packaging_tests` when there are any changes in PySpark ### Why are the changes needed? #42146 make CI run `run_python_packaging_tests` only within `pyspark-errors` (see https://github.com/apache/spark/actions/runs/5666118302/job/15359190468 and https://github.com/apache/spark/actions/runs/5668071930/job/15358091003) ![image](https://github.com/apache/spark/assets/7322292/aef5cd4c-87ee-4b52-add3-e19ca131cdf1) but I ignored that `pyspark-errors` maybe skipped (because no related source changes), so the `run_python_packaging_tests` maybe also skipped unexpectedly (see https://github.com/apache/spark/actions/runs/5666523657/job/15353485731) ![image](https://github.com/apache/spark/assets/7322292/c2517d39-efcf-4a95-8562-1507dad35794) this PR is to run `run_python_packaging_tests` even if `pyspark-errors` is skipped ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? updated CI Closes #42173 from zhengruifeng/infra_followup. Lead-authored-by: Ruifeng Zheng <[email protected]> Co-authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]> (cherry picked from commit f794734) Signed-off-by: Ruifeng Zheng <[email protected]>

…das-slow-connect GA testing time ### What changes were proposed in this pull request? The pr aims to balancing `pyspark-pandas-connect` and `pyspark-pandas-slow-connect` GA testing time. ### Why are the changes needed? After pr: #42146, the difference in testing time between `pyspark-pandas-connect` and `pyspark-pandas-slow-connect` is a bit significant, which affects the overall running time. In order to make GA operation more efficient and stable. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - Pass GA. - Manually monitor GA. Closes #42115 from panbingkun/free_disk_space. Lead-authored-by: panbingkun <[email protected]> Co-authored-by: panbingkun <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>

### What changes were proposed in this pull request? cherry-pick apache#42146 to 3.4 ### Why are the changes needed? can not cherry-pick clearly, so make this PR ### Does this PR introduce _any_ user-facing change? no, infra-only ### How was this patch tested? updated CI Closes apache#42172 from zhengruifeng/cp_fix. Authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

### What changes were proposed in this pull request? run `run_python_packaging_tests` when there are any changes in PySpark ### Why are the changes needed? apache#42146 make CI run `run_python_packaging_tests` only within `pyspark-errors` (see https://github.com/apache/spark/actions/runs/5666118302/job/15359190468 and https://github.com/apache/spark/actions/runs/5668071930/job/15358091003) ![image](https://github.com/apache/spark/assets/7322292/aef5cd4c-87ee-4b52-add3-e19ca131cdf1) but I ignored that `pyspark-errors` maybe skipped (because no related source changes), so the `run_python_packaging_tests` maybe also skipped unexpectedly (see https://github.com/apache/spark/actions/runs/5666523657/job/15353485731) ![image](https://github.com/apache/spark/assets/7322292/c2517d39-efcf-4a95-8562-1507dad35794) this PR is to run `run_python_packaging_tests` even if `pyspark-errors` is skipped ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? updated CI Closes apache#42173 from zhengruifeng/infra_followup. Lead-authored-by: Ruifeng Zheng <[email protected]> Co-authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]> (cherry picked from commit f794734) Signed-off-by: Ruifeng Zheng <[email protected]>

github-actions bot added the BUILD label Jul 25, 2023

HyukjinKwon reviewed Jul 25, 2023

View reviewed changes

github-actions bot added the INFRA label Jul 25, 2023

zhengruifeng changed the title ~~[WIP][DO_NOT_MERGE][INFRA] Disable python packaging tests~~ [WIP][DO_NOT_MERGE][INFRA] Move python packaging tests to a separate test module Jul 25, 2023

zhengruifeng changed the title ~~[WIP][DO_NOT_MERGE][INFRA] Move python packaging tests to a separate test module~~ [WIP][DO_NOT_MERGE][INFRA] Move python packaging tests to a separate module Jul 25, 2023

zhengruifeng force-pushed the infra_skip_py_packing_tests branch 2 times, most recently from caf49cd to 7bd48e9 Compare July 25, 2023 07:42

github-actions bot removed the INFRA label Jul 25, 2023

zhengruifeng force-pushed the infra_skip_py_packing_tests branch from 7bd48e9 to 35ccb99 Compare July 25, 2023 07:42

github-actions bot added the INFRA label Jul 25, 2023

zhengruifeng force-pushed the infra_skip_py_packing_tests branch from a1f5da4 to ba071c4 Compare July 25, 2023 12:47

zhengruifeng changed the title ~~[WIP][DO_NOT_MERGE][INFRA] Move python packaging tests to a separate module~~ [SPARK-44544][INFRA] Move python packaging tests to a separate module Jul 25, 2023

zhengruifeng mentioned this pull request Jul 26, 2023

[SPARK-44524][BUILD] Balancing pyspark-pandas-connect and pyspark-pandas-slow-connect GA testing time #42115

Closed

zhengruifeng added 5 commits July 26, 2023 09:51

init

37b9be4

del conda install

6e1e79a

add packaging test back

7ee9003

del exports

e4bebaf

refine

d9dcf5a

simplify

134a4ff

zhengruifeng force-pushed the infra_skip_py_packing_tests branch from b204c5c to 134a4ff Compare July 26, 2023 02:49

zhengruifeng changed the title ~~[SPARK-44544][INFRA] Move python packaging tests to a separate module~~ [SPARK-44544][INFRA] Deduplicate run_python_packaging_tests Jul 26, 2023

fix bash

89d365f

zhengruifeng force-pushed the infra_skip_py_packing_tests branch from 604131c to c0a46fd Compare July 26, 2023 04:49

fix bash again

04c271e

fix bash again

zhengruifeng force-pushed the infra_skip_py_packing_tests branch from c0a46fd to 04c271e Compare July 26, 2023 04:50

HyukjinKwon approved these changes Jul 26, 2023

View reviewed changes

panbingkun approved these changes Jul 26, 2023

View reviewed changes

zhengruifeng closed this in 748eaff Jul 26, 2023

zhengruifeng deleted the infra_skip_py_packing_tests branch July 26, 2023 07:53

LuciferYang reviewed Jul 26, 2023

View reviewed changes

This was referenced Jul 26, 2023

[SPARK-44544][INFRA][3.4] Deduplicate run_python_packaging_tests #42172

Closed

[SPARK-44544][INFRA][FOLLOWUP] Force run run_python_packaging_tests #42173

Closed

[SPARK-44544][INFRA] Deduplicate run_python_packaging_tests #42146

[SPARK-44544][INFRA] Deduplicate run_python_packaging_tests #42146

Uh oh!

Conversation

zhengruifeng commented Jul 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

HyukjinKwon Jul 25, 2023

Choose a reason for hiding this comment

Uh oh!

zhengruifeng Jul 25, 2023

Choose a reason for hiding this comment

Uh oh!

panbingkun Jul 25, 2023

Choose a reason for hiding this comment

Uh oh!

zhengruifeng Jul 25, 2023

Choose a reason for hiding this comment

Uh oh!

zhengruifeng Jul 25, 2023

Choose a reason for hiding this comment

Uh oh!

panbingkun Jul 25, 2023

Choose a reason for hiding this comment

Uh oh!

zhengruifeng commented Jul 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhengruifeng commented Jul 25, 2023

Uh oh!

HyukjinKwon commented Jul 26, 2023

Uh oh!

panbingkun commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Jul 26, 2023

Uh oh!

zhengruifeng commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhengruifeng commented Jul 26, 2023

Uh oh!

zhengruifeng commented Jul 26, 2023

Uh oh!

LuciferYang left a comment

Choose a reason for hiding this comment

Uh oh!

panbingkun commented Jul 26, 2023

Uh oh!

zhengruifeng commented Jul 26, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[SPARK-44544][INFRA] Deduplicate `run_python_packaging_tests` #42146

[SPARK-44544][INFRA] Deduplicate `run_python_packaging_tests` #42146

zhengruifeng commented Jul 25, 2023 •

edited

Loading

zhengruifeng commented Jul 25, 2023 •

edited

Loading

panbingkun commented Jul 26, 2023 •

edited

Loading

zhengruifeng commented Jul 26, 2023 •

edited

Loading