Commit eba4f5c
committed
[SPARK-37531][INFRA][PYTHON][TESTS] Use PyArrow 6.0.0 in Python 3.9 tests at GitHub Action job
### What changes were proposed in this pull request?
This PR aims to use `PyArrow 6.0.0` in `Python 3.9` unit tests at GitHub Action jobs.
Although the new change is removing `<5.0.0' limitation, there are other minor changes because it's built more recently, too.
- dongjoon-hyun/ApacheSparkGitHubActionImage@4f7408f
```
- RUN python3.9 -m pip install numpy 'pyarrow<5.0.0' pandas scipy xmlrunner plotly>=4.8 sklearn 'mlflow>=1.0'
+ RUN python3.9 -m pip install numpy pyarrow pandas scipy xmlrunner plotly>=4.8 sklearn 'mlflow>=1.0'
```
```
$ docker run -it --rm dongjoon/apache-spark-github-action-image:20211116 pip3.9 list > 20211116
$ docker run -it --rm dongjoon/apache-spark-github-action-image:20210930 pip3.9 list > 20210930
$ diff 20210930 20211116
# The following is manually formatted for simplicity.
...
Jinja2 3.0.1 3.0.3
mlflow 1.20.2 1.21.0
numpy 1.21.2 1.21.4
pandas 1.3.3 1.3.4
plotly 5.3.1 5.4.0
pyarrow 4.0.1 6.0.0
scikit-learn 1.0 1.0.1
scipy 1.7.1 1.7.2
```
### Why are the changes needed?
SPARK-37342 upgrade Apache Arrow to 6.0.0 in Java/Scala.
This is a corresponding upgrade in PySpark.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the GitHub Action.
Closes #34793 from dongjoon-hyun/SPARK-37531.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>1 parent f99e2e6 commit eba4f5c
1 file changed
+3
-3
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
254 | 254 | | |
255 | 255 | | |
256 | 256 | | |
257 | | - | |
| 257 | + | |
258 | 258 | | |
259 | 259 | | |
260 | 260 | | |
| |||
358 | 358 | | |
359 | 359 | | |
360 | 360 | | |
361 | | - | |
| 361 | + | |
362 | 362 | | |
363 | 363 | | |
364 | 364 | | |
| |||
425 | 425 | | |
426 | 426 | | |
427 | 427 | | |
428 | | - | |
| 428 | + | |
429 | 429 | | |
430 | 430 | | |
431 | 431 | | |
| |||
0 commit comments