ARROW-7717: [CI] Have nightly integration test for Spark's latest release #6316

kszucs · 2020-01-29T10:47:27Z

No description provided.

kszucs · 2020-01-29T10:47:54Z

@ursabot crossbow submit test-conda-python-3.6-spark-2.4.4

kszucs · 2020-01-29T10:48:22Z

AMD64 Conda Crossbow Submit (#88674) builder has been succeeded.

Revision: 0f4de52129403022aaee438b118132679e19a6a8

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-468

Task	Status
test-conda-python-3.6-spark-2.4.4

github-actions · 2020-01-29T11:01:33Z

https://issues.apache.org/jira/browse/ARROW-7717

kszucs · 2020-01-29T12:42:49Z

@ursabot crossbow submit test-conda-python-3.6-spark-2.4.4

kszucs · 2020-01-29T12:43:15Z

AMD64 Conda Crossbow Submit (#88694) builder has been succeeded.

Revision: 03a4cfd685c03aaa11924730c08e4f90b0167c14

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-469

Task	Status
test-conda-python-3.6-spark-2.4.4

kszucs · 2020-01-29T13:24:36Z

@BryanCutler neither 2.4.4 nor 2.4.5-rc1 spark versions seem compatible with arrow > 0.15.

Is arrow 0.16 going to be supported only for spark > 3?

BryanCutler · 2020-01-29T18:22:17Z

@BryanCutler neither 2.4.4 nor 2.4.5-rc1 spark versions seem compatible with arrow > 0.15.
Is arrow 0.16 going to be supported only for spark > 3?

@kszucs Spark versions 2.4.x need to have the env variable ARROW_PRE_0_15_IPC_FORMAT=1 to work with pyarrow >= 0.15. See also https://github.com/apache/spark/blob/master/docs/sql-pyspark-pandas-with-arrow.md#compatibility-setting-for-pyarrow--0150-and-spark-23x-24x

kszucs · 2020-01-29T18:58:38Z

Thanks Bryan, I’ll try that then.

kszucs · 2020-02-12T14:11:09Z

@ursabot crossbow submit test-conda-python-3.6-spark-2.4.4 test-conda-python-3.7-spark-master

kszucs · 2020-02-12T14:11:36Z

AMD64 Conda Crossbow Submit (#90452) builder has been succeeded.

Revision: ce2060550fdab643808cc8e4dfba4ae052771832

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-505

Task	Status
test-conda-python-3.6-spark-2.4.4
test-conda-python-3.7-spark-master

BryanCutler · 2020-02-12T20:08:45Z

@kszucs I did some local testing and I think there are a number of issues with getting Spark 2.4.4 tests working with the latest pyarrow and pandas >= 1.0.0. It might be better to setup a test for the upcoming release of Spark 3.0.0, the branch has been cut here https://github.com/apache/spark/tree/branch-3.0, that should work fine.

kszucs · 2020-02-12T20:41:48Z

SGTM

wesm · 2020-06-06T13:18:07Z

@BryanCutler @kszucs what do you think about setting up a Spark 3.x integration test?

kszucs · 2020-06-08T08:47:31Z

Definitely, I'm going to roll up this PR and create a jira for spark 3.0 integration testing.

kszucs · 2020-06-08T08:54:56Z

Or just try to add it in this PR.

kszucs · 2020-06-08T08:55:34Z

@github-actions crossbow submit spark

github-actions · 2020-06-08T08:59:00Z

Revision: e91d5c3

Submitted crossbow builds: ursa-labs/crossbow @ actions-297

Task	Status
test-conda-python-3.6-spark-2.4.6
test-conda-python-3.7-spark-branch-3.0
test-conda-python-3.8-spark-master

kszucs · 2020-06-08T09:53:18Z

@BryanCutler could you please fix the build errors? The master/3: the build error might need to be fixed on spark's side.

kszucs · 2020-06-08T09:56:33Z

@github-actions crossbow submit test-conda-python-3.6-spark-2.4.6

github-actions · 2020-06-08T10:08:04Z

Revision: a914eea

Submitted crossbow builds: ursa-labs/crossbow @ actions-298

Task	Status
test-conda-python-3.6-spark-2.4.6

BryanCutler · 2020-06-08T18:23:40Z

Yeah, definitely would be good to test against Spark 3.0. I'll have to work on a patch to handle some of the recent changes to Arrow Java to fix the build.

kszucs · 2020-06-08T19:29:32Z

Thanks You! Although Spark 2 seems to fail as well.

BryanCutler · 2020-06-08T20:16:11Z

Yup, the Spark 2 issues are probably due to the same changes - I'll take a look

BryanCutler · 2020-06-24T18:12:05Z

@kszucs I submitted a patch to fix Java compilation with Spark master and branch-3.0, and tested locally with the latest pyarrow so Spark integration tests should pass for these as of this morning.

For Spark branch-2.4, it will require applying a different patch with the testing script, but maybe we should rethink this strategy a little. Spark 2.4.x will not upgrade Arrow Java anymore, so it would be more correct to take the latest Spark 2.4.6 release out of the box and test that with pyarrow from master. Is that possible to setup, wdyt?

kszucs · 2020-06-24T20:08:17Z

@kszucs I submitted a patch to fix Java compilation with Spark master and branch-3.0, and tested locally with the latest pyarrow so Spark integration tests should pass for these as of this morning.

Thanks Bryan!

For Spark branch-2.4, it will require applying a different patch with the testing script, but maybe we should rethink this strategy a little. Spark 2.4.x will not upgrade Arrow Java anymore, so it would be more correct to take the latest Spark 2.4.6 release out of the box and test that with pyarrow from master. Is that possible to setup, wdyt?

I'm not sure that I understand you correctly :)

keep arrow master <=> spark branch-3.0
keep arrow master <=> spark master
and keep arrow master <=> spark 2.4.6 but execute only the pyarrow tests?

BryanCutler · 2020-06-24T22:19:47Z

I mean the current process for integration tests with the master branch is to build Spark with Arrow Java master, then run Java and Python tests. That process is good for Spark master and branch-3.0, but with Spark branch-2.4, it doesn't make too much sense to rebuild Spark with Arrow Java master. This is because the dependency on Arrow Java is pretty much frozen for branch-2.4 (unless there is a serious enough bug), but different versions of pyarrow can be used. It would be best to take the latest Spark 2.4.6 release out of the box and run the pyspark tests from the script with pyarrow from master. We are not really concerned about Java compatibility since it can't be upgraded anyway, but that still allows us to flush out any Python problems.

BryanCutler · 2020-07-01T17:08:56Z

@kszucs the nightly against Spark master have been passing. Do you think you could update this to just add the test against branch-3.0 and remove branch-2.4 for now? I'm not sure if the bot will pick it up from this PR, but I'll try to kick it off a test run.

BryanCutler · 2020-07-01T17:10:24Z

@github-actions crossbow submit test-conda-python-3.7-spark-branch-3.0

BryanCutler

LGTM, I think we could skip branch-2.4 for now and branch-3.0 should be working

BryanCutler · 2020-07-01T17:10:02Z

dev/tasks/tasks.yml

+      # use the master branch of spark, so prevent reusing any layers
+      run: --no-leaf-cache --env ARROW_PRE_0_15_IPC_FORMAT=1 conda-python-spark
+
+  test-conda-python-3.7-spark-branch-3.0:


looks like this should be named python-3.8 also

github-actions · 2020-07-01T17:14:11Z

Revision: a914eea

Submitted crossbow builds: ursa-labs/crossbow @ actions-371

Task	Status
test-conda-python-3.7-spark-branch-3.0

kszucs · 2020-07-01T21:13:40Z

@github-actions crossbow submit test-spark

github-actions · 2020-07-01T21:15:46Z

Revision: 55d9411

Submitted crossbow builds: ursa-labs/crossbow @ actions-372

Task	Status
test-conda-python-3.7-spark-branch-3.0
test-conda-python-3.8-spark-master

kszucs

+1, thanks @BryanCutler!

BryanCutler · 2020-07-02T16:53:44Z

This is great, thanks @kszucs !

kszucs force-pushed the spark-2 branch 2 times, most recently from ca545f9 to ce20605 Compare February 12, 2020 14:10

wesm force-pushed the master branch from 5fe5b88 to aa55967 Compare April 19, 2020 22:47

kszucs force-pushed the master branch from 1b71ca7 to 5093b80 Compare April 20, 2020 19:21

kszucs added 5 commits June 8, 2020 10:48

nightly test for spark's latest release

bde7d10

remove patches argument from spark installation script

bfbaea1

fix argument number in usage

fc0cd40

conditionally export ARROW_PRE_0_15_IPC_FORMAT

598319b

set as nightly

e91d5c3

kszucs force-pushed the spark-2 branch from eeba52f to e91d5c3 Compare June 8, 2020 08:54

set ARROW_PRE_0_15_IPC_FORMAT for spark 2 build

a914eea

BryanCutler mentioned this pull request Jun 24, 2020

[SPARK-32080][SPARK-31998][SQL] Simplify ArrowColumnVector ListArray accessor apache/spark#28915

Closed

BryanCutler reviewed Jul 1, 2020

View reviewed changes

remove spark 2.4

55d9411

kszucs commented Jul 2, 2020

View reviewed changes

kszucs closed this in 0d789ac Jul 2, 2020

asfimport mentioned this pull request Jul 2, 2020

[CI] Have nightly integration test for Spark's latest release #23959

Closed

ARROW-7717: [CI] Have nightly integration test for Spark's latest release #6316

ARROW-7717: [CI] Have nightly integration test for Spark's latest release #6316

Uh oh!

Conversation

kszucs commented Jan 29, 2020

Uh oh!

kszucs commented Jan 29, 2020

Uh oh!

kszucs commented Jan 29, 2020

Uh oh!

github-actions bot commented Jan 29, 2020

Uh oh!

kszucs commented Jan 29, 2020

Uh oh!

kszucs commented Jan 29, 2020

Uh oh!

kszucs commented Jan 29, 2020

Uh oh!

BryanCutler commented Jan 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kszucs commented Jan 29, 2020

Uh oh!

kszucs commented Feb 12, 2020

Uh oh!

kszucs commented Feb 12, 2020

Uh oh!

BryanCutler commented Feb 12, 2020

Uh oh!

kszucs commented Feb 12, 2020

Uh oh!

wesm commented Jun 6, 2020

Uh oh!

kszucs commented Jun 8, 2020

Uh oh!

kszucs commented Jun 8, 2020

Uh oh!

kszucs commented Jun 8, 2020

Uh oh!

github-actions bot commented Jun 8, 2020

Uh oh!

kszucs commented Jun 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kszucs commented Jun 8, 2020

Uh oh!

github-actions bot commented Jun 8, 2020

Uh oh!

BryanCutler commented Jun 8, 2020

Uh oh!

kszucs commented Jun 8, 2020

Uh oh!

BryanCutler commented Jun 8, 2020

Uh oh!

BryanCutler commented Jun 24, 2020

Uh oh!

kszucs commented Jun 24, 2020

Uh oh!

BryanCutler commented Jun 24, 2020

Uh oh!

BryanCutler commented Jul 1, 2020

Uh oh!

BryanCutler commented Jul 1, 2020

Uh oh!

BryanCutler left a comment

Choose a reason for hiding this comment

Uh oh!

BryanCutler Jul 1, 2020

Choose a reason for hiding this comment

Uh oh!

kszucs Jul 1, 2020

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 1, 2020

Uh oh!

kszucs commented Jul 1, 2020

Uh oh!

github-actions bot commented Jul 1, 2020

Uh oh!

kszucs left a comment

BryanCutler commented Jan 29, 2020 •

edited

Loading

kszucs commented Jun 8, 2020 •

edited

Loading