[SPARK-46283][INFRA] Remove `streaming-kinesis-asl` module from `MODULES_TO_TEST` for branch-3.x daily tests #44204

LuciferYang · 2023-12-06T08:41:39Z

What changes were proposed in this pull request?

After the merge of #43736, the master branch began testing the streaming-kinesis-asl module.

At the same time, because the daily test will reuse build_and_test.yml, the daily test of branch-3.x also began testing streaming-kinesis-asl.

However, in branch-3.x, the env ENABLE_KINESIS_TESTS is hard-coded as 1 in dev/sparktestsupport/modules.py:

spark/dev/sparktestsupport/modules.py

Lines 332 to 346 in 1321b4e

    
           streaming_kinesis_asl = Module( 
        
               name="streaming-kinesis-asl", 
        
               dependencies=[tags, core], 
        
               source_file_regexes=[ 
        
                   "connector/kinesis-asl/", 
        
                   "connector/kinesis-asl-assembly/", 
        
               ], 
        
               build_profile_flags=[ 
        
                   "-Pkinesis-asl", 
        
               ], 
        
               environ={"ENABLE_KINESIS_TESTS": "1"}, 
        
               sbt_test_goals=[ 
        
                   "streaming-kinesis-asl/test", 
        
               ], 
        
           )

which leads to the failure of the daily test of branch-3.x:

[info] org.apache.spark.streaming.kinesis.WithoutAggregationKinesisStreamSuite *** ABORTED *** (1 second, 14 milliseconds)
[info]   java.lang.Exception: Kinesis tests enabled using environment variable ENABLE_KINESIS_TESTS
[info] but could not find AWS credentials. Please follow instructions in AWS documentation
[info] to set the credentials in your system such that the DefaultAWSCredentialsProviderChain
[info] can find the credentials.
[info]   at org.apache.spark.streaming.kinesis.KinesisTestUtils$.getAWSCredentials(KinesisTestUtils.scala:258)
[info]   at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[info]   at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[info]   at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[info]   at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[info]   at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info]   at org.apache.spark.streaming.kinesis.KinesisStreamTests.$anonfun$beforeAll$1(KinesisStreamSuite.scala:61)
[info]   at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info]   at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[info]   at org.apache.spark.streaming.kinesis.KinesisStreamTests.runIfTestsEnabled(KinesisStreamSuite.scala:42)
[info]   at org.apache.spark.streaming.kinesis.KinesisStreamTests.beforeAll(KinesisStreamSuite.scala:59)
[info]   at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info]   at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info]   at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info]   at org.apache.spark.streaming.kinesis.KinesisStreamTests.org$scalatest$BeforeAndAfter$$super$run(KinesisStreamSuite.scala:42)
[info]   at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:273)
[info]   at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:271)
[info]   at org.apache.spark.streaming.kinesis.KinesisStreamTests.run(KinesisStreamSuite.scala:42)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info]   at java.lang.Thread.run(Thread.java:750)
[info] Test run org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite started
[info] Test org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite.testJavaKinesisDStreamBuilderOldApi started
[info] Test org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite.testJavaKinesisDStreamBuilder started
[info] Test run org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite finished: 0 failed, 0 ignored, 2 total, 0.244s
[info] ScalaTest
[info] Run completed in 8 seconds, 542 milliseconds.
[info] Total number of tests run: 31
[info] Suites: completed 4, aborted 4
[info] Tests: succeeded 31, failed 0, canceled 0, ignored 0, pending 0
[info] *** 4 SUITES ABORTED ***
[error] Error: Total 37, Failed 0, Errors 4, Passed 33
[error] Error during tests:
[error]   org.apache.spark.streaming.kinesis.WithoutAggregationKinesisBackedBlockRDDSuite
[error]   org.apache.spark.streaming.kinesis.WithAggregationKinesisBackedBlockRDDSuite
[error]   org.apache.spark.streaming.kinesis.WithAggregationKinesisStreamSuite
[error]   org.apache.spark.streaming.kinesis.WithoutAggregationKinesisStreamSuite
[error] (streaming-kinesis-asl / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 13 s, completed Dec 5, 2023 12:03:53 PM

This PR adds a conditional judgment for the build task:

if [[ "$MODULES_TO_TEST" == *"streaming-kinesis-asl"* ]] && [[ "${{ inputs.branch }}" =~ ^branch-3 ]]; then 
  MODULES_TO_TEST=${MODULES_TO_TEST//streaming-kinesis-asl, /}
fi

When MODULES_TO_TEST contains streaming-kinesis-asl and the test branch is not master, it removes the streaming-kinesis-asl, substring from MODULES_TO_TEST and reassigns it to MODULES_TO_TEST. This avoids the testing of the streaming-kinesis-asl module in branch-3.x.

Why are the changes needed?

Prevent the daily tests of branch-3.x from testing the streaming-kinesis-asl module.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Monitor GA after merged

Was this patch authored or co-authored using generative AI tooling?

No

.github/workflows/build_and_test.yml

This reverts commit b7acf54.

LuciferYang · 2023-12-06T09:15:12Z

cc @HyukjinKwon @dongjoon-hyun @zhengruifeng

If we don't want to backport #43736 to branch-3.x, I think this PR could be a possible solution.

.github/workflows/build_and_test.yml

Co-authored-by: Hyukjin Kwon <[email protected]>

junyuc25 · 2023-12-06T12:30:07Z

Thank you for fixing this :).

LuciferYang · 2023-12-06T15:39:03Z

.github/workflows/build_and_test.yml

        # Hive "other tests" test needs larger metaspace size based on experiment.
        if [[ "$MODULES_TO_TEST" == "hive" ]] && [[ "$EXCLUDED_TAGS" == "org.apache.spark.tags.SlowHiveTest" ]]; then export METASPACE_SIZE=2g; fi
+        // SPARK-46283: should delete the following env replacement after SPARK 3.x EOL
+        if [[ "$MODULES_TO_TEST" == *"streaming-kinesis-asl"* ]] && [[ "${{ inputs.branch }}" =~ ^branch-3 ]]; then 


After reconsidering, I have changed this part to match branches that start with branch-3. This way, there is no need to change this conditional expression again after cutting branch-4.x

dongjoon-hyun

+1, LGTM.

LuciferYang · 2023-12-07T03:10:41Z

Thanks @dongjoon-hyun @HyukjinKwon @LuciferYang @junyuc25 ~

LuciferYang · 2023-12-07T15:28:21Z

The recent daily test has run successfully:

dongjoon-hyun · 2023-12-07T16:12:10Z

Great! Thank you for the updates, @LuciferYang .

…LES_TO_TEST` for branch-3.x daily tests ### What changes were proposed in this pull request? After the merge of apache#43736, the master branch began testing the `streaming-kinesis-asl` module. At the same time, because the daily test will reuse `build_and_test.yml`, the daily test of branch-3.x also began testing `streaming-kinesis-asl`. However, in branch-3.x, the env `ENABLE_KINESIS_TESTS` is hard-coded as 1 in `dev/sparktestsupport/modules.py`: https://github.com/apache/spark/blob/1321b4e64deaa1e58bf297c25b72319083056568/dev/sparktestsupport/modules.py#L332-L346 which leads to the failure of the daily test of branch-3.x: - branch-3.3: https://github.com/apache/spark/actions/runs/7111246311 - branch-3.4: https://github.com/apache/spark/actions/runs/7098435892 - branch-3.5: https://github.com/apache/spark/actions/runs/7099811235 ``` [info] org.apache.spark.streaming.kinesis.WithoutAggregationKinesisStreamSuite *** ABORTED *** (1 second, 14 milliseconds) [info] java.lang.Exception: Kinesis tests enabled using environment variable ENABLE_KINESIS_TESTS [info] but could not find AWS credentials. Please follow instructions in AWS documentation [info] to set the credentials in your system such that the DefaultAWSCredentialsProviderChain [info] can find the credentials. [info] at org.apache.spark.streaming.kinesis.KinesisTestUtils$.getAWSCredentials(KinesisTestUtils.scala:258) [info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58) [info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57) [info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168) [info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181) [info] at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84) [info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.$anonfun$beforeAll$1(KinesisStreamSuite.scala:61) [info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41) [info] at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39) [info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.runIfTestsEnabled(KinesisStreamSuite.scala:42) [info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.beforeAll(KinesisStreamSuite.scala:59) [info] at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212) [info] at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210) [info] at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208) [info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.org$scalatest$BeforeAndAfter$$super$run(KinesisStreamSuite.scala:42) [info] at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:273) [info] at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:271) [info] at org.apache.spark.streaming.kinesis.KinesisStreamTests.run(KinesisStreamSuite.scala:42) [info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321) [info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517) [info] at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414) [info] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [info] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [info] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [info] at java.lang.Thread.run(Thread.java:750) [info] Test run org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite started [info] Test org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite.testJavaKinesisDStreamBuilderOldApi started [info] Test org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite.testJavaKinesisDStreamBuilder started [info] Test run org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite finished: 0 failed, 0 ignored, 2 total, 0.244s [info] ScalaTest [info] Run completed in 8 seconds, 542 milliseconds. [info] Total number of tests run: 31 [info] Suites: completed 4, aborted 4 [info] Tests: succeeded 31, failed 0, canceled 0, ignored 0, pending 0 [info] *** 4 SUITES ABORTED *** [error] Error: Total 37, Failed 0, Errors 4, Passed 33 [error] Error during tests: [error] org.apache.spark.streaming.kinesis.WithoutAggregationKinesisBackedBlockRDDSuite [error] org.apache.spark.streaming.kinesis.WithAggregationKinesisBackedBlockRDDSuite [error] org.apache.spark.streaming.kinesis.WithAggregationKinesisStreamSuite [error] org.apache.spark.streaming.kinesis.WithoutAggregationKinesisStreamSuite [error] (streaming-kinesis-asl / Test / test) sbt.TestsFailedException: Tests unsuccessful [error] Total time: 13 s, completed Dec 5, 2023 12:03:53 PM ``` This PR adds a conditional judgment for the build task: ```shell if [[ "$MODULES_TO_TEST" == *"streaming-kinesis-asl"* ]] && [[ "${{ inputs.branch }}" =~ ^branch-3 ]]; then MODULES_TO_TEST=${MODULES_TO_TEST//streaming-kinesis-asl, /} fi ``` When `MODULES_TO_TEST` contains `streaming-kinesis-asl` and the test branch is not master, it removes the `streaming-kinesis-asl, ` substring from `MODULES_TO_TEST` and reassigns it to `MODULES_TO_TEST`. This avoids the testing of the `streaming-kinesis-asl` module in branch-3.x. ### Why are the changes needed? Prevent the daily tests of branch-3.x from testing the streaming-kinesis-asl module. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Monitor GA after merged ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#44204 from LuciferYang/SPARK-46283. Lead-authored-by: yangjie01 <[email protected]> Co-authored-by: YangJie <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

LuciferYang added 2 commits December 6, 2023 16:34

init

049550a

init

b0b2c3d

github-actions bot added the INFRA label Dec 6, 2023

for test

b7acf54

LuciferYang commented Dec 6, 2023

View reviewed changes

.github/workflows/build_and_test.yml Outdated Show resolved Hide resolved

LuciferYang mentioned this pull request Dec 6, 2023

[SPARK-32246][BUILD][INFRA] Enable streaming-kinesis-asl tests in Github Action CI #43736

Closed

Revert "for test"

f9e22e0

This reverts commit b7acf54.

HyukjinKwon reviewed Dec 6, 2023

View reviewed changes

.github/workflows/build_and_test.yml Outdated Show resolved Hide resolved

HyukjinKwon approved these changes Dec 6, 2023

View reviewed changes

LuciferYang and others added 5 commits December 6, 2023 18:37

Update .github/workflows/build_and_test.yml

ae40716

Co-authored-by: Hyukjin Kwon <[email protected]>

remove BRANCH

a6dee9f

test echo

0b00975

test echo

1a68951

revert

e120b8c

zhengruifeng approved these changes Dec 6, 2023

View reviewed changes

change to match start with branch-3

a026530

LuciferYang commented Dec 6, 2023

View reviewed changes

dongjoon-hyun approved these changes Dec 6, 2023

View reviewed changes

dongjoon-hyun closed this in 809cec0 Dec 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-46283][INFRA] Remove `streaming-kinesis-asl` module from `MODULES_TO_TEST` for branch-3.x daily tests #44204

[SPARK-46283][INFRA] Remove `streaming-kinesis-asl` module from `MODULES_TO_TEST` for branch-3.x daily tests #44204

Uh oh!

LuciferYang commented Dec 6, 2023 •

edited

Loading

Uh oh!

Uh oh!

LuciferYang commented Dec 6, 2023

Uh oh!

Uh oh!

junyuc25 commented Dec 6, 2023

Uh oh!

LuciferYang Dec 6, 2023

Uh oh!

dongjoon-hyun left a comment

Uh oh!

LuciferYang commented Dec 7, 2023

Uh oh!

LuciferYang commented Dec 7, 2023

Uh oh!

dongjoon-hyun commented Dec 7, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	streaming_kinesis_asl = Module(
	name="streaming-kinesis-asl",
	dependencies=[tags, core],
	source_file_regexes=[
	"connector/kinesis-asl/",
	"connector/kinesis-asl-assembly/",
	],
	build_profile_flags=[
	"-Pkinesis-asl",
	],
	environ={"ENABLE_KINESIS_TESTS": "1"},
	sbt_test_goals=[
	"streaming-kinesis-asl/test",
	],
	)

[SPARK-46283][INFRA] Remove streaming-kinesis-asl module from MODULES_TO_TEST for branch-3.x daily tests #44204

[SPARK-46283][INFRA] Remove streaming-kinesis-asl module from MODULES_TO_TEST for branch-3.x daily tests #44204

Uh oh!

Conversation

LuciferYang commented Dec 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

Uh oh!

LuciferYang commented Dec 6, 2023

Uh oh!

Uh oh!

junyuc25 commented Dec 6, 2023

Uh oh!

LuciferYang Dec 6, 2023

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Dec 7, 2023

Uh oh!

LuciferYang commented Dec 7, 2023

Uh oh!

dongjoon-hyun commented Dec 7, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[SPARK-46283][INFRA] Remove `streaming-kinesis-asl` module from `MODULES_TO_TEST` for branch-3.x daily tests #44204

[SPARK-46283][INFRA] Remove `streaming-kinesis-asl` module from `MODULES_TO_TEST` for branch-3.x daily tests #44204

LuciferYang commented Dec 6, 2023 •

edited

Loading