Skip to content

Commit 911dcd3

Browse files
committed
[SPARK-33173][CORE][TESTS] Use eventually to check numOnTaskFailed in PluginContainerSuite
### What changes were proposed in this pull request? This PR aims to use `eventually` to fix the flakiness of the test case `SPARK-33088: executor failed tasks trigger plugin calls`. ### Why are the changes needed? The test case checks like the following. ```scala assert(TestSparkPlugin.executorPlugin.numOnTaskStart == 2) assert(TestSparkPlugin.executorPlugin.numOnTaskSucceeded == 0) assert(TestSparkPlugin.executorPlugin.numOnTaskFailed == 2) ``` Although first and second passed, the third can fail. - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-hive-2.3-jdk-11/lastCompletedBuild/testReport/org.apache.spark.internal.plugin/PluginContainerSuite/SPARK_33088__executor_failed_tasks_trigger_plugin_calls/ - https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129919/testReport/ ``` sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: 1 did not equal 2 at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472) at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471) at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231) at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295) at org.apache.spark.internal.plugin.PluginContainerSuite.$anonfun$new$8(PluginContainerSuite.scala:161) ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? This only improves the robustness. Closes #30072 from dongjoon-hyun/SPARK-33173. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 0411def commit 911dcd3

File tree

1 file changed

+5
-3
lines changed

1 file changed

+5
-3
lines changed

core/src/test/scala/org/apache/spark/internal/plugin/PluginContainerSuite.scala

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -156,9 +156,11 @@ class PluginContainerSuite extends SparkFunSuite with BeforeAndAfterEach with Lo
156156
case t: Throwable => // ignore exception
157157
}
158158

159-
assert(TestSparkPlugin.executorPlugin.numOnTaskStart == 2)
160-
assert(TestSparkPlugin.executorPlugin.numOnTaskSucceeded == 0)
161-
assert(TestSparkPlugin.executorPlugin.numOnTaskFailed == 2)
159+
eventually(timeout(10.seconds), interval(100.millis)) {
160+
assert(TestSparkPlugin.executorPlugin.numOnTaskStart == 2)
161+
assert(TestSparkPlugin.executorPlugin.numOnTaskSucceeded == 0)
162+
assert(TestSparkPlugin.executorPlugin.numOnTaskFailed == 2)
163+
}
162164
}
163165

164166
test("plugin initialization in non-local mode") {

0 commit comments

Comments
 (0)