[SPARK-31730][CORE][TEST] Fix flaky tests in BarrierTaskContextSuite

[jiangxb1987]

What changes were proposed in this pull request?

To wait until all the executors have started before submitting any job. This could avoid the flakiness caused by waiting for executors coming up.

How was this patch tested?

Existing tests.

[dongjoon-hyun]

Thank you so much, @jiangxb1987 .

[Ngone51]

LGTM

[SparkQA]

Test build #122856 has finished for PR 28584 at commit b08a272.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jiangxb1987]

"SPARK-31485: barrier stage should fail if only partial tasks are launched" This test case is flaky, @Ngone51 can you take a look?

[Ngone51]

sure, thanks!

[SparkQA]

Test build #122860 has finished for PR 28584 at commit b08a272.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #122864 has finished for PR 28584 at commit b08a272.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[Ngone51]

retest this please.

[SparkQA]

Test build #122862 has finished for PR 28584 at commit b08a272.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #122865 has finished for PR 28584 at commit b08a272.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[WeichenXu123]

Also fix this test like:

  test("share messages with allGather() call") {
    val conf = new SparkConf()
      .setMaster("local-cluster[4, 1, 1024]")
      .setAppName("test-cluster")
    sc = new SparkContext(conf)
    val rdd = sc.makeRDD(1 to 4, 4)
    val rdd2 = rdd.barrier().mapPartitions { it =>
      val context = BarrierTaskContext.get()
      // Sleep for a random time before global sync.
      Thread.sleep(Random.nextInt(1000))
      // Pass partitionId message in
      val message: String = context.partitionId().toString
      val messages: Array[String] = context.allGather(message)
      Iterator.single(messages.toList)
    }
    // Take a sorted list of all the partitionId messages
    val messages_list = rdd2.collect()
    assert (messages_list.length === 4)
    for (messages <- messages_list) {
      assert (messages === (0 until 4).map(_.toString).toList)
    }
  }

[Ngone51]

Opened a separate PR #28596 to fix the test @WeichenXu123

[dongjoon-hyun]

Retest this please.

[dongjoon-hyun]

Since #28596 is merged, shall we restart to proceed this PR? Thanks.

[SparkQA]

Test build #123066 has finished for PR 28584 at commit b08a272.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[Ngone51]

ping @jiangxb1987 Could you please update this?

[SparkQA]

Test build #123106 has finished for PR 28584 at commit 83a34c2.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[Ngone51]

retest this please.

[SparkQA]

Test build #123110 has finished for PR 28584 at commit 83a34c2.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[Ngone51]

retest this please.

[SparkQA]

Test build #123126 has finished for PR 28584 at commit 83a34c2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jiangxb1987]

Retest this please.

[jiangxb1987]

Retest this please.

[SparkQA]

Test build #123134 has finished for PR 28584 at commit 83a34c2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jiangxb1987]

Retest this please.

[jiangxb1987]

Let's trigger one more time to ensure the suite is no longer flaky.

[SparkQA]

Test build #123138 has finished for PR 28584 at commit 83a34c2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jiangxb1987]

Retest this please.

[SparkQA]

Test build #123199 has finished for PR 28584 at commit 83a34c2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jiangxb1987]

Thanks, merged to master/3.0 !

[jiangxb1987]

Opened #28658 for branch-3.0