[CELEBORN-1648] Refine AppUniqueId with UUID suffix #2810

chenkovsky · 2024-10-14T13:19:17Z

What changes were proposed in this pull request?

We can add randomUUID as an suffix to solve it

Why are the changes needed?

currently, we cannot guarantee application id is really unique. this may lead to data issue.

Does this PR introduce any user-facing change?

No

How was this patch tested?

test locally

SteNicholas · 2024-10-14T13:38:09Z

common/src/main/scala/org/apache/celeborn/common/util/Utils.scala

  }
+
+  def appUniqueIdWithUUID(appUiqueId: String): String = {
+    appUiqueId + "-" + UUID.randomUUID()


Could you add config to allow user to add UUID suffix? IMO, not all cases should add UUID suffix. Right?

better to move appUniqueIdWithUUID(appUiqueId: String) to CelebornConf. Additionally, we can simplify the UUID by removing the dashes to create a shorter version, then we can use celebornConf.appUniqueIdWithUUIDSuffix

Suggested change

appUiqueId + "-" + UUID.randomUUID()

def appUniqueIdWithUUIDSuffix(appId: String): String = {

if (clientApplicationUUIDSuffixEnabled) {

appId + "-" + UUID.randomUUID().toString().replaceAll("-", "")

} else {

appId

}

}

SteNicholas · 2024-10-14T13:39:49Z

client-flink/common/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleMaster.java

        if (lifecycleManager == null) {
-          celebornAppId = FlinkUtils.toCelebornAppId(lifecycleManagerTimestamp, jobID);
+          celebornAppId =
+              Utils.appUniqueIdWithUUID(


Why does flink app id need to add UUID suffix?

so you think there's no application id colision problem for flink?

Yes. cc @reswqa, @RexXiong.

Yes. cc @reswqa, @RexXiong.

Currently Flink Appid is already random and concat with current timestamp, but I think it's fine if we add flag for UUID suffix

I didn't expect that flink app id(JobId + timestamp) might conflict. Did you actually suffer from this problem.

@SteNicholas please review it again, I reverted the change.

RexXiong · 2024-10-15T01:19:13Z

common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala

      .createWithDefault(0.4)

+  val CLIENT_APPLICATION_UUID: ConfigEntry[Boolean] =
+    buildConf("celeborn.client.application.uuid.enabled")


perfer celeborn.client.application.uuidSuffix.enabled

RexXiong · 2024-10-15T01:20:18Z

common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala

  def clientExcludeReplicaOnFailureEnabled: Boolean =
    get(CLIENT_EXCLUDE_PEER_WORKER_ON_FAILURE_ENABLED)
  def clientMrMaxPushData: Long = get(CLIENT_MR_PUSH_DATA_MAX)
+  def clientApplicationUUID: Boolean = get(CLIENT_APPLICATION_UUID)


clientApplicationUUID -> clientApplicationUUIDSuffixEnabled

RexXiong · 2024-10-15T01:30:21Z

common/src/main/scala/org/apache/celeborn/common/util/Utils.scala

  }
+
+  def appUniqueIdWithUUID(appUiqueId: String): String = {
+    appUiqueId + "-" + UUID.randomUUID()


better to move appUniqueIdWithUUID(appUiqueId: String) to CelebornConf. Additionally, we can simplify the UUID by removing the dashes to create a shorter version, then we can use celebornConf.appUniqueIdWithUUIDSuffix

Suggested change

appUiqueId + "-" + UUID.randomUUID()

def appUniqueIdWithUUIDSuffix(appId: String): String = {

if (clientApplicationUUIDSuffixEnabled) {

appId + "-" + UUID.randomUUID().toString().replaceAll("-", "")

} else {

appId

}

}

SteNicholas · 2024-10-15T07:25:20Z

common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala

+    buildConf("celeborn.client.application.uuid.enabled")
+      .categories("client")
+      .version("0.6.0")
+      .doc("When `true`, add uuid suffix to application id")


Suggested change

.doc("When `true`, add uuid suffix to application id")

.doc("Whether to add UUID suffix for application id for unique. When `true`, add UUID suffix for unique application id.")

SteNicholas · 2024-10-15T07:27:03Z

client-flink/common/src/main/java/org/apache/celeborn/plugin/flink/RemoteShuffleMaster.java

        if (lifecycleManager == null) {
-          celebornAppId = FlinkUtils.toCelebornAppId(lifecycleManagerTimestamp, jobID);
-          LOG.info("CelebornAppId: {}", celebornAppId);
+          String applicationId = FlinkUtils.toCelebornAppId(lifecycleManagerTimestamp, jobID);


Please revert this change. Meanwhile, remove line 87~89.

Please revert this change. Meanwhile, remove line 87~89.

ok, let's revert it.
should we describe that the flag is only for spark and mr in doc ?

SteNicholas · 2024-10-15T07:27:17Z

...link-1.20/src/main/java/org/apache/celeborn/plugin/flink/tiered/CelebornTierMasterAgent.java

      synchronized (CelebornTierMasterAgent.class) {
        if (lifecycleManager == null) {
-          celebornAppId = FlinkUtils.toCelebornAppId(lifecycleManagerTimestamp, jobID);
+          String applicationId = FlinkUtils.toCelebornAppId(lifecycleManagerTimestamp, jobID);


RexXiong

LGTM, only a minor comment

RexXiong · 2024-10-17T03:48:44Z

common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala

+    buildConf("celeborn.client.application.uuidSuffix.enabled")
+      .categories("client")
+      .version("0.6.0")
+      .doc("Whether to add UUID suffix for application id for unique. When `true`, add UUID suffix for unique application id.")


This configuration only takes effect when using the Spark and MR engines, so we must specify this in doc.

RexXiong · 2024-10-17T05:36:59Z

common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala

+    buildConf("celeborn.client.application.uuidSuffix.enabled")
+      .categories("client")
+      .version("0.6.0")
+      .doc("Whether to add UUID suffix for application id for unique. When `true`, add UUID suffix for unique application id. Currently, this only applies to Spark and MapPartition.")


MapPartition?

reswqa

Thanks for the update.

RexXiong · 2024-10-17T09:39:09Z

Thanks, merge to main(v0.6.0)

### What changes were proposed in this pull request? We can add randomUUID as an suffix to solve it ### Why are the changes needed? currently, we cannot guarantee application id is really unique. this may lead to data issue. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? test locally Closes apache#2810 from chenkovsky/feature/uuid_appid. Authored-by: Chongchen Chen <[email protected]> Signed-off-by: Shuang <[email protected]>

mridulm · 2024-11-03T16:32:47Z

Circling back to this - is this a case where multiple application attempts were involved ? (in which case, using appId + attemptId is the right approach) - or when different application;s application id actually collided?

Given spark (and MR ?) use application id + attempt id for a few things already (like event files), I am trying to understand why UUID was necessary.

chenkovsky · 2024-11-04T02:22:16Z

Circling back to this - is this a case where multiple application attempts were involved ? (in which case, using appId + attemptId is the right approach) - or when different application;s application id actually collided?

Given spark (and MR ?) use application id + attempt id for a few things already (like event files), I am trying to understand why UUID was necessary.

Hi, @mridulm . In our cluster, most of time, applicationId is unique, but id collision will take place occasionally. when it happens, it's very hard to find the root cause of data missing problem. For spark or MR, user can also override appId, add uuid suffix is also a fool-proof design.

SteNicholas reviewed Oct 14, 2024

View reviewed changes

RexXiong reviewed Oct 15, 2024

View reviewed changes

SteNicholas reviewed Oct 15, 2024

View reviewed changes

[CELEBORN-1648] Refine AppUniqueId with UUID suffix

ff6a9e9

chenkovsky force-pushed the feature/uuid_appid branch from 09fe17a to ff6a9e9 Compare October 15, 2024 11:27

RexXiong approved these changes Oct 17, 2024

View reviewed changes

RexXiong reviewed Oct 17, 2024

View reviewed changes

[CELEBORN-1648] Update doc for UUID suffix

d7a1613

chenkovsky force-pushed the feature/uuid_appid branch from 655bba2 to d7a1613 Compare October 17, 2024 06:02

reswqa approved these changes Oct 17, 2024

View reviewed changes

RexXiong approved these changes Oct 17, 2024

View reviewed changes

RexXiong closed this in eb49ed7 Oct 17, 2024

SteNicholas mentioned this pull request Jul 9, 2025

[GLUTEN-10151][CELEBORN] Bump Celeborn version to 0.6.0 apache/incubator-gluten#10152

Merged

-    appUiqueId + "-" + UUID.randomUUID()
+  def appUniqueIdWithUUIDSuffix(appId: String): String = {
+     if (clientApplicationUUIDSuffixEnabled) {
+        appId + "-" + UUID.randomUUID().toString().replaceAll("-", "")
+     } else {
+       appId
+     }
+   }

	.doc("When `true`, add uuid suffix to application id")
	.doc("Whether to add UUID suffix for application id for unique. When `true`, add UUID suffix for unique application id.")

[CELEBORN-1648] Refine AppUniqueId with UUID suffix #2810

[CELEBORN-1648] Refine AppUniqueId with UUID suffix #2810

Uh oh!

Conversation

chenkovsky commented Oct 14, 2024

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SteNicholas Oct 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RexXiong left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

reswqa left a comment

Choose a reason for hiding this comment

Uh oh!

RexXiong commented Oct 17, 2024

Uh oh!

mridulm commented Nov 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chenkovsky commented Nov 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

SteNicholas Oct 14, 2024 •

edited

Loading

mridulm commented Nov 3, 2024 •

edited

Loading