[HUDI-5278] Support more conf to cluster procedure by KnightChess · Pull Request #7304 · apache/hudi

KnightChess · 2022-11-25T09:57:59Z

Change Logs

spark sql cluster procedure support new params: op, order_strategy, options

Impact

none

Risk level (write none, low medium or high below)

low

Documentation Update

will open doc pr to update

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieClusteringConfig.java

...ark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/RunClusteringProcedure.scala

leesf · 2022-11-26T04:05:46Z

.../hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestClusteringProcedure.scala

+      assert(1 == metaClient.getActiveTimeline.getCompletedReplaceTimeline.getInstants.count())
+      assert(0 == metaClient.getActiveTimeline.filterPendingReplaceTimeline().getInstants.count())
+
+      spark.sql(s"call run_clustering(table => '$tableName')")


missing test case for scheduleandexecute and invalid op?

scheduleandexecute is default, I will add invalid op case

oh, it already has invalid case checkExceptionContain(s"call run_clustering(table => '$tableName', op => 'null')")("Invalid value")

leesf · 2022-11-26T04:11:27Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieClusteringConfig.java

+    /**
+     * only execute then pending clustering plans
+     */
+    EXECUTE("execute"),


can we execute specific pending clustering plan instead of all pending clustering plans?

leesf · 2022-11-28T01:08:17Z

...ark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/RunClusteringProcedure.scala

+
+    pendingClustering = instantsStr match {
+      case Some(inst) =>
+        operator = ClusteringOperator.EXECUTE


here why we need set operator to EXECUTE but in line#144 we do not need?

Because the user does not specify the instants

I think we need check if users specify the instants with SCHEDULE and SCHEDULE_AND_EXECUTE, we should throw exception instead of set it to EXECUTE when specify instants.

please put the line operator = ClusteringOperator.EXECUTE below the line logInfo("No op") and please change logInfo("No op") to logInfo("No op and set it to EXECUTE with instants specified.")

the line can not put below logInfo("No op"), operator default is scheduleAndExecute, if user specific instants, it need be set to execute after check

KnightChess · 2022-11-28T15:20:41Z

@leesf can you help me resolve this issue, after lastest success ci code, the code has not change any all, and my local compile is success

leesf

LGTM

leesf · 2022-11-29T04:30:38Z

@KnightChess would you please rebase to latest master as I see the master has some code changed.

KnightChess · 2022-11-29T04:52:33Z

@leesf done, and I update the test to resolve Java CI compile eroor, but I still don't know why

KnightChess · 2022-11-29T05:03:08Z

flink moudle error, and I found

KnightChess · 2022-11-29T05:14:01Z

I will reopen ci after #7319 be merged

KnightChess · 2022-11-29T07:36:46Z

Don't merge, I meet some question in cluster

KnightChess · 2022-11-29T09:01:10Z

Don't merge, I meet some question in cluster

look like is something bug in our Internal version, open version is ok, no blocked

leesf · 2022-11-29T13:25:16Z

@hudi-bot run azure

hudi-bot · 2022-11-29T20:05:27Z

CI report:

20a511d UNKNOWN
3f8f9a4 Azure: FAILURE Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

KnightChess · 2022-11-30T01:46:33Z

@leesf @stream2000 thanks for review

boneanxs · 2023-06-01T02:29:02Z

.../hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestClusteringProcedure.scala

+     """.stripMargin)
+
+      val fileNum = 20
+      val numRecords = 400000


Hey @KnightChess, do we need so many files and records per file for this test? This test currently could cost much time.

KnightChess force-pushed the more-conf-cluster-procedure branch from 627aedf to 1b87e59 Compare November 25, 2022 11:52

stream2000 reviewed Nov 25, 2022

View reviewed changes

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieClusteringConfig.java Outdated Show resolved Hide resolved

stream2000 reviewed Nov 25, 2022

View reviewed changes

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieClusteringConfig.java Show resolved Hide resolved

stream2000 reviewed Nov 25, 2022

View reviewed changes

...ark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/RunClusteringProcedure.scala Outdated Show resolved Hide resolved

leesf reviewed Nov 26, 2022

View reviewed changes

leesf self-assigned this Nov 26, 2022

leesf reviewed Nov 28, 2022

View reviewed changes

KnightChess closed this Nov 28, 2022

KnightChess reopened this Nov 28, 2022

leesf closed this Nov 29, 2022

leesf reopened this Nov 29, 2022

leesf approved these changes Nov 29, 2022

View reviewed changes

leesf closed this Nov 29, 2022

leesf reopened this Nov 29, 2022

KnightChess force-pushed the more-conf-cluster-procedure branch from e839fd6 to 728191c Compare November 29, 2022 04:45

KnightChess force-pushed the more-conf-cluster-procedure branch from 728191c to 20a511d Compare November 29, 2022 05:13

[HUDI-5278]support more conf to cluster procedure

3f8f9a4

KnightChess force-pushed the more-conf-cluster-procedure branch from 20a511d to 3f8f9a4 Compare November 29, 2022 06:09

codope added area:sql SQL interfaces priority:high Significant impact; potential bugs labels Nov 29, 2022

codope changed the title ~~[HUDI-5278]support more conf to cluster procedure~~ [HUDI-5278] Support more conf to cluster procedure Nov 29, 2022

leesf merged commit 418091b into apache:master Nov 30, 2022

KnightChess deleted the more-conf-cluster-procedure branch November 30, 2022 01:46

vamshigv pushed a commit to vamshigv/hudi that referenced this pull request Dec 1, 2022

[HUDI-5278] Support more conf to cluster procedure (apache#7304)

e212347

satishkotha pushed a commit that referenced this pull request Dec 13, 2022

[HUDI-5278] Support more conf to cluster procedure (#7304)

a83916c

XuQianJin-Stars pushed a commit that referenced this pull request Feb 11, 2023

[HUDI-5278] Support more conf to cluster procedure (#7304)

ebf99a7

fengjian428 pushed a commit to fengjian428/hudi that referenced this pull request Apr 5, 2023

[HUDI-5278] Support more conf to cluster procedure (apache#7304)

64e2e2e

lvyanquan mentioned this pull request Apr 6, 2023

[HUDI-6041] add options input to Bootstrap Procedure for passing hudi properties #8387

Merged

4 tasks

lvyanquan mentioned this pull request Apr 13, 2023

[HUDI-6041] add options input to Bootstrap Procedure for passing hu… DTStack/hudi#5

Closed

boneanxs reviewed Jun 1, 2023

View reviewed changes

Conversation

KnightChess commented Nov 25, 2022

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KnightChess commented Nov 28, 2022

Uh oh!

leesf left a comment

Choose a reason for hiding this comment

Uh oh!

leesf commented Nov 29, 2022

Uh oh!

KnightChess commented Nov 29, 2022

Uh oh!

KnightChess commented Nov 29, 2022

Uh oh!

KnightChess commented Nov 29, 2022

Uh oh!

KnightChess commented Nov 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KnightChess commented Nov 29, 2022

Uh oh!

leesf commented Nov 29, 2022

Uh oh!

hudi-bot commented Nov 29, 2022

CI report:

Uh oh!

KnightChess commented Nov 30, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

KnightChess commented Nov 29, 2022 •

edited

Loading