[HUDI-3475] Initialize hudi table management module #5926

yuzhaojing · 2022-06-21T08:53:28Z

prep step #6732

vinothchandar · 2022-07-20T01:01:05Z

@yuzhaojing - I have given @prasannarajaperumal full context on this. and he will take over this. and I ll check-in/chime in as well as needed

yuzhaojing · 2022-07-20T02:16:44Z

@yuzhaojing - I have given @prasannarajaperumal full context on this. and he will take over this. and I ll check-in/chime in as well as needed

Ok, will communicate with him.

yuzhaojing · 2022-08-02T08:41:42Z

@prasannarajaperumal Can you take a look for this pr?

prasannarajaperumal

On a more general note - how are these API calls triggered? TableManagementService would ideally monitor all the Hudi datasets and keep track of the various stats and have configurations to trigger each of these table management actions. Are you planning on doing this as a follow up?

prasannarajaperumal · 2022-09-02T04:35:28Z

...management-service/src/main/java/org/apache/hudi/table/management/entity/InstanceStatus.java

+
+package org.apache.hudi.table.management.entity;
+
+public enum InstanceStatus {


May be we can split State (SCHEDULED, RUNNING, COMPLETED) and Status (SUCESS, FAIL). Completed state can either be a success or fail.

prasannarajaperumal · 2022-09-02T04:37:11Z

...ment-service/src/main/java/org/apache/hudi/table/management/executor/BaseActionExecutor.java

+        break;
+      case FLINK:
+      default:
+        throw new IllegalStateException("Unexpected value: " + instance.getExecutionEngine());


Engine {0} not supported

prasannarajaperumal · 2022-09-02T04:44:25Z

...t-service/src/main/java/org/apache/hudi/table/management/executor/submitter/SparkEngine.java

+
+  @Override
+  public Map<String, String> getJobParams(Instance instance) {
+    Map<String, String> sparkParams = new HashMap<>();


We could possibly register a execution engine and config with a seperate API endpoint and use that engineConfig as an input to run a table service. ServiceConfig is not generalized to any engine today - would rather get this as a input through the rest API.

prasannarajaperumal · 2022-09-02T04:46:28Z

...table-management-service/src/main/java/org/apache/hudi/table/management/entity/Instance.java

+@ToString
+@NoArgsConstructor
+@AllArgsConstructor
+public class Instance {


Better name for this would TableManagementAction? Also rename Action as ActionType

prasannarajaperumal · 2022-09-02T04:48:35Z

...-service/src/main/java/org/apache/hudi/table/management/store/impl/RelationDBBasedStore.java

+
+import java.util.List;
+
+public class RelationDBBasedStore implements MetadataStore {


Ideally we want this to be built over the HudiTimeline persistence of RFC-36 (Metastore server). These 2 RFC's have persistence layer shared. I am okay to track this as an action item and proceed here.

prasannarajaperumal · 2022-09-02T04:50:50Z

hudi-table-management-service/pom.xml

+            <version>1.7.25</version>
+        </dependency>
+
+        <dependency>


Can you check on the license of this third-party? Seems like apache 2.0 (https://github.com/brettwooldridge/HikariCP/blob/dev/LICENSE) but want to make sure.

prasannarajaperumal · 2022-09-02T04:53:32Z

...table-management-service/src/main/java/org/apache/hudi/table/management/entity/Instance.java

+@ToString
+@NoArgsConstructor
+@AllArgsConstructor
+public class Instance {


JavaDoc and comments on the fields used here will be helpful.

xushiyan

roughly went through the code base and commented about some improvements. will continue review tmr

xushiyan · 2023-01-17T21:16:38Z

...-service-manager/src/main/java/org/apache/hudi/table/service/manager/util/DateTimeUtils.java

+import java.util.Calendar;
+import java.util.Date;
+
+public class DateTimeUtils {


can we consolidate this with org.apache.hudi.common.util.DateTimeUtils ? and pls add some UTs

xushiyan · 2023-01-17T21:18:33Z

...er/src/main/java/org/apache/hudi/table/service/manager/store/jdbc/SqlSessionFactoryUtil.java

+import java.sql.PreparedStatement;
+import java.util.stream.Collectors;
+
+public class SqlSessionFactoryUtil {


can we consolidate this with org.apache.hudi.metaserver.store.jdbc.SqlSessionFactoryUtils ? and more UTs?

In the follow-up PR, hudi-platform-common will be extracted for unification, let us follow up.

xushiyan · 2023-01-17T21:21:57Z

...e-manager/src/main/java/org/apache/hudi/table/service/manager/HoodieTableServiceManager.java

+  }
+
+  public static void main(String[] args) throws Exception {
+    System.out.println("SPARK_HOME = " + System.getenv("SPARK_HOME"));


can you review all occurrence of system.out/err.print and replace with logger ?

xushiyan · 2023-01-17T21:25:43Z

...able-service-manager/src/main/java/org/apache/hudi/table/service/manager/RequestHandler.java

+    @Override
+    public void handle(@NotNull Context context) throws Exception {
+      boolean success = true;
+      long beginTs = System.currentTimeMillis();


use org.apache.hudi.common.util.HoodieTimer as a standard for compute code execution time

xushiyan · 2023-01-18T05:14:32Z

...rvice-manager/src/main/java/org/apache/hudi/table/service/manager/store/jdbc/JdbcMapper.java

+import org.apache.ibatis.session.RowBounds;
+import org.apache.ibatis.session.SqlSession;
+
+public class JdbcMapper {


we should standardize jdbc interaction, at least for metaserver and TSM. We can have a follow up jira for this - having a module hudi-platform-service/hudi-platform-common for common components & classes maybe ?

xushiyan · 2023-01-18T05:18:51Z

...-manager/src/main/java/org/apache/hudi/table/service/manager/store/impl/InstanceService.java

+  public void updateExecutionInfo(Instance instance) {
+    int retryNum = 0;
+    try {
+      while (retryNum++ < 3) {


use retryhelper here?

Will update it.

xushiyan · 2023-01-18T05:24:31Z

...service-manager/src/main/java/org/apache/hudi/table/service/manager/service/BaseService.java

+
+  void init();
+
+  void startService();


why not just call it start() ? . :)

xushiyan · 2023-01-18T05:26:33Z

.../src/main/java/org/apache/hudi/table/service/manager/executor/submitter/ExecutionEngine.java

+
+import java.util.Map;
+
+public abstract class ExecutionEngine {


can you please review all new classes for TSM models and add javadoc to explain the use case ?

Sure, will add doc.

xushiyan · 2023-01-18T05:29:53Z

...ce-manager/src/main/java/org/apache/hudi/table/service/manager/entity/AssistQueryEntity.java

+
+import lombok.Getter;
+
+import java.util.Date;


pls avoid all java.util.Date, which is not thread safe. let's change all to java.time.* instead. Also for timestamp, can you see if better with all Long type?

xushiyan · 2023-01-18T05:32:51Z

...ervice-manager/src/main/java/org/apache/hudi/table/service/manager/common/CommandConfig.java

+  @Parameter(names = {"-instance-submit-timeout-sec"}, description = "Instance Submit Timeout Sec")
+  public Integer instanceSubmitTimeoutSec = 600;
+
+  @Parameter(names = {"-spark-submit-jar-path"}, description = "Spark Submit Jar Path")


this is spark-specific? it's under common/CommandConfig so it better be engine agnostic.

hudi-bot · 2023-01-18T07:56:07Z

CI report:

ed783b4 UNKNOWN
0919434 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

Zouxxyy · 2023-01-29T01:44:54Z

hudi-platform-service/hudi-table-service-manager/src/main/resources/mybatis/Instance.xml

+    <result column="update_time" property="updateTime" javaType="java.util.Date"/>
+  </resultMap>
+
+  <update id="createInstance">


It might be misunderstood with saveInstance below, maybe we can call it createInstanceTable?

Zouxxyy · 2023-01-29T02:23:13Z

...ervice-manager/src/main/java/org/apache/hudi/table/service/manager/common/CommandConfig.java

+  @Parameter(names = {"-instance-submit-timeout-sec"}, description = "Instance Submit Timeout Sec")
+  public Integer instanceSubmitTimeoutSec = 600;
+
+  @Parameter(names = {"-spark-submit-jar-path"}, description = "Spark Submit Jar Path")


Does spark-submit-jar refer to hudi-cli? Its parameters seem to be

sparkLauncher.addAppArgs("COMPACT_RUN", master, sparkMemory, instance.getBasePath(), instance.getTableName(), instance.getInstant(), parallelism, "", maxRetryNum, "")

Zouxxyy · 2023-01-29T02:34:45Z

...ager/src/main/java/org/apache/hudi/table/service/manager/executor/submitter/SparkEngine.java

+      sparkLauncher.setConf(entry.getKey(), entry.getValue());
+    }
+
+    sparkLauncher.addSparkArg("--queue", instance.getQueue());


Maybe we can use spark.yarn.queue and put it in the jobParams above?

Zouxxyy · 2023-01-29T02:37:16Z

...ager/src/main/java/org/apache/hudi/table/service/manager/executor/submitter/SparkEngine.java

+    //    sparkLauncher.addAppArgs(SparkCommand.COMPACT_RUN.toString(), master, sparkMemory, client.getBasePath(),
+    //        client.getTableConfig().getTableName(), compactionInstantTime, parallelism, schemaFilePath,
+    //        retry, propsFilePath);
+    sparkLauncher.addAppArgs("COMPACT_RUN", master, sparkMemory, instance.getBasePath(),


COMPACT_RUN might be set as a parameter

Zouxxyy · 2023-01-29T02:39:59Z

...ce-manager/src/main/java/org/apache/hudi/table/service/manager/entity/AssistQueryEntity.java

+
+  private int maxRetry;
+
+  private Date queryStartTime = DateTimeUtils.addDay(-3);


What does this -3 mean ~

Zouxxyy · 2023-01-29T02:44:50Z

...-manager/src/main/java/org/apache/hudi/table/service/manager/handlers/CompactionHandler.java

+      throw new RuntimeException("Instance not exist: " + instance);
+    }
+    // 2. update status
+    metadataStore.updateStatus(instance);


It feels like no changes have been made here

Zouxxyy · 2023-01-29T02:48:39Z

hudi-platform-service/hudi-table-service-manager/src/main/resources/mybatis/Instance.xml

+  <resultMap type="org.apache.hudi.table.service.manager.entity.Instance" id="InstanceMapping">
+    <result column="id" property="id" javaType="java.lang.Long"/>
+    <result column="db_name" property="dbName"/>
+    <result column="table_name" property="tableName"/>


Is the tbl table canceled? I noticed that tbl_id is used here in rfc

yuzhaojing requested review from XuQianJin-Stars, prashantwason, vinothchandar and xushiyan June 21, 2022 08:54

yuzhaojing added priority:blocker Production down; release blocker priority:high Significant impact; potential bugs big-needle-movers and removed priority:blocker Production down; release blocker labels Jun 21, 2022

yuzhaojing force-pushed the HUDI-3475 branch from 4dc0fc0 to 6a6b31a Compare June 28, 2022 07:23

yuzhaojing mentioned this pull request Jun 30, 2022

[HUDI-4148] Preparations and client for hudi table manager service #5681

Closed

5 tasks

vinothchandar assigned vinothchandar and prasannarajaperumal Jul 13, 2022

yuzhaojing force-pushed the HUDI-3475 branch from 6a6b31a to e94d6fc Compare August 1, 2022 14:04

xushiyan self-assigned this Aug 15, 2022

xushiyan requested a review from prasannarajaperumal August 15, 2022 13:57

prasannarajaperumal reviewed Sep 2, 2022

View reviewed changes

yihua added priority:critical Production degraded; pipelines stalled and removed priority:high Significant impact; potential bugs labels Sep 13, 2022

yuzhaojing force-pushed the HUDI-3475 branch 3 times, most recently from 4fbf616 to 9c63087 Compare January 5, 2023 13:56

yuzhaojing force-pushed the HUDI-3475 branch from 9c63087 to ed783b4 Compare January 17, 2023 10:07

[HUDI-3475] Initialize hudi table management module

0919434

yuzhaojing force-pushed the HUDI-3475 branch from a943461 to 0919434 Compare January 18, 2023 04:13

xushiyan requested changes Jan 18, 2023

View reviewed changes

Zouxxyy reviewed Jan 29, 2023

View reviewed changes

github-actions bot added the size:XL PR with lines of changes > 1000 label Feb 26, 2024

vinothchandar unassigned vinothchandar, xushiyan and prasannarajaperumal May 23, 2025

hudi-bot mentioned this pull request Dec 9, 2025

Initialize hudi table management module #15034

Open


		package org.apache.hudi.table.management.entity;

		public enum InstanceStatus {


		import java.util.List;

		public class RelationDBBasedStore implements MetadataStore {


		import java.util.Map;

		public abstract class ExecutionEngine {


		private int maxRetry;

		private Date queryStartTime = DateTimeUtils.addDay(-3);

[HUDI-3475] Initialize hudi table management module #5926

Are you sure you want to change the base?

[HUDI-3475] Initialize hudi table management module #5926

Conversation

yuzhaojing commented Jun 21, 2022 • edited by xushiyan Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vinothchandar commented Jul 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yuzhaojing commented Jul 20, 2022

Uh oh!

yuzhaojing commented Aug 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

prasannarajaperumal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xushiyan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hudi-bot commented Jan 18, 2023

CI report:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuzhaojing commented Jun 21, 2022 •

edited by xushiyan

Loading

vinothchandar commented Jul 20, 2022 •

edited

Loading

yuzhaojing commented Aug 2, 2022 •

edited

Loading