[HUDI-3112] Fix KafkaConnect can not sync to Hive Problem by cdmikechen · Pull Request #4458 · apache/hudi

cdmikechen · 2021-12-28T07:51:01Z

What is the purpose of the pull request

KafkaConnect use org.apache.hudi.DataSourceUtils to build HiveSyncConfig now, but DataSourceUtils import some spark dependencies. So that Hive sync will fail because of the application of related classes.

Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.types.DataType
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 66 more

Brief change log

Add a separate method to create HiveSyncConfig

Verify this pull request

Need to add Hive sync test by https://issues.apache.org/jira/browse/HUDI-2673

Committer checklist

Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

cdmikechen · 2022-01-02T12:02:38Z

@yihua Thanks for reviewing this

yihua · 2022-01-06T22:55:37Z

+  /**
+   * Build Hive Sync Config
+   */
+  public HiveSyncConfig buildSyncConfig(TypedProperties props, String tableBasePath) {


If the problem is due to irrelevant classes to Kafka Connect imported from DataSourceUtils, is it possible to move DataSourceUtils::buildHiveSyncConfig to a different/new util class so buildHiveSyncConfig() can still be reused here, instead of duplicating the code in hudi-kafka-connect module? Should that solve the problem?

@yihua
I had checked the Hudi project when I modified the codes. Besides hive synchronization in spark, Flink also has the same problem. However, in the flink, they also redeclared a new set of variables to solve the problem.

hudi/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java

Lines 576 to 610 in e9efbdb

public static final ConfigOption<String> HIVE_SYNC_MODE = ConfigOptions

.key("hive_sync.mode")

.stringType()

.defaultValue("jdbc")

.withDescription("Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql, default 'jdbc'");

public static final ConfigOption<String> HIVE_SYNC_USERNAME = ConfigOptions

.key("hive_sync.username")

.stringType()

.defaultValue("hive")

.withDescription("Username for hive sync, default 'hive'");

public static final ConfigOption<String> HIVE_SYNC_PASSWORD = ConfigOptions

.key("hive_sync.password")

.stringType()

.defaultValue("hive")

.withDescription("Password for hive sync, default 'hive'");

public static final ConfigOption<String> HIVE_SYNC_JDBC_URL = ConfigOptions

.key("hive_sync.jdbc_url")

.stringType()

.defaultValue("jdbc:hive2://localhost:10000")

.withDescription("Jdbc URL for hive sync, default 'jdbc:hive2://localhost:10000'");

public static final ConfigOption<String> HIVE_SYNC_METASTORE_URIS = ConfigOptions

.key("hive_sync.metastore.uris")

.stringType()

.defaultValue("")

.withDescription("Metastore uris for hive sync, default ''");

public static final ConfigOption<String> HIVE_SYNC_PARTITION_FIELDS = ConfigOptions

.key("hive_sync.partition_fields")

.stringType()

.defaultValue("")

.withDescription("Partition fields for hive sync, default ''");

Considering that if unification is a relatively large part of adjustment, it may be a better way to solve it with a new issue. Because there are some Scala logic in hive sync, it cannot be split directly.

@cdmikechen Understood. I'm thinking about only moving util methods related Hive sync configs, not the Hive sync logic, to a separate Util class. The worry I have is that hive sync configs are spread into different places now and they may diverge if we forget to update all of them to be consistent.

We can keep this PR as is for now. @cdmikechen could you create a Jira ticket to track the Hive sync config unification, which will be done in a different PR in future?

nsivabalan · 2022-01-08T15:42:30Z

@cdmikechen @yihua : We are targeting this patch for 0.10.1. We have code freeze planned this monday. Would be nice to get this in by then. Wanted to send out a reminder.

yihua · 2022-01-09T07:01:00Z

+  /**
+   * Build Hive Sync Config
+   */
+  public HiveSyncConfig buildSyncConfig(TypedProperties props, String tableBasePath) {


@cdmikechen Understood. I'm thinking about only moving util methods related Hive sync configs, not the Hive sync logic, to a separate Util class. The worry I have is that hive sync configs are spread into different places now and they may diverge if we forget to update all of them to be consistent.

We can keep this PR as is for now. @cdmikechen could you create a Jira ticket to track the Hive sync config unification, which will be done in a different PR in future?

yihua · 2022-01-09T07:04:06Z

+  /**
+   * Build Hive Sync Config
+   */
+  public HiveSyncConfig buildSyncConfig(TypedProperties props, String tableBasePath) {


Let's move this util method to KafkaConnectUtils class.

cdmikechen · 2022-01-09T09:32:36Z

@yihua
Hi~ I've moved buildSyncConfig method to KafkaConnectUtils and open a new issue https://issues.apache.org/jira/browse/HUDI-3199

hudi-bot · 2022-01-09T10:31:14Z

CI report:

8d1205d Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

yihua

LGTM

nsivabalan assigned yihua Jan 3, 2022

nsivabalan added the priority:critical Production degraded; pipelines stalled label Jan 3, 2022

yihua requested changes Jan 6, 2022

View reviewed changes

yihua approved these changes Jan 9, 2022

View reviewed changes

yihua requested changes Jan 9, 2022

View reviewed changes

[HUDI-3112] Fix KafkaConnect can not sync to Hive Problem

8d1205d

cdmikechen force-pushed the HUDI-3112 branch from c60da50 to 8d1205d Compare January 9, 2022 09:31

cdmikechen requested a review from yihua January 9, 2022 09:31

yihua approved these changes Jan 9, 2022

View reviewed changes

yihua merged commit e9a7f49 into apache:master Jan 9, 2022

nsivabalan pushed a commit that referenced this pull request Jan 11, 2022

[HUDI-3112] Fix KafkaConnect cannot sync to Hive Problem (#4458)

2b98d90

vinishjail97 mentioned this pull request Jan 24, 2022

FixIgnoreKey nsivabalan/hudi#11

Closed

5 tasks

vingov pushed a commit to vingov/hudi that referenced this pull request Jan 26, 2022

[HUDI-3112] Fix KafkaConnect cannot sync to Hive Problem (apache#4458)

9d72bbe

liusenhua pushed a commit to liusenhua/hudi that referenced this pull request Mar 1, 2022

[HUDI-3112] Fix KafkaConnect cannot sync to Hive Problem (apache#4458)

80ada81

vingov pushed a commit to vingov/hudi that referenced this pull request Apr 3, 2022

[HUDI-3112] Fix KafkaConnect cannot sync to Hive Problem (apache#4458)

3910ce9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HUDI-3112] Fix KafkaConnect can not sync to Hive Problem#4458

[HUDI-3112] Fix KafkaConnect can not sync to Hive Problem#4458
yihua merged 1 commit intoapache:masterfrom
cdmikechen:HUDI-3112

cdmikechen commented Dec 28, 2021 •

edited

Loading

Uh oh!

cdmikechen commented Jan 2, 2022

Uh oh!

yihua Jan 6, 2022

Uh oh!

cdmikechen Jan 7, 2022

Uh oh!

yihua Jan 9, 2022 •

edited

Loading

Uh oh!

nsivabalan commented Jan 8, 2022

Uh oh!

yihua Jan 9, 2022 •

edited

Loading

Uh oh!

yihua Jan 9, 2022

Uh oh!

cdmikechen commented Jan 9, 2022

Uh oh!

hudi-bot commented Jan 9, 2022

Uh oh!

yihua left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	public static final ConfigOption<String> HIVE_SYNC_MODE = ConfigOptions
	.key("hive_sync.mode")
	.stringType()
	.defaultValue("jdbc")
	.withDescription("Mode to choose for Hive ops. Valid values are hms, jdbc and hiveql, default 'jdbc'");

	public static final ConfigOption<String> HIVE_SYNC_USERNAME = ConfigOptions
	.key("hive_sync.username")
	.stringType()
	.defaultValue("hive")
	.withDescription("Username for hive sync, default 'hive'");

	public static final ConfigOption<String> HIVE_SYNC_PASSWORD = ConfigOptions
	.key("hive_sync.password")
	.stringType()
	.defaultValue("hive")
	.withDescription("Password for hive sync, default 'hive'");

	public static final ConfigOption<String> HIVE_SYNC_JDBC_URL = ConfigOptions
	.key("hive_sync.jdbc_url")
	.stringType()
	.defaultValue("jdbc:hive2://localhost:10000")
	.withDescription("Jdbc URL for hive sync, default 'jdbc:hive2://localhost:10000'");

	public static final ConfigOption<String> HIVE_SYNC_METASTORE_URIS = ConfigOptions
	.key("hive_sync.metastore.uris")
	.stringType()
	.defaultValue("")
	.withDescription("Metastore uris for hive sync, default ''");

	public static final ConfigOption<String> HIVE_SYNC_PARTITION_FIELDS = ConfigOptions
	.key("hive_sync.partition_fields")
	.stringType()
	.defaultValue("")
	.withDescription("Partition fields for hive sync, default ''");

Conversation

cdmikechen commented Dec 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the pull request

Brief change log

Verify this pull request

Committer checklist

Uh oh!

cdmikechen commented Jan 2, 2022

Uh oh!

yihua Jan 6, 2022

Choose a reason for hiding this comment

Uh oh!

cdmikechen Jan 7, 2022

Choose a reason for hiding this comment

Uh oh!

yihua Jan 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nsivabalan commented Jan 8, 2022

Uh oh!

yihua Jan 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yihua Jan 9, 2022

Choose a reason for hiding this comment

Uh oh!

cdmikechen commented Jan 9, 2022

Uh oh!

hudi-bot commented Jan 9, 2022

CI report:

Uh oh!

yihua left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

cdmikechen commented Dec 28, 2021 •

edited

Loading

yihua Jan 9, 2022 •

edited

Loading

yihua Jan 9, 2022 •

edited

Loading