[HUDI-2909] Handle logical type in TimestampBasedKeyGenerator #4203

codope · 2021-12-03T07:32:44Z

What is the purpose of the pull request

#3944 handled logical timestamp type for complex keygen. This PR fixes HUDI-2909 by handing it for timestamp based keygen.

When the new config hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled is set to true, consistent value will be generated for a logical timestamp type column like timestamp-millis and timestamp-micros, irrespective of whether row-writer is enabled. Disabled by default so as not to break the pipeline that deploy either fully row-writer path or non row-writer path. For example, if it is kept disabled then record key of timestamp type with value 2016-12-29 09:54:00 will be written as timestamp 2016-12-29 09:54:00.0 in row-writer path, while it will be written as long value 1483023240000000 in non row-writer path. If enabled, then the timestamp value will be written in both the cases.

Brief change log

(for example:)

Modify AnnotationLocation checkstyle rule in checkstyle.xml

Verify this pull request

Added UT and manually verified with spark datasource.

Committer checklist

Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

nsivabalan · 2021-12-15T20:01:16Z

@codope : is this good to review. If there are any pending work, I can take it up as I am focusing on all issues and jiras. let me know.

codope · 2021-12-31T15:52:45Z

@codope : is this good to review. If there are any pending work, I can take it up as I am focusing on all issues and jiras. let me know.

@nsivabalan It is now ready to review. Both row-writer and non-row writer path will emit same value for logical timestamp type column if the config is enabled.

nsivabalan

Let's see if we can avoid changing public apis in KeyGenerator.

nsivabalan · 2022-01-04T21:45:29Z

hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java

  }

+  public boolean isConsistentLogicalTimestampEnabled() {
+    if (getBoolean(KeyGeneratorOptions.KEYGENERATOR_CONSISTENT_LOGICAL_TIMESTAMP_ENABLED) == null) {


wouldn't getBoolean return the default if not found?

.../hudi-client-common/src/main/java/org/apache/hudi/keygen/TimestampBasedAvroKeyGenerator.java

...t/hudi-spark-client/src/test/java/org/apache/hudi/keygen/TestTimestampBasedKeyGenerator.java

...datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/SqlKeyGenerator.scala

YannByron · 2022-01-05T02:58:23Z

@nsivabalan @codope
I have a discussion related to this implement. In this pr, most of work is just to pass isConsistentLogicalTimestampEnabled to the method HoodieAvroUtils.convertValueForAvroLogicalTypes. What if we have another config need to do this in the future?

codope · 2022-01-06T12:30:27Z

@nsivabalan @codope I have a discussion related to this implement. In this pr, most of work is just to pass isConsistentLogicalTimestampEnabled to the method HoodieAvroUtils.convertValueForAvroLogicalTypes. What if we have another config need to do this in the future?

@YannByron You bring up a good point. Adding another config in future is tedious. However, the intention behind adding a new config was to avoid discrepancy in existing pipelines. @nsivabalan has explained this in more detail on the jira HUDI-2909. I do not expect such changes to be frequent. Nevertheless, i'll try to avoid making incompatible changes to public APIs.

hudi-common/src/main/java/org/apache/hudi/keygen/constant/KeyGeneratorOptions.java

nsivabalan · 2022-01-06T21:39:36Z

@codope : TestGenericRecordAndRowConsistency test failed in CI. Can you please check.

codope · 2022-01-07T08:16:08Z

@hudi-bot run azure

Add config to enable consistent logical timestamp values Address review comments Add an example in the config doc Cleanup spark context in test

hudi-bot · 2022-01-08T13:37:10Z

CI report:

684c7df Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

nsivabalan

LGTM.

* [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator Timestampbased key generator was returning diff values for row writer and non row writer path. this patch fixes it and is guarded by a config flag (`hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled`)

alexeykudinkin · 2022-01-12T01:18:15Z

hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java

   */
-  public static String getNestedFieldValAsString(GenericRecord record, String fieldName, boolean returnNullIfNotFound) {
-    Object obj = getNestedFieldVal(record, fieldName, returnNullIfNotFound);
+  public static String getNestedFieldValAsString(GenericRecord record, String fieldName, boolean returnNullIfNotFound, boolean consistentLogicalTimestampEnabled) {


@codope @nsivabalan folks, i don't believe this is the right fix to address the problem: bubbling down this config value we're now exposing every user of this API to be aware of it, which, very likely, will have very little to do with it.

…#4203) * [HUDI-2909] Handle logical type in TimestampBasedKeyGenerator Timestampbased key generator was returning diff values for row writer and non row writer path. this patch fixes it and is guarded by a config flag (`hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled`)

codope added the priority:blocker Production down; release blocker label Dec 3, 2021

codope mentioned this pull request Dec 3, 2021

Revert "[HUDI-2495] Resolve inconsistent key generation for timestamp… #4201

Merged

5 tasks

codope removed the priority:blocker Production down; release blocker label Dec 4, 2021

nsivabalan self-assigned this Dec 15, 2021

nsivabalan added the priority:critical Production degraded; pipelines stalled label Dec 15, 2021

codope force-pushed the hudi-2909-timestamp-keygen branch from e12df0f to 5b68cad Compare December 31, 2021 15:50

nsivabalan requested changes Jan 4, 2022

View reviewed changes

codope force-pushed the hudi-2909-timestamp-keygen branch from 5b68cad to 2c0565c Compare January 6, 2022 15:57

nsivabalan reviewed Jan 6, 2022

View reviewed changes

hudi-common/src/main/java/org/apache/hudi/keygen/constant/KeyGeneratorOptions.java Show resolved Hide resolved

codope force-pushed the hudi-2909-timestamp-keygen branch 2 times, most recently from fde98bb to cb2d16c Compare January 7, 2022 07:19

[HUDI-2909] Handle logical type in TimestampBasedKeyGenerator

9c22d6c

Add config to enable consistent logical timestamp values Address review comments Add an example in the config doc Cleanup spark context in test

codope force-pushed the hudi-2909-timestamp-keygen branch from cb2d16c to 9c22d6c Compare January 7, 2022 08:27

codope added 4 commits January 7, 2022 16:43

Close park context

62636d2

Disable TestHoodieSparkSqlWriter#testThrowExceptionInvalidSerializer

41e949d

Close spark context in TestHoodieSparkSqlWriter

fc2dfc7

Add a test for SqlKeygenerator

684c7df

nsivabalan approved these changes Jan 8, 2022

View reviewed changes

nsivabalan merged commit 8275499 into apache:master Jan 8, 2022

nsivabalan mentioned this pull request Jan 10, 2022

[BUG] Data corrupted in the timestamp field to 1970-01-01 19:45:30.000 after subsequent upsert run #4552

Closed

alexeykudinkin reviewed Jan 12, 2022

View reviewed changes

vinishjail97 mentioned this pull request Jan 24, 2022

FixIgnoreKey nsivabalan/hudi#11

Closed

5 tasks

[HUDI-2909] Handle logical type in TimestampBasedKeyGenerator #4203

[HUDI-2909] Handle logical type in TimestampBasedKeyGenerator #4203

Uh oh!

Conversation

codope commented Dec 3, 2021 • edited by nsivabalan Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the pull request

Brief change log

Verify this pull request

Committer checklist

Uh oh!

nsivabalan commented Dec 15, 2021

Uh oh!

codope commented Dec 31, 2021

Uh oh!

nsivabalan left a comment

Choose a reason for hiding this comment

Uh oh!

nsivabalan Jan 4, 2022

Choose a reason for hiding this comment

Uh oh!

YannByron Jan 5, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

YannByron commented Jan 5, 2022

Uh oh!

codope commented Jan 6, 2022

Uh oh!

Uh oh!

nsivabalan commented Jan 6, 2022

Uh oh!

codope commented Jan 7, 2022

Uh oh!

hudi-bot commented Jan 8, 2022

CI report:

Uh oh!

nsivabalan left a comment

Choose a reason for hiding this comment

Uh oh!

alexeykudinkin Jan 12, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codope commented Dec 3, 2021 •

edited by nsivabalan

Loading