[HUDI-8235] Adding support for EPOCHMICROSECONDS in TimestampBasedAvroKeyGenerator

[sydneybeal]

Change Logs

Adding support for EPOCHMICROSECONDS in TimestampBasedAvroKeyGenerator

Impact

Users can pass value EPOCHMICROSECONDS to config hoodie.deltastreamer.keygen.timebased.timestamp.type, key gen field is parsed into the correct partitions i.e. yyyy/mm/dd

Risk level (write none, low medium or high below)

Low

For testing I created a jar locally with hudi-utilities-bundle_2.12-0.12.2 with my code change and substituted it in my Spark job - it worked as planned, microsecond timestamps are being partitioned as expected now. Running Google Dataproc Hudi Deltastreamer job, reading from Confluent Cloud Debezium Avro Kafka topic, writing Hudi outputs to S3.

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

This would require an update to the keygen timestamp.type in the documentation to include EPOCHMICROSECONDS in this section of the documentation

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• [] Adequate tests were added if applicable
• [] CI passed

[bvaradar]

@sydneybeal : Thanks a lot for contributing to Hudi. It looks like the PR validation is failing. Can you take a look at this ? Also, Can you add a unit test for this change.

[bvaradar]

@sydneybeal : Pinging to see if you can address the comments ?

[sydneyhoran]

@bvaradar Hi there I just realized I made this PR using my old github account so I've missed these notifications. I will look at addressing this test case as soon as I have a chance.

[yihua]

@sydneybeal Thanks for your contribution. Since it's been a while, I've rebased the PR and added a couple of tests, along with docs update.

[balaji-varadarajan-ai]

@yihua LGTM. Does the existing behavior treat these types as strings ? In that case, we will have change in behavior with this patch (need release description about this).

[yihua]

@yihua LGTM. Does the existing behavior treat these types as strings ? In that case, we will have change in behavior with this patch (need release description about this).

Thanks. There is no behavior change to existing types; EPOCHMICROSECONDS is a new supported timestamp type like EPOCHMILLISECONDS in the Timestamp Key Generator. Config docs is updated so when we cut the release docs the configuration page will show this newly supported type.

Approved by Balaji above through a different account

[hudi-bot]

CI report:

• 600d589 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build