[HUDI-251] JDBC incremental load to HUDI DeltaStreamer #962

taherk77 · 2019-10-18T10:32:47Z

No description provided.

…correct bin packing

…he GenericRecord Removing Converter hierarchy as we now depend purely on JavaSerialization and require the payload to be java serializable

…ory as partition paths

…itions

(1) Apply transformation when using delta-streamer to ingest data. (2) Add Hudi Incremental Source for Delta Streamer (3) Allow delta-streamer config-property to be passed as command-line (4) Add Hive Integration to Delta-Streamer and address Review comments (5) Ensure MultiPartKeysValueExtractor handle hive style partition description (6) Reuse same spark session on both source and transformer (7) Support extracting partition fields from _hoodie_partition_path for HoodieIncrSource (8) Reuse Binary Avro coders (9) Add push down filter for Incremental source (10) Add Hoodie DeltaStreamer metrics to track total time taken

… at any time to avoid race conditions

typo: bundle jar with unrecognized variables

Exclude common dependencies that are available in Presto

… during on-the-fly merge of Real Time tables (#956) * Fix issue with incorrect column mapping casusing bad data, during on-the-fly merge of Real Time tables

…ubator group

taherk77 · 2019-10-18T10:35:05Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JDBCSource.java

Will reimplement this section with the last comment, we should do hoodie.datasource.jdbc.extra.options.fetchsize, hoodie.datasource.jdbc.extra.options.timestampFormat ..etc

taherk77 · 2019-10-18T10:44:20Z

hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestJdbcSource.java

The way we do jdbc incremental pull is that we will run use the checkpoint value and prepare a ppd query, the query we prepare inside the JDBCSource with the incremental value is "(select * from table where incremental_column > '2019-09-09 15:45:01.53' ) rdbms_table" and we except that the incremental column is either a timestamp or an int/long val.

This worked well when we tried it with MYSQL and mysql was ok converting the string column into int or long or timestamp based on whichever column we compared it with. However, derby is not doing that and is throwing exceptions. if the ppd query is changed to "(select * from table where incremental_column >Timestamp( '2019-09-09 15:45:01.53') ) rdbms_table" then it works as the incremental column value has been casted/converted to timestamp, but normal strings do not work here.

Any help on this will be appreciated!

Is it possible to get the type of columns? If it is a timestamp, just convert or cast to Sting?

leesf · 2019-10-18T11:32:21Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JDBCSource.java

To keep with the current log framework, use LogManager.getLogger(JDBCSource.class) instead?

I found that that logger doesn't allow me to do {}

Yes, please use + or StringFormat yet.

leesf · 2019-10-18T15:29:56Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JDBCSource.java

System.out.println => LOG.info?

That was for test purposes. The PR is not yet fully done. Work is still in progress

leesf · 2019-10-18T15:31:24Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JDBCSource.java

remove this line ?

That too is there just for test purposes as I was facing issues with derby which u have highlighted above

leesf · 2019-10-18T15:32:08Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JDBCSource.java

need indentation

I think the indentation is perfect as it is ide indented and follows checkstyle

leesf · 2019-10-18T15:44:52Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JDBCSource.java

if (lastCheckpoint.isPresent()) { if (StringUtils.isNullOrEmpty(lastCheckpoint.get())) { LOG.warn("Previous checkpoint entry was null or empty. Falling back to full jdbc pull."); return full(isIncremental); } else { return incremental(ppdQuery, lastCheckpoint, isIncremental); } } else { LOG.info("No previous checkpoints found.. "); return full(isIncremental); }

maybe would be changed to the code snippet since it is more clear and readable.

leesf · 2019-10-18T15:52:12Z

hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JDBCSource.java

typo internall

leesf · 2019-10-18T15:53:06Z

hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestJdbcSource.java

leesf · 2019-10-18T15:55:23Z

hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestJdbcSource.java

could remove when use Assert.assertEquals

leesf · 2019-10-18T15:56:17Z

hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestJdbcSource.java

any Asserts?

PR is not final yet. WIP

leesf · 2019-10-18T15:57:06Z

hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestJdbcSource.java

can be removed too

leesf · 2019-10-18T15:58:36Z

hudi-utilities/src/test/resources/log4j-surefire.properties

Why change this?

leesf · 2019-10-18T16:01:51Z

Thanks for opening the PR @taherk77 . I have left some comments.
PS: Please also fix the travis errors. maybe you could run mvn clean install locally to avoid some checkstyle errors.

vinothchandar · 2019-10-23T15:10:49Z

@taherk77 why closing this?

taherk77 · 2019-10-23T15:19:58Z

@taherk77 why closing this?

While pushing some new code i messed my git repo, so closed this one. Will reopen again

Vinoth Chandar and others added 30 commits November 30, 2018 16:23

Update committership for balaji

3dfeb04

Returning empty Statues for an empty spark partition caused due to in…

f42aa0b

…correct bin packing

Serializing the complete payload object instead of serializing just t…

78926ca

…he GenericRecord Removing Converter hierarchy as we now depend purely on JavaSerialization and require the payload to be java serializable

feat(SparkDataSource): add structured streaming

86207e6

fix some spell errorin Hudi

27789b9

Ensure Hoodie works for non-partitioned Hive table

cf2269b

General enhancements

e13dd8c

Add m2 directory to Travis cache

55b73be

Enabling hard deletes for MergeOnRead table type

cbe5893

Passing a path filter to avoid including folders under .hoodie direct…

8eba0e5

…ory as partition paths

Filtering partition paths before performing a list status on all part…

c2f85a5

…itions

Adding compaction to HoodieClient example

46b6c81

cleaner should now use commit timeline and not include deltacomits

b9401b4

Fixing behavior of Merge/CreateHandle for invalid/wrong schema records

e389078

Using immutable map instead of mutables to generate parameters

4ed5331

Updating new slack signup link

eceddad

Fix Hoodie Record Reader to work with non-partitioned dataset

3e18769

Table rollback for inflight compactions MUST not delete instant files…

b8887b4

… at any time to avoid race conditions

typo: bundle jar with unrecongnized variables

1263bb4

Update site url in README

720e0d3

Create CNAME

6891e05

handle no such element exception in HoodieSparkSqlWriter

713fb47

Close FSDataInputStream for meta file open in HoodiePartitionMetadata

0ff5970

Merge pull request #570 from yaooqinn/hiveJarSuffix

ed6a434

typo: bundle jar with unrecognized variables

Create hoodie-presto bundle jar

374c287

Exclude common dependencies that are available in Presto

Update RELEASE_NOTES for 0.4.5

91d3976

[maven-release-plugin] prepare release hoodie-0.4.5

a117a33

[maven-release-plugin] prepare for next development iteration

8ba34aa

Upgrade various jar, gem versions for maintenance

bddc5b1

umehrot2 and others added 8 commits October 16, 2019 02:05

[HUDI-298] Fix issue with incorrect column mapping casusing bad data,…

3e0050c

… during on-the-fly merge of Real Time tables (#956) * Fix issue with incorrect column mapping casusing bad data, during on-the-fly merge of Real Time tables

[HUDI-301] fix path error when update a non-partition MOR table

6ce8ab1

Shade and relocate Avro dependency in hadoop-mr-bundle

bf02a6d

[HUDI-121] Fix licensing issues found during RC voting by general inc…

da41319

…ubator group

Update RELEASE Notes in master

845d507

[HUDI-121] Fix issues in release scripts

4782edb

[HUDI-40] Add parquet support for the Delta Streamer (#949)

f872cb5

[HUDI-251] JDBC incremental load to HUDI DeltaStreamer

302df97

taherk77 commented Oct 18, 2019

View reviewed changes

leesf reviewed Oct 18, 2019

View reviewed changes

hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JDBCSource.java Outdated

Copy link

Contributor

leesf Oct 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo internall

leesf reviewed Oct 18, 2019

View reviewed changes

hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestJdbcSource.java Outdated

Copy link

Contributor

leesf Oct 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

leesf reviewed Oct 18, 2019

View reviewed changes

hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestJdbcSource.java Outdated

Copy link

Contributor

leesf Oct 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can be removed too

leesf reviewed Oct 18, 2019

View reviewed changes

hudi-utilities/src/test/resources/log4j-surefire.properties Outdated

Copy link

Contributor

leesf Oct 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why change this?

taher-koitawala added 4 commits October 23, 2019 20:09

[HUDI-251] JDBC incremental load to HUDI DeltaStreamer

f9ec472

Merge branch 'master' of https://github.com/taherk77/incubator-hudi

343727a

Merge branch 'master' of https://github.com/taherk77/incubator-hudi

8124208

Merge branch 'master' of https://github.com/taherk77/incubator-hudi

792ad80

taherk77 closed this Oct 23, 2019

hudi-bot mentioned this pull request Nov 30, 2025

Fix Azure CI timeout for UT FT other modules #16221

Closed

[HUDI-251] JDBC incremental load to HUDI DeltaStreamer #962

[HUDI-251] JDBC incremental load to HUDI DeltaStreamer #962

Uh oh!

Conversation

taherk77 commented Oct 18, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leesf commented Oct 18, 2019

Uh oh!

vinothchandar commented Oct 23, 2019

Uh oh!

taherk77 commented Oct 23, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants