Flink: Add the iceberg files committer to collect data files and commit to iceberg table.#1185

Merged

rdblue merged 30 commits intoapache:masterfrom

openinx:flink-committer

Aug 28, 2020

Commits on Aug 27, 2020

Flink: Add the iceberg files committer to collect data files and commit to iceberg table.
openinx
committed
Add comments for the critical members
openinx
committed
Read the max-committed-checkpoint-id only when restoring the job.
openinx
committed
Fix the broken unit tests.
openinx
committed
Refactor to use the SortedMapTypeInfo
openinx
committed
Handle the cases that two different jobs write the the same table.
openinx
committed
Minor fix
openinx
committed
Make IcebergFilesCommitter extends from AbstractStreamOperator for committing txn for bounded stream
openinx
committed
Add the unit tests for ordered and disordered events between two continues checkpoints
openinx
committed
Fix the checkstyle
openinx
committed
Add a check to validate the parsed maxCommittedCheckpointId.
openinx
committed
Improve the flink data stream unit tests.
openinx
committed
Addressing comments.
openinx
committed
Minor fixes
openinx
committed
Introduce orc to the filesCommitter unit tests.
openinx
committed
Use the TableLoader to load table lazily inside icebergFilesCommitter
openinx
committed
Advancing the max-committed-checkpoint-id when there's no data files to commit.
openinx
committed
Fix the broken unit tests after ORC was introduced
openinx
committed
Keep the increased max-committed-checkpoint-id
openinx
committed
Refactor the FlinkSink builder to support both DataStream<Row> dan DataStream<RowData>
openinx
committed
Close the table loader once we disposed in iceberg files committer
openinx
committed
Addressing the comments
openinx
committed
Add parquet to the unit tests.
openinx
committed
Add unit tests for bounded stream
openinx
committed
Rebase to master and fix the complie error
openinx
committed
Move the writer classes to sink package
openinx
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Flink: Add the iceberg files committer to collect data files and commit to iceberg table.#1185

Flink: Add the iceberg files committer to collect data files and commit to iceberg table.#1185
rdblue merged 30 commits intoapache:masterfrom
openinx:flink-committer

Commits on Aug 27, 2020

Flink: Add the iceberg files committer to collect data files and commit to iceberg table.

Add comments for the critical members

Read the max-committed-checkpoint-id only when restoring the job.

Fix the broken unit tests.

Refactor to use the SortedMapTypeInfo

Handle the cases that two different jobs write the the same table.

Minor fix

Make IcebergFilesCommitter extends from AbstractStreamOperator for committing txn for bounded stream

Add the unit tests for ordered and disordered events between two continues checkpoints

Fix the checkstyle

Add a check to validate the parsed maxCommittedCheckpointId.

Improve the flink data stream unit tests.

Addressing comments.

Minor fixes

Introduce orc to the filesCommitter unit tests.

Use the TableLoader to load table lazily inside icebergFilesCommitter

Advancing the max-committed-checkpoint-id when there's no data files to commit.

Fix the broken unit tests after ORC was introduced

Keep the increased max-committed-checkpoint-id

Refactor the FlinkSink builder to support both DataStream<Row> dan DataStream<RowData>

Close the table loader once we disposed in iceberg files committer

Addressing the comments

Add parquet to the unit tests.

Add unit tests for bounded stream

Rebase to master and fix the complie error

Move the writer classes to sink package

Commits on Aug 28, 2020

Addressing comments from Ryan

Minor changes: removing stale comments etc.

Refactor the FlinkSink API and provide detailed javadoc.

Fix the broken unit tests