Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improvement] Use Append Iceberg Commit for Append Mode #187

Closed
wobu opened this issue Apr 19, 2023 · 2 comments · Fixed by #188
Closed

[Improvement] Use Append Iceberg Commit for Append Mode #187

wobu opened this issue Apr 19, 2023 · 2 comments · Fixed by #188
Labels
enhancement New feature or request

Comments

@wobu
Copy link

wobu commented Apr 19, 2023

When using the append mode in the debezium server iceberg project still Overwrite Commits are produced in the Iceberg table even if Append Commits would also technical work.

Append Commits / Snapshots in the Iceberg Table are have some benefits:

This should be easy fixable:

Enhance the code at https://github.com/memiiso/debezium-server-iceberg/blob/master/debezium-server-iceberg-sink/src/main/java/io/debezium/server/iceberg/tableoperator/IcebergTableOperator.java#L182
with this check https://github.com/apache/iceberg/blob/master/flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java#L349

@ismailsimsek ismailsimsek added the enhancement New feature or request label Apr 19, 2023
@ismailsimsek
Copy link
Member

@wobu good point. adding if else with files.deleteFiles().length > 0 should be enough right?

do you want to contribute it?

@wobu
Copy link
Author

wobu commented Apr 20, 2023

@ismailsimsek shoud be sufficient with my understanding

we will contribute, if we decide to focus on using this project. Currently the decision hasn't been made. Until then anyone is allowed to contribute :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants