Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
enhance documentation #61
enhance documentation #61
Changes from all commits
b98f42c
2dac340
File filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
have you had a chance to test it? i think schema change events are not processed and should not cause any error. schema change events are saved to file if its configured with
debezium.source.database.history
anddebezium.source.database.history.file.filename
.I believe we could remove this section:
Schema change events can make the connector throw error. To workaround this, turn off schema change event in source setting.
and link this page to give more details about current schema change behavior
Schema Change Behaviour : https://github.com/memiiso/debezium-server-iceberg/blob/master/docs/DOCS.md#schema-change-behaviour
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is what I found when using this with SQL Server, you can check the following log
This is the config when I didn't turn off schema change capture for SQL Server
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it seems like consumer is trying to create table
debeziumevents.debeziumcdc_tutorial
to store schema changes. but its failing because field 'tableChanges' is in Array type. Currently iceberg consumer is not supporting Array data type that's why exception thrown.documentation
not sure what is the best way to explain it. maybe something like
Schema change topic
has unsupported data type Array, its recommended to disable itThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
seems like even with array data type support its not possible to save
Schema change topic
. seems liketableChanges
has kind of special type. it makes sense to recommend disable it for all connectors.its failing with
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, I also checked iceberg document when I found that error and decided to avoid that by turn off schema change event
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could we highlight that is possible to pass any iceberg configuration using
debezium.sink.iceberg.<my.iceberg.config>=xyz
format? .it could be useful to know there are many iceberg configs and its possible to set them withapplication.properties
.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe the detailed docs mentioned it already. You can check it here https://github.com/chulucninh09/debezium-server-iceberg/blob/document-enhancement/docs/DOCS.md#configuring-iceberg