Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
enhance documentation #61
enhance documentation #61
Changes from 1 commit
b98f42c
2dac340
File filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
have you had a chance to test it? i think schema change events are not processed and should not cause any error. schema change events are saved to file if its configured with
debezium.source.database.history
anddebezium.source.database.history.file.filename
.I believe we could remove this section:
Schema change events can make the connector throw error. To workaround this, turn off schema change event in source setting.
and link this page to give more details about current schema change behavior
Schema Change Behaviour : https://github.com/memiiso/debezium-server-iceberg/blob/master/docs/DOCS.md#schema-change-behaviour
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is what I found when using this with SQL Server, you can check the following log
This is the config when I didn't turn off schema change capture for SQL Server
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it seems like consumer is trying to create table
debeziumevents.debeziumcdc_tutorial
to store schema changes. but its failing because field 'tableChanges' is in Array type. Currently iceberg consumer is not supporting Array data type that's why exception thrown.documentation
not sure what is the best way to explain it. maybe something like
Schema change topic
has unsupported data type Array, its recommended to disable itThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
seems like even with array data type support its not possible to save
Schema change topic
. seems liketableChanges
has kind of special type. it makes sense to recommend disable it for all connectors.its failing with
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, I also checked iceberg document when I found that error and decided to avoid that by turn off schema change event
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i believe its is possible to use
application.properties
to configure AWS S3all the settings starting with
debezium.sink.iceberg.<my.config>
are passed to iceberg and from there used by icebergexample from unit test
https://github.com/memiiso/debezium-server-iceberg/blob/master/debezium-server-iceberg-sink/src/test/java/io/debezium/server/iceberg/testresources/S3Minio.java#L131
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yup, saw that too, by default, I suggest turn off
debezium.sink.iceberg.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain
to provide easier way to inject aws credential