Fails to start the server

[jaehyeon-kim]

Hi

I just made a quick trial of running the server with the following configuration.

debezium.sink.type=iceberg
debezium.sink.iceberg.table-prefix=debeziumcdc_
debezium.sink.iceberg.catalog-name=mycatalog
debezium.sink.iceberg.table-namespace=debeziumevents
debezium.sink.iceberg.fs.defaultFS=s3a://<my-bucket>
debezium.sink.iceberg.warehouse=s3a://<my-bucket>/iceberg_warehouse
debezium.sink.iceberg.type=hadoop
debezium.sink.iceberg.catalog-name=mycatalog
debezium.sink.iceberg.com.amazonaws.services.s3.enableV4=true
debezium.sink.iceberg.com.amazonaws.services.s3a.enableV4=true
debezium.sink.iceberg.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain
debezium.sink.iceberg.fs.s3a.access.key=<my-aws-access-key>
debezium.sink.iceberg.fs.s3a.secret.key=<my-secret-access-key>
debezium.sink.iceberg.fs.s3a.path.style.access=true
debezium.sink.iceberg.fs.s3a.endpoint=http://localhost:9000 # minio specific setting
debezium.sink.iceberg.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
debezium.sink.iceberg.upsert=true
debezium.sink.iceberg.upsert-keep-deletes=true
debezium.format.value.schemas.enable=true
debezium.format.key.schemas.enable=true

At first I encountered the following error.

Then I added hadoop-aws-2.7.0.jar to the lib folder. This time I saw another error as shown below.

I may try to add JAR files manually but concern if I made a mistake.

Cheers,
Jaehyeon

[ismailsimsek]

it seems like it was missing few aws dependency
could you please use the latest release and try again?

[jaehyeon-kim]

Hi

It works fine when I checked with debezium/example-postgres:1.7.0.Final as the data source.

Since it fails to create tables because of nested data as shown below, I added extra config, which is indicated by your upcoming blog post.

# enable event schemas
debezium.format.value.schemas.enable=true
debezium.format.value=json
# complex nested data types are not supported, do event flattening. unwrap message!
debezium.transforms=unwrap
debezium.transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
debezium.transforms.unwrap.add.fields=op,table,source.ts_ms,db
debezium.transforms.unwrap.delete.handling.mode=rewrite
debezium.transforms.unwrap.drop.tombstones=true

Also I disabled the minio specific setting.

# debezium.sink.iceberg.fs.s3a.endpoint=http://localhost:9000 # minio specific setting

At least I can get started. Thank you so much!

Cheers,
Jaehyeon