-
Notifications
You must be signed in to change notification settings - Fork 35
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sql server config & minio with inmemory catalog nog working #236
Comments
this config must be set as mandatory, for some reason. not sure it has any effect. with above settings, schema history should be stored in destination iceberg table
also its set to false |
I don't get it. The setting is mandatory, but we don't have any kafka servers in this setup, right? The log file if I include
|
@nelo-0 could you try with following config
config name is changed in recent releases |
@ismailsimsek, sorry for the late reply but I was out of the office for a short holiday. Thanks for the fix, the sql server config seems to work. But I think the config with the inmemory catalog does not work. The connector log is not showing any errors at the initial startup. However, I can't see any new files in my S3 bucket... Really strange because the logs are saying new tables are written. After inserting a new row in the sql server table the connector crashes with:
Config file:
Full log file:
|
@nelo-0 could you try it with |
@ismailsimsek That was a typo indeed. Unfortunately, after fixing the typo the error was still there. Finally got it working with spark-iceberg en the icerberg rest api catalog. Thanks to https://tabular.io/blog/docker-spark-and-iceberg/ Final setup: sqlserver = docker-compose.yml
main-setup = docker-compose.yml
application.properties
This is really promising. Thanks for all the help and hard work so far. I will continue my journey to try setting this up with the Azure Databricks hive metastore en writing the iceberg files to ADLS. |
Fantastic, please feel free to share your experience, A documentation page to explain your setup on azure definitely will be helpful for other users. recently some users were asking about azure storage setup and hive meta store catalog setup |
We are testing a setup in docker with sqlserver and minio:
docker-compose.yml
application.properties:
De container will crash if we leave out
debezium.source.schema.history.internal.kafka.topic=schemahistory
and
debezium.source.schema.history.internal.kafka.bootstrap.servers=empty
but that are clearly kafka parameters.The text was updated successfully, but these errors were encountered: