From 0bc8358543076ed1b74eed65a83785e37591656d Mon Sep 17 00:00:00 2001 From: Ismail Simsek <6005685+ismailsimsek@users.noreply.github.com> Date: Sun, 31 Dec 2023 22:59:19 +0100 Subject: [PATCH] Improve documentation --- docs/DOCS.md | 37 ++++++++++++++++++++----------------- 1 file changed, 20 insertions(+), 17 deletions(-) diff --git a/docs/DOCS.md b/docs/DOCS.md index 0fcc75ad..f100540a 100644 --- a/docs/DOCS.md +++ b/docs/DOCS.md @@ -15,23 +15,26 @@ tables created automatically with the first start. #### Configuration properties -| Config | Default | Description | -|----------------------------------------------------|-------------------|------------------------------------------------------------------------------------------------------------------| -| `debezium.sink.iceberg.warehouse` | | Root path of the Iceberg data warehouse | -| `debezium.sink.iceberg.catalog-name` | `default` | User-specified Iceberg catalog name. | -| `debezium.sink.iceberg.table-namespace` | `default` | A namespace in the catalog. ex: `SELECT * FROM prod.db.table -- catalog: prod, namespace: db, table: table` | -| `debezium.sink.iceberg.table-prefix` | `` | Iceberg table name prefix, Added to destination iceberg table names. | -| `debezium.sink.iceberg.write.format.default` | `parquet` | Default file format for the table; `parquet`, `avro`, or `orc` | -| `debezium.sink.iceberg.allow-field-addition` | `true` | Allow field addition to target tables | -| `debezium.sink.iceberg.upsert` | `true` | Running upsert mode overwriting updated rows. explained below. | -| `debezium.sink.iceberg.upsert-keep-deletes` | `true` | When running with upsert mode, keeps deleted rows in target table. | -| `debezium.sink.iceberg.upsert-dedup-column` | `__source_ts_ms` | With upsert mode used to deduplicate data. row with highest `__source_ts_ms` kept. _ - dont change!_ | -| `debezium.sink.iceberg.upsert-op-column` | `__op` | Used with upsert mode. _dont change!_ | -| `debezium.sink.iceberg.destination-regexp` | `` | Regexp to modify destination table. With this its possible to map `table_ptt1`,`table_ptt2` to `table_combined`. | -| `debezium.sink.iceberg.destination-regexp-replace` | `` | Regexp Replace part to modify destination table | -| `debezium.sink.batch.batch-size-wait` | `NoBatchSizeWait` | Batch size wait strategy to optimize data files and upload interval. explained below. | -| `debezium.sink.iceberg.{iceberg.prop.name}` | | [Iceberg config](https://iceberg.apache.org/docs/latest/configuration/) passed to Iceberg | +| Config | Default | Description | +|------------------------------------------------------------------------------------|---------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `debezium.sink.iceberg.warehouse` | | Root path of the Iceberg data warehouse | +| `debezium.sink.iceberg.catalog-name` | `default` | User-specified Iceberg catalog name. | +| `debezium.sink.iceberg.table-namespace` | `default` | A namespace in the catalog. ex: `SELECT * FROM prod.db.table -- catalog: prod, namespace: db, table: table` | +| `debezium.sink.iceberg.table-prefix` | `` | Iceberg table name prefix, Added to destination iceberg table names. | +| `debezium.sink.iceberg.write.format.default` | `parquet` | Default file format for the table; `parquet`, `avro`, or `orc` | +| `debezium.sink.iceberg.allow-field-addition` | `true` | Allow field addition to target tables | +| `debezium.sink.iceberg.upsert` | `true` | Running upsert mode overwriting updated rows. explained below. | +| `debezium.sink.iceberg.upsert-keep-deletes` | `true` | When running with upsert mode, keeps deleted rows in target table. | +| `debezium.sink.iceberg.upsert-dedup-column` | `__source_ts_ms` | With upsert mode used to deduplicate data. row with highest `__source_ts_ms` kept. _dont change!_ | +| `debezium.sink.iceberg.upsert-op-column` | `__op` | Used with upsert mode. _dont change!_ | +| `debezium.sink.iceberg.destination-regexp` | `` | Regexp to modify destination table. With this its possible to map `table_ptt1`,`table_ptt2` to `table_combined`. | +| `debezium.sink.iceberg.destination-regexp-replace` | `` | Regexp Replace part to modify destination table | +| `debezium.sink.batch.batch-size-wait` | `NoBatchSizeWait` | Batch size wait strategy to optimize data files and upload interval. explained below. | +| `debezium.sink.iceberg.{iceberg.prop.name}` | | [Iceberg config](https://iceberg.apache.org/docs/latest/configuration/) passed to Iceberg | +| `debezium.source.offset.storage` | `io.debezium.server.iceberg.offset.IcebergOffsetBackingStore` | The name of the Java class that is responsible for persistence of connector offsets. see [debezium doc](https://debezium.io/documentation/reference/stable/development/engine.html#advanced-consuming) | +| `debezium.source.offset.storage.iceberg.table-name` | `debezium_offset_storage` | Destination table name to store connector offsets. | +| `debezium.source.schema.history.internal` | `io.debezium.server.iceberg.history.IcebergSchemaHistory` | The name of the Java class that is responsible for persistence of the database schema history. see [debezium doc](https://debezium.io/documentation/reference/stable/development/engine.html#database-history-properties) | +| `debezium.source.schema.history.internal.iceberg.table-name` | `debezium_schema_history_storage` | Destination table name to store database schema history. | ### Upsert