Skip to content

Commit

Permalink
Improve documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
@ismailsimsek
ismailsimsek committed Dec 31, 2023
1 parent 32f5caf commit 6c74d9b
Show file tree
Hide file tree
Showing 2 changed files with 23 additions and 18 deletions.
2 changes: 2 additions & 0 deletions debezium-server-iceberg-dist/src/main/resources/distro/conf/application.properties.example
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,10 @@ debezium.format.value=json
debezium.format.key=json

# saving debezium state data to destination, iceberg tables
# see https://debezium.io/documentation/reference/stable/development/engine.html#advanced-consuming
debezium.source.offset.storage=io.debezium.server.iceberg.offset.IcebergOffsetBackingStore
debezium.source.offset.storage.iceberg.table-name=debezium_offset_storage_custom_table
# see https://debezium.io/documentation/reference/stable/development/engine.html#database-history-properties
debezium.source.schema.history.internal=io.debezium.server.iceberg.history.IcebergSchemaHistory
debezium.source.schema.history.internal.iceberg.table-name=debezium_database_history_storage_test

Expand Down
39 changes: 21 additions & 18 deletions docs/DOCS.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,23 +15,26 @@ tables created automatically with the first start.

#### Configuration properties

| Config | Default | Description |
|----------------------------------------------------|-------------------|------------------------------------------------------------------------------------------------------------------|
| `debezium.sink.iceberg.warehouse` | | Root path of the Iceberg data warehouse |
| `debezium.sink.iceberg.catalog-name` | `default` | User-specified Iceberg catalog name. |
| `debezium.sink.iceberg.table-namespace` | `default` | A namespace in the catalog. ex: `SELECT * FROM prod.db.table -- catalog: prod, namespace: db, table: table` |
| `debezium.sink.iceberg.table-prefix` | `` | Iceberg table name prefix, Added to destination iceberg table names. |
| `debezium.sink.iceberg.write.format.default` | `parquet` | Default file format for the table; `parquet`, `avro`, or `orc` |
| `debezium.sink.iceberg.allow-field-addition` | `true` | Allow field addition to target tables |
| `debezium.sink.iceberg.upsert` | `true` | Running upsert mode overwriting updated rows. explained below. |
| `debezium.sink.iceberg.upsert-keep-deletes` | `true` | When running with upsert mode, keeps deleted rows in target table. |
| `debezium.sink.iceberg.upsert-dedup-column` | `__source_ts_ms` | With upsert mode used to deduplicate data. row with highest `__source_ts_ms` kept. _
dont change!_ |
| `debezium.sink.iceberg.upsert-op-column` | `__op` | Used with upsert mode. _dont change!_ |
| `debezium.sink.iceberg.destination-regexp` | `` | Regexp to modify destination table. With this its possible to map `table_ptt1`,`table_ptt2` to `table_combined`. |
| `debezium.sink.iceberg.destination-regexp-replace` | `` | Regexp Replace part to modify destination table |
| `debezium.sink.batch.batch-size-wait` | `NoBatchSizeWait` | Batch size wait strategy to optimize data files and upload interval. explained below. |
| `debezium.sink.iceberg.{iceberg.prop.name}` | | [Iceberg config](https://iceberg.apache.org/docs/latest/configuration/) passed to Iceberg |
| Config | Default | Description |
|------------------------------------------------------------------------------------|---------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `debezium.sink.iceberg.warehouse` | | Root path of the Iceberg data warehouse |
| `debezium.sink.iceberg.catalog-name` | `default` | User-specified Iceberg catalog name. |
| `debezium.sink.iceberg.table-namespace` | `default` | A namespace in the catalog. ex: `SELECT * FROM prod.db.table -- catalog: prod, namespace: db, table: table` |
| `debezium.sink.iceberg.table-prefix` | `` | Iceberg table name prefix, Added to destination iceberg table names. |
| `debezium.sink.iceberg.write.format.default` | `parquet` | Default file format for the table; `parquet`, `avro`, or `orc` |
| `debezium.sink.iceberg.allow-field-addition` | `true` | Allow field addition to target tables |
| `debezium.sink.iceberg.upsert` | `true` | Running upsert mode overwriting updated rows. explained below. |
| `debezium.sink.iceberg.upsert-keep-deletes` | `true` | When running with upsert mode, keeps deleted rows in target table. |
| `debezium.sink.iceberg.upsert-dedup-column` | `__source_ts_ms` | With upsert mode used to deduplicate data. row with highest `__source_ts_ms` kept. _dont change!_ |
| `debezium.sink.iceberg.upsert-op-column` | `__op` | Used with upsert mode. _dont change!_ |
| `debezium.sink.iceberg.destination-regexp` | `` | Regexp to modify destination table. With this its possible to map `table_ptt1`,`table_ptt2` to `table_combined`. |
| `debezium.sink.iceberg.destination-regexp-replace` | `` | Regexp Replace part to modify destination table |
| `debezium.sink.batch.batch-size-wait` | `NoBatchSizeWait` | Batch size wait strategy to optimize data files and upload interval. explained below. |
| `debezium.sink.iceberg.{iceberg.prop.name}` | | [Iceberg config](https://iceberg.apache.org/docs/latest/configuration/) passed to Iceberg |
| `debezium.source.offset.storage` | `io.debezium.server.iceberg.offset.IcebergOffsetBackingStore` | The name of the Java class that is responsible for persistence of connector offsets. see [debezium doc](https://debezium.io/documentation/reference/stable/development/engine.html#advanced-consuming) |
| `debezium.source.offset.storage.iceberg.table-name` | `debezium_offset_storage` | Destination table name to store connector offsets. |
| `debezium.source.schema.history.internal` | `io.debezium.server.iceberg.history.IcebergSchemaHistory` | The name of the Java class that is responsible for persistence of the database schema history. see [debezium doc](https://debezium.io/documentation/reference/stable/development/engine.html#database-history-properties) |
| `debezium.source.schema.history.internal.iceberg.table-name` | `debezium_schema_history_storage` | Destination table name to store database schema history. |

### Upsert

Expand Down Expand Up @@ -131,7 +134,7 @@ debezium.source.database.history.iceberg.table-name=debezium_database_history_st
## Debezium Event Flattening
Iceberg consumer requires event flattening. This is required configuration.
Iceberg consumer requires event flattening. This is required/mandatory configuration. For further details on `Message transformations` please see [debezium doc](https://debezium.io/documentation/reference/stable/development/engine.html#engine-message-transformations)
```properties
debezium.transforms=unwrap
Expand Down

0 comments on commit 6c74d9b

Please sign in to comment.