From 9a46b53d4628740ba7d844ac90c8c74d455572ec Mon Sep 17 00:00:00 2001 From: Alex Rhee Date: Mon, 7 Apr 2025 20:56:29 -0700 Subject: [PATCH] add doc updates for conditional writes lock provider --- website/docs/concurrency_control.md | 8 +++++++- website/docs/configurations.md | 13 +++++++++++++ 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/website/docs/concurrency_control.md b/website/docs/concurrency_control.md index d835e06290f96..e35da25201a4e 100644 --- a/website/docs/concurrency_control.md +++ b/website/docs/concurrency_control.md @@ -39,7 +39,13 @@ When a transaction attempts to modify a resource that is currently locked by ano In case of multi-writing in Hudi, the locks are acquired on the Hudi table for a very short duration during specific phases (such as just before committing the writes or before scheduling table services) instead of locking for the entire span of time. This approach allows multiple writers to work on the same table simultaneously, increasing concurrency and avoids conflicts. -There are 4 different lock providers that require different configurations to be set. Please refer to comprehensive locking configs [here](https://hudi.apache.org/docs/next/configurations#LOCK). +There are 4 different lock providers that require different configurations to be set. 1 lock provider requires no configurations. Please refer to comprehensive locking configs [here](https://hudi.apache.org/docs/next/configurations#LOCK). + +#### Conditional writes based +``` +hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.ConditionalWriteLockProvider +``` +No configs are required for this lock provider, however as of 1.0.2 this lock provider is only supported for S3. #### Zookeeper based ``` diff --git a/website/docs/configurations.md b/website/docs/configurations.md index 0758147e0df0b..4f819fb563e37 100644 --- a/website/docs/configurations.md +++ b/website/docs/configurations.md @@ -1130,6 +1130,19 @@ Configs that control DynamoDB based locking mechanisms required for concurrency | [hoodie.write.lock.wait_time_ms](#hoodiewritelockwait_time_ms) | 60000 | Lock Acquire Wait Timeout in milliseconds
`Config Param: LOCK_ACQUIRE_WAIT_TIMEOUT_MS_PROP_KEY`
`Since Version: 0.10.0` | --- +#### Storage based conditional write locks configurations {#ConditionalWrite-based-Locks-Configurations} +Configs that control conditional write based locking mechanisms required for concurrency control between writers to a Hudi table. Concurrency between Hudi's own table services are auto managed internally. + +[**Advanced Configs**](#ConditionalWrite-based-Locks-Configurations-advanced-configs) + + +| Config Name | Default | Description | +| --------------------------------------------------------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| [hoodie.write.lock.conditional_write.locks_location](#hoodiewritelockconditional_write_locks_location) | (N/A) | For conditional writes based lock provider, the centralized location to write all locks to. If not provided, locks are written to each table's base path: `.hoodie/.locks/table_lock.json`.
`Config Param: CONDITIONAL_WRITE_LOCKS_LOCATION`
`Since Version: 1.0.2.1` | +| [hoodie.write.lock.conditional_write.heartbeat_poll_ms](#hoodiewritelockconditional_write_heartbeat_poll_ms) | 30000 | For conditional writes based lock provider, How often to renew each held lock, default is 30 seconds.
`Config Param: CONDITIONAL_WRITE_HEARTBEAT_POLL_MS`
`Since Version: 1.0.2.0` | +| [hoodie.write.lock.conditional_write.lock_validity_timeout_ms](#hoodiewritelockconditional_write_lock_validity_timeout_ms) | 300000 | For conditional writes based lock provider, the amount of time each new lock is valid before expiring, default is 5 minutes.
`Config Param: CONDITIONAL_WRITE_LOCK_VALIDITY_TIMEOUT_MS`
`Since Version: 1.0.2` | | +--- + ### Key Generator Configs {#KEY_GENERATOR} Hudi maintains keys (record key + partition path) for uniquely identifying a particular record. These configs allow developers to setup the Key generator class that extracts these out of incoming records.