Separate CTE materialization session Properties from exchange Materialization#21625
Conversation
61e6304 to
d9b0043
Compare
a91ee90 to
d00e266
Compare
|
Updated CTE tests to use the new property |
|
Could documentation for this new session property be added? Possibly in Properties Reference. |
d00e266 to
9165e6f
Compare
|
Codenotify: Notifying subscribers in CODENOTIFY files for diff 00faa91...ab406a4.
|
9165e6f to
179a9b0
Compare
steveburnett
left a comment
There was a problem hiding this comment.
Thanks for the documentation! One nit only.
There was a problem hiding this comment.
| This setting determines how many buckets/writers should be used when materializing the CTEs, potentially affecting the performance of queries involving CTE materialization. | |
| This setting determines how many buckets or writers should be used when materializing the CTEs, potentially affecting the performance of queries involving CTE materialization. |
Avoid slashes. See GitLab doc recommended word list entry for slashes for advice and suggestions.
179a9b0 to
b43067a
Compare
|
Added the docs, but isn't this page redundant since we already have @ConfigDescription? Does it make more sense to add @ConfigDescription to all new session properties and pull them in the docs page during releases? |
steveburnett
left a comment
There was a problem hiding this comment.
LGTM@ (docs)
Pull branch, local build, docs look great. Thanks!
That's a great idea! I'll look into this possibility for generating the docs. Thanks very much for bringing it up. |
b43067a to
cedb080
Compare
cedb080 to
4827f25
Compare
|
@steveburnett Added documentation for cte-materialization-strategy property, please review |
steveburnett
left a comment
There was a problem hiding this comment.
LGTM! (docs)
Pull branch, local build of docs, looks good. Thanks!
tdcmeehan
left a comment
There was a problem hiding this comment.
LGTM % nit on the first commit, and please also add session property documentation.
Please make sure it follows our guidelines. Suggested:
Add cte_hash_partition_count and config property
This commit adds the `cte_hash_partition_count`, and corresponding config property `query.cte-hash-partition-count`, which controls number of writers for CTE materialization effectively
steveburnett
left a comment
There was a problem hiding this comment.
Took another look, two minor suggestions.
4827f25 to
a4b68e6
Compare
steveburnett
left a comment
There was a problem hiding this comment.
The tiniest of nits.
This commit adds the `cte_hash_partition_count`, and corresponding config property `query.cte-hash-partition-count`, which controls number of writers for CTE materialization effectively Also adds `cte_partitioning_provider_catalog`, and corresponding config `query.cte-partitioning-provider-catalog` which specifies which catalog should be used for CTE materialization and specifies partitioning
a4b68e6 to
ab406a4
Compare
steveburnett
left a comment
There was a problem hiding this comment.
LGTM! (docs)
Pull branch, local build of docs.
Current testing has shown that recommended value is 4x but further performance testing might be needed.
Meta recommends setting hash partition for materialized exchange as 5X - 10X the cluster size in exchange materialization and we do not want that as the hash partition count of non materialized exchanges
https://prestodb.io/docs/current/admin/exchange-materialization.html#using-exchange-materialization
This is needed since its good separation between features since these are cross used
Description
Motivation and Context
Impact
Test Plan
Contributor checklist
Release Notes
Please follow release notes guidelines and fill in the release notes below.