-
Notifications
You must be signed in to change notification settings - Fork 3.4k
Docs: Add WRITE LOCALLY ORDERED BY and WRITE DISTRIBUTED BY in spark-ddl.md
#3820
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -360,3 +360,29 @@ ALTER TABLE prod.db.sample WRITE ORDERED BY category ASC NULLS LAST, id DESC NUL | |
| !!! Note | ||
| Table write order does not guarantee data order for queries. It only affects how data is written to the table. | ||
|
|
||
| Only local sorting can be set at the same time, use `LOCALLY ORDERED BY` | ||
|
|
||
| ```sql | ||
| ALTER TABLE prod.db.sample WRITE LOCALLY ORDERED BY category, id | ||
| -- use optional ASC/DEC keyword to specify sort order of each field (default ASC) | ||
| ALTER TABLE prod.db.sample WRITE LOCALLY ORDERED BY category ASC, id DESC | ||
| -- use optional NULLS FIRST/NULLS LAST keyword to specify null order of each field (default FIRST) | ||
| ALTER TABLE prod.db.sample WRITE LOCALLY ORDERED BY category ASC NULLS LAST, id DESC NULLS FIRST | ||
| ``` | ||
| ### `ALTER TABLE ... WRITE DISTRIBUTED BY PARTITION` | ||
|
|
||
| Iceberg tables can be configured with a hash distribution where tuples that share the same values for clustering expressions are | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The requirement is to distribute by partition. Hash distribution is an implementation detail. Instead, I think this should state that
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. okey |
||
| co-located in the same partition. | ||
|
|
||
| To set the write hash for a table, use `WRITE DISTRIBUTED BY PARTITION`: | ||
|
|
||
| ```sql | ||
| ALTER TABLE prod.db.sample WRITE DISTRIBUTED BY PARTITION | ||
| ``` | ||
| Iceberg tables can also be configured to use hash distribution and use local sort order. | ||
|
|
||
| ```sql | ||
| ALTER TABLE prod.db.sample WRITE DISTRIBUTED BY PARTITION LOCALLY ORDERED BY category, id | ||
| ``` | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should first state that
WRITE ORDERED BYsets a global ordering where rows are ordered across tasks, like usingORDER BYin anINSERTcommand. Then introduceLOCALLY ORDERED BYto order within each task but not across tasks.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
update