Skip to content
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions site/docs/spark-ddl.md
Original file line number Diff line number Diff line change
Expand Up @@ -360,3 +360,29 @@ ALTER TABLE prod.db.sample WRITE ORDERED BY category ASC NULLS LAST, id DESC NUL
!!! Note
Table write order does not guarantee data order for queries. It only affects how data is written to the table.

Only local sorting can be set at the same time, use `LOCALLY ORDERED BY`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should first state that WRITE ORDERED BY sets a global ordering where rows are ordered across tasks, like using ORDER BY in an INSERT command. Then introduce LOCALLY ORDERED BY to order within each task but not across tasks.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update


```sql
ALTER TABLE prod.db.sample WRITE LOCALLY ORDERED BY category, id
-- use optional ASC/DEC keyword to specify sort order of each field (default ASC)
ALTER TABLE prod.db.sample WRITE LOCALLY ORDERED BY category ASC, id DESC
-- use optional NULLS FIRST/NULLS LAST keyword to specify null order of each field (default FIRST)
ALTER TABLE prod.db.sample WRITE LOCALLY ORDERED BY category ASC NULLS LAST, id DESC NULLS FIRST
```
### `ALTER TABLE ... WRITE DISTRIBUTED BY PARTITION`

Iceberg tables can be configured with a hash distribution where tuples that share the same values for clustering expressions are

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The requirement is to distribute by partition. Hash distribution is an implementation detail. Instead, I think this should state that WRITE DISTRIBUTED BY PARTITION will guarantee that a given partition is handled by one writer.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okey

co-located in the same partition.

To set the write hash for a table, use `WRITE DISTRIBUTED BY PARTITION`:

```sql
ALTER TABLE prod.db.sample WRITE DISTRIBUTED BY PARTITION
```
Iceberg tables can also be configured to use hash distribution and use local sort order.

```sql
ALTER TABLE prod.db.sample WRITE DISTRIBUTED BY PARTITION LOCALLY ORDERED BY category, id
```