Skip to content
Merged
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions docs/docs/spark-procedures.md
Original file line number Diff line number Diff line change
Expand Up @@ -974,6 +974,38 @@ Collect statistics of the snapshot with id `snap1` of table `my_table` for colum
CALL catalog_name.system.compute_table_stats(table => 'my_table', snapshot_id => 'snap1', columns => array('col1', 'col2'));
```

## Partition Statistics

### `compute_partition_stats`

This procedure computes the stats incrementally from the last snapshot that has `PartitionStatisticsFile` until the given

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: a 'PartitionStatisticsFile' (need article for complete sentence)

@szehon-ho szehon-ho Jul 14, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the stats => partitions statistics.

By the way, can we link this text the spec about partition stats? (like the above command)

snapshot (uses current snapshot if not specified) and writes the combined result into a `PartitionStatisticsFile`
after merging the partition stats. It performs a full compute if previous statistics file does not exist. It also registers the

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should specify type of stats file, and also add article:

if the previous partition statistics file

@szehon-ho szehon-ho Jul 14, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets get rid of 'merging the partition stats'? I feel its implied from 'incrementally' and 'combined'

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit (needs article and specify stats file): 'the previous partition statistics file'

`PartitionStatisticsFile` to table metadata.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the table metadata (need article)


| Argument Name | Required? | Type | Description |
|---------------|-----------|---------------|---------------------------------------------------------------------------------------|
| `table` | ✔️ | string | Name of the table |
| `snapshot_id` | | string | Id of the snapshot to collect partition stats. By default current snapshot id is used |

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'is used' sounds unnecessary. Hm how about just: defaults to current snapshot id?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

collect => compute (follows the command name)


#### Output

| Output Name | Type | Description |
|-------------------|--------|-----------------------------------------------------------|
| `partition_statistics_file` | string | Path to partition stats file created from by this command |

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'to the partition stats file'. (add article)

Also , 'from by', only need one. Use by?


#### Examples

Collect partition statistics of the latest snapshot of table `my_table`
```sql
CALL catalog_name.system.compute_partition_stats('my_table');
```

Collect partition statistics of the snapshot with id `snap1` of table `my_table`
```sql
CALL catalog_name.system.compute_partition_stats(table => 'my_table', snapshot_id => 'snap1' );

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small nit: extra space at end.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

```

## Table Replication

The `rewrite_table_path` procedure prepares an Iceberg table for copying to another location.
Expand Down