Skip to content

Docs: Enhance create_changelog_view usage #9889

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
flyrain merged 1 commit into from
Mar 13, 2024
Merged

Docs: Enhance create_changelog_view usage #9889

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions docs/docs/spark-procedures.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -756,8 +756,8 @@ Creates a view that contains the changes from a given table.
| `table` | ✔️ | string | Name of the source table for the changelog |
| `changelog_view` | | string | Name of the view to create |
| `options` | | map<string, string> | A map of Spark read options to use |
| `net_changes` | | boolean | Whether to output net changes (see below for more information). Defaults to false. |
| `compute_updates` | | boolean | Whether to compute pre/post update images (see below for more information). Defaults to false. |
| `net_changes` | | boolean | Whether to output net changes (see below for more information). Defaults to false. It must be false when `compute_updates` is true. |
| `compute_updates` | | boolean | Whether to compute pre/post update images (see below for more information). Defaults to true if `identifer_columns` are provided; otherwise, defaults to false. |
| `identifier_columns` | | array<string> | The list of identifier columns to compute updates. If the argument `compute_updates` is set to true and `identifier_columns` are not provided, the table’s current identifier fields will be used. |

Here is a list of commonly used Spark read options:
Expand Down Expand Up @@ -823,7 +823,10 @@ second snapshot deleted 1 record.
|2 | Bob |INSERT |0 |5390529835796506035|
|1 | Alice |DELETE |1 |8764748981452218370|

Create a changelog view that computes net changes. It removes intermediate changes and only outputs the net changes.
#### Net Changes

The procedure can remove intermediate changes across multiple snapshots, and only outputs the net changes. Here is an example to create a changelog view that computes net changes.

```sql
CALL spark_catalog.system.create_changelog_view(
table => 'db.tbl',
Expand Down