Skip to content

Docs: Add stream-from-timestamp in spark-configuration.md #3732

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rdblue merged 6 commits into from
Jan 4, 2022
Merged

Docs: Add stream-from-timestamp in spark-configuration.md #3732

Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
384e041
Add stream-from-timestamp in spark-configuration.md
rajarshisarkar Dec 13, 2021
1d1582f
Implement review comments
rajarshisarkar Dec 14, 2021
7a1957c
Implement review comments
rajarshisarkar Dec 15, 2021
88c89e8
Implement review comments
rajarshisarkar Dec 22, 2021
ad98e2d
Implement review comments
rajarshisarkar Jan 4, 2022
f9a65dd
Update spark-configuration.md
rdblue Jan 4, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions site/docs/spark-configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,7 @@ spark.read
| file-open-cost | As per table property | Overrides this table's read.split.open-file-cost |
| vectorization-enabled | As per table property | Overrides this table's read.parquet.vectorization.enabled |
| batch-size | As per table property | Overrides this table's read.parquet.vectorization.batch-size |
| stream-from-timestamp | Long.MIN_VALUE | Timestamp in milliseconds, start streaming this table from the first snapshot that occurs at or after this timestamp |

@rdblue rdblue Dec 20, 2021 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that the default is not set, so the default behavior is not to stream from a timestamp but to stream from the oldest snapshot.

@rdblue rdblue Dec 21, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may also want to update this to "first known ancestor snapshot" and add a note:

!!! Note
    If `stream-from-timestamp` is before the oldest ancestor snapshot in the table, the oldest ancestor will be used.

@rajarshisarkar rajarshisarkar Dec 22, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, for the inputs @rdblue.

The default value Long.MIN_VALUE is set here: https://github.com/apache/iceberg/blob/master/spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/SparkReadConf.java#L215-L220

I have updated the wordings to "first known ancestor snapshot".

I have added the note about the default behavior. Do let me know if I should keep the note in the default column & remove Long.MIN_VALUE.

@rdblue rdblue Dec 29, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove Long.MIN_VALUE. While that's in the code, we want to document behavior, not the specific implementation.

@rajarshisarkar rajarshisarkar Jan 4, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification, @rdblue. I have pushed the change.


### Write options

Expand Down