Add docs for iceberg expire snapshots and delete orphan files#12093
Conversation
homar
force-pushed
the
homar/add_docs_for_iceberg_expire_snapshots_and_delete_orphan_files
branch
from
c2d7e74 to
382c1b8
Compare
There was a problem hiding this comment.
The procedure affects all snapshots that are older than the time period configured with the retention_threshold parameter.
There was a problem hiding this comment.
drop "config" or use "catalog configuration property" ..
also probably reword the sentence to be specific .. "safety measure" doesnt really spell out what happens .. I think this is the case... snapshots within min-retention time period are not affected by the procedure.. its basically an override ..
example scenario: min-retention set to 30 days
call with retention_threshold 21 days
snapshots from 30 days ago and older are deleted ..
call with 30 days
nothing happens
call with 60 days
snapshots from 60 days ago and older are deleted ..
There was a problem hiding this comment.
nope, you just can't run the procedure with retention_threshold smaller than iceberg.expire_snapshots.min-retention
There was a problem hiding this comment.
ok .. well.. then we need to clarify that in the docs.
And also .. is there an useful error message when you try?
There was a problem hiding this comment.
yes there is an useful message but it is quite long: Retention specified (33.00s) is shorter than the minimum retention configured in the system (7.00d). Minimum retention can be changed with iceberg.expire_snapshots.min-retention configuration property or iceberg.expire_snapshots_min_retention session property
homar
force-pushed
the
homar/add_docs_for_iceberg_expire_snapshots_and_delete_orphan_files
branch
from
382c1b8 to
f4f01e3
Compare
homar
force-pushed
the
homar/add_docs_for_iceberg_expire_snapshots_and_delete_orphan_files
branch
from
351f417 to
f4f01e3
Compare
There was a problem hiding this comment.
This sentence is too convoluted . how about
The value for ``retention_threshold`` must be higher than ``iceberg.expire_snapshots.min-retention`` in the catalog.
f4f01e3 to
f4a3673
Compare
|
@mosabua are we good to go with this ? |
mosabua
left a comment
mosabua
left a comment
There was a problem hiding this comment.
Ideally we wrap the source at 80 char but its more important to get this merged ..
|
@martint could we get this in for 379 .. feature shipped in 378 already.. |
3 participants
Description
fixes #12086
Related issues, pull requests, and links
Documentation
( ) No documentation is needed.
(x) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.
Release notes
( ) No release notes entries required.
( ) Release notes entries required with the following suggested text: