Support skip archive for Glue in Iceberg by ebyhr · Pull Request #14336 · trinodb/trino

ebyhr · 2022-09-28T03:28:58Z

Description

TestIcebergGlueCatalogSkipArchive.testSkipArchive fails until updating IAM policy.
Fixes #13413

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# Iceberg
* Add support for skipping archive when committing a table in Glue when
  the `iceberg.glue.skip-archive` configuration property is set to true. ({issue}`13413`)

ebyhr · 2022-09-29T09:03:03Z

@electrum Can you add glue:GetTableVersions and glue:BatchDeleteTableVersion action in IAM for presto-ci account? Sample screenshot:

erichwang · 2022-10-12T00:12:47Z

For what it's worth, I think we should just get rid of glue archiving altogether. Either that, or at some point we should make it default skip archiving. Otherwise, every Trino Iceberg cluster will blow up after so many thousands of transactions in Glue (which shouldn't be the default behavior out of the box).

ebyhr · 2022-10-12T04:57:53Z

To support removing old archives, we need two more API calls (getTableVersions & batchDeleteTableVersion{Async}) & two more IAM permissions (GetTableVersions & BatchDeleteTableVersion). I lean toward enabling the flag by default in the future.

findinpath · 2022-10-12T14:25:59Z

...ino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/glue/IcebergGlueCatalogConfig.java

Why false by default?

For backward compatibility. I can enable by default if there's no objection.

it's unlikely that most people care about the Glue table versions for Iceberg tables, but it's also not unreasonable that someone uses that.

i think we should have it disabled (false) for now

Agreed, disabled by default makes sense.

findinpath · 2022-10-12T14:26:45Z

Do you plan any follow-ups on hive/delta as well?

findepi · 2022-10-13T14:36:11Z

...g/src/main/java/io/trino/plugin/iceberg/catalog/glue/GlueIcebergTableOperationsProvider.java

Looks like we want to unify this with plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/glue/TrinoGlueCatalogFactory.java by providing some Glue client provider interface, especially that createAsyncGlueClient has this parameter Optional

Added a preparatory commit to unify them. Let me know if it's different from your intention.

findepi · 2022-10-14T12:46:09Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/glue/GlueClientProvider.java

+public class GlueClientProvider
+        implements Provider<AWSGlueAsync>
+{
+    private final AWSGlueAsync glueClient;


i think the Provider.get should do the instantiation so that the provided instance can be bound with various scopes.
ie. this works correctly only because the Module uses SINGLETON.

findepi · 2022-10-14T12:47:53Z

...ino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/glue/IcebergGlueCatalogConfig.java

it's unlikely that most people care about the Glue table versions for Iceberg tables, but it's also not unreasonable that someone uses that.

i think we should have it disabled (false) for now

findepi · 2022-10-14T12:49:08Z

...no-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/glue/SkipArchiveRequestHandler.java

+        if (request instanceof UpdateTableRequest updateTableRequest) {
+            return updateTableRequest.withSkipArchive(skipArchive);
+        }
+        return request;


Glue Catalog uses only a couple of different requests.
Can we enumerate all these, to ensure this class gets updated if any new request variant is added?
for example, UpdateTableRequestV2 or something?

electrum · 2022-10-18T23:31:01Z

@ebyhr sorry for the delay. Done now.

colebow · 2022-11-01T23:55:45Z

@ebyhr does this need to be included in docs as a new property, or is this just propagation of a property that already exists elsewhere?

findinpath · 2022-11-02T05:42:05Z

@Config("iceberg.glue.skip-archive")
@ConfigDescription("Skip archiving an old table version when creating a new version in a commit")

This one needs to be documented. (follow-up PR)

cla-bot bot added the cla-signed label Sep 28, 2022

findepi assigned ebyhr Sep 28, 2022

ebyhr force-pushed the ebi/iceberg-glue-skip-archive branch 2 times, most recently from 9f653a8 to affa7c8 Compare October 12, 2022 04:54

findinpath reviewed Oct 12, 2022

View reviewed changes

findinpath approved these changes Oct 12, 2022

View reviewed changes

ebyhr requested review from electrum and findepi October 13, 2022 00:03

ebyhr force-pushed the ebi/iceberg-glue-skip-archive branch 2 times, most recently from 48e5dfa to 616e5b7 Compare October 13, 2022 12:33

findepi reviewed Oct 13, 2022

View reviewed changes

ebyhr force-pushed the ebi/iceberg-glue-skip-archive branch from 616e5b7 to 81d22c7 Compare October 14, 2022 03:32

findepi reviewed Oct 14, 2022

View reviewed changes

findepi approved these changes Oct 14, 2022

View reviewed changes

ebyhr force-pushed the ebi/iceberg-glue-skip-archive branch 2 times, most recently from 152f88c to bbb3182 Compare October 26, 2022 05:45

ebyhr added 2 commits October 26, 2022 20:31

Extract Glue client provider in Iceberg

c683c28

Support skip archive for Glue in Iceberg

fbcc74b

ebyhr force-pushed the ebi/iceberg-glue-skip-archive branch from bbb3182 to fbcc74b Compare October 26, 2022 11:31

ebyhr merged commit d0eadf7 into master Oct 27, 2022

ebyhr deleted the ebi/iceberg-glue-skip-archive branch October 27, 2022 05:15

ebyhr mentioned this pull request Oct 27, 2022

Release notes for 402 #14780

Closed

github-actions bot added this to the 402 milestone Oct 27, 2022

colebow mentioned this pull request Nov 1, 2022

Add Trino 402 release notes #14862

Merged

blvckcoffee mentioned this pull request Apr 4, 2023

Add docs for property to skip glue archive #16880

Merged

findinpath mentioned this pull request Nov 14, 2023

Skip archiving in Glue an old table version during a table update operation #19739

Merged

nineinchnick mentioned this pull request Oct 17, 2024

Add hive.metastore.glue.skip-archive config option #23817

Merged

Conversation

ebyhr commented Sep 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Release notes

Uh oh!

ebyhr commented Sep 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

erichwang commented Oct 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ebyhr commented Oct 12, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

findinpath commented Oct 12, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

electrum commented Oct 18, 2022

Uh oh!

colebow commented Nov 1, 2022

Uh oh!

findinpath commented Nov 2, 2022

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

6 participants

ebyhr commented Sep 28, 2022 •

edited

Loading

ebyhr commented Sep 29, 2022 •

edited

Loading

erichwang commented Oct 12, 2022 •

edited

Loading