Skip to content

Mitigate Glue ThrottlingException via backoff retry strategy #28815

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
chenjian2664 merged 1 commit into from
Mar 25, 2026
Merged

Mitigate Glue ThrottlingException via backoff retry strategy #28815

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/glue/GlueMetastoreModule.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@
import software.amazon.awssdk.services.glue.GlueClient;
import software.amazon.awssdk.services.glue.GlueClientBuilder;
import software.amazon.awssdk.services.glue.model.ConcurrentModificationException;
import software.amazon.awssdk.services.glue.model.ThrottlingException;
import software.amazon.awssdk.services.sts.StsClient;
import software.amazon.awssdk.services.sts.StsClientBuilder;
import software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider;
Expand Down Expand Up @@ -151,7 +152,7 @@ public static GlueClient createGlueClient(GlueHiveMetastoreConfig config, @ForGl
glue.overrideConfiguration(builder -> builder
.executionInterceptors(ImmutableList.copyOf(executionInterceptors))
.retryStrategy(retryBuilder -> retryBuilder
.retryOnException(throwable -> throwable instanceof ConcurrentModificationException)
.retryOnException(throwable -> throwable instanceof ConcurrentModificationException || throwable instanceof ThrottlingException)
Comment thread
chenjian2664 marked this conversation as resolved.
.backoffStrategy(BackoffStrategy.exponentialDelay(
java.time.Duration.ofMillis(20),
java.time.Duration.ofMillis(1500)))
Expand Down
5 changes: 5 additions & 0 deletions plugin/trino-iceberg/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -454,6 +454,11 @@
<artifactId>kms</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>retries-spi</artifactId>
<scope>runtime</scope>
</dependency>

<dependency>
<groupId>software.amazon.awssdk</groupId>
Expand Down
19 changes: 18 additions & 1 deletion ...rino-iceberg/src/test/java/io/trino/plugin/iceberg/catalog/glue/TestTrinoGlueCatalog.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,12 +35,17 @@
import io.trino.spi.security.PrincipalType;
import io.trino.spi.security.TrinoPrincipal;
import org.junit.jupiter.api.Test;
import software.amazon.awssdk.retries.api.BackoffStrategy;
import software.amazon.awssdk.services.glue.GlueClient;
import software.amazon.awssdk.services.glue.GlueClientBuilder;
import software.amazon.awssdk.services.glue.model.ConcurrentModificationException;
import software.amazon.awssdk.services.glue.model.ThrottlingException;

import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.List;
import java.util.Map;
import java.util.Optional;
Expand Down Expand Up @@ -86,7 +91,7 @@ protected void createNamespaceWithProperties(TrinoCatalog catalog, String namesp

private TrinoCatalog createGlueTrinoCatalog(boolean useUniqueTableLocations, boolean useSystemSecurity)
{
GlueClient glueClient = GlueClient.create();
GlueClient glueClient = createGlueClient();
IcebergGlueCatalogConfig catalogConfig = new IcebergGlueCatalogConfig();
return new TrinoGlueCatalog(
new CatalogName("catalog_name"),
Expand All @@ -110,6 +115,18 @@ private TrinoCatalog createGlueTrinoCatalog(boolean useUniqueTableLocations, boo
directExecutor());
}

private static GlueClient createGlueClient()
{
GlueClientBuilder glue = GlueClient.builder();

glue.overrideConfiguration(builder -> builder
.retryStrategy(retryBuilder -> retryBuilder
.retryOnException(throwable -> throwable instanceof ConcurrentModificationException || throwable instanceof ThrottlingException)
.backoffStrategy(BackoffStrategy.exponentialDelay(Duration.ofMillis(20), Duration.ofMillis(1500)))
.maxAttempts(10)));
return glue.build();
}

/**
* Similar to {@link #testNonLowercaseNamespace()}, but creates the Glue database via Glue API, in case Glue starts allowing non-lowercase names.
*/
Expand Down