Skip to content

Configure retries for trino-filesystem-gcs#20003

Merged
electrum merged 3 commits intotrinodb:masterfrom
elonazoulay:gcs_retry
Dec 13, 2023
Merged

Configure retries for trino-filesystem-gcs#20003
electrum merged 3 commits intotrinodb:masterfrom
elonazoulay:gcs_retry

Conversation

@elonazoulay
Copy link
Member

Description

Resolves #19943

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

@cla-bot cla-bot bot added the cla-signed label Dec 3, 2023
@elonazoulay elonazoulay requested a review from electrum December 3, 2023 00:57
@github-actions github-actions bot added tests:hive bigquery BigQuery connector labels Dec 3, 2023
@elonazoulay
Copy link
Member Author

Notes:
Tried different settings for the retries and optimized for the lowest amount of requests, i.e. maxAttempts.
Other settings that worked:

  • Only set gcs.retry.max-attempts: 200 - This works has no back off and issues more calls than the default settings

Max attempts < 20 and delay multiplier < 2.0 also resulted in 429's.

Also: the retry strategy needs to be uniform and the trino-filesystem api ensures that the idempotency requirements are not violated - see retries

@ebyhr
Copy link
Member

ebyhr commented Dec 5, 2023

/test-with-secrets sha=b932b90bcfadfeaf7cea8fc940a5b39f00af0b7f

@github-actions
Copy link

github-actions bot commented Dec 5, 2023

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/7096161114

@elonazoulay
Copy link
Member Author

Looks like the failure above was unrelated, trying again.

@elonazoulay
Copy link
Member Author

/test-with-secrets sha=b932b90bcfadfeaf7cea8fc940a5b39f00af0b7f

@elonazoulay
Copy link
Member Author

/test-with-secrets sha=52edc07ebeaef3375db3a88049033e30287b938a

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove this config. Users just care that it works, so we should make it work automatically. After reading that page, I'm still not sure why you would change it from UNIFORM to DEFAULT, so I doubt any users would either.

@electrum
Copy link
Member

electrum commented Dec 9, 2023

Looks good overall. We definitely need this as the tests are flaky now (and might mean that queries will also fail).

Multiple modules depend on org.threeten.bp.Duration.
Move to the root pom.
@electrum
Copy link
Member

I'll merge this for now since it's in good shape and fixes the frequent test failures. We can follow up on the comments.

@Test
void testCreateFileRetry()
{
assertThatNoException().isThrownBy(() -> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused by the purpose of assertThatNoException(). Any exception should fail the test. Isn't that the normal behavior of unit tests?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it captures intent nicely - for the case when you just want to check code runs fine and do not do any assertions.
Otherwise, if there is code without assertions in test method, it kinda looks that test writer forgot about sth.

@electrum electrum merged commit 3be8c2f into trinodb:master Dec 13, 2023
@github-actions github-actions bot added this to the 435 milestone Dec 13, 2023
@elonazoulay
Copy link
Member Author

Thanks! I will push a follow up pr with the remaining suggestions to reduce the time and add some comments to explain the test.

@mosabua
Copy link
Member

mosabua commented Dec 14, 2023

@elonazoulay are you sending docs? And does this need a release notes entry?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bigquery BigQuery connector cla-signed

Development

Successfully merging this pull request may close these issues.

Flaky TestDeltaLakeGcsConnectorSmokeTest.testOptimizeUsingForcedPartitioning

6 participants