Skip to content

Update Azure & AWS SDKs in exchange filesystem #17100

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
kokosing merged 1 commit into from
Apr 25, 2023
Merged

Update Azure & AWS SDKs in exchange filesystem #17100

kokosing merged 1 commit into from
Apr 25, 2023

Conversation

@wendigo
Copy link
Contributor

@wendigo wendigo commented Apr 18, 2023

No description provided.

@cla-bot cla-bot bot added the cla-signed label Apr 18, 2023
hashhar
hashhar reviewed Apr 18, 2023
View reviewed changes
@electrum
Copy link
Member

electrum commented Apr 18, 2023

Lots of Netty conflicts. We should update Netty to 4.1.91.Final

@wendigo
Copy link
Contributor Author

wendigo commented Apr 18, 2023

@electrum I'll do it

@wendigo wendigo force-pushed the serafin/update-aws-azure-sdks branch from 43d5f19 to 6ef8d1c Compare April 19, 2023 08:12
@wendigo wendigo force-pushed the serafin/update-aws-azure-sdks branch 3 times, most recently from 70f1f01 to d9ebe60 Compare April 19, 2023 15:08
@github-actions github-actions bot added the delta-lake Delta Lake connector label Apr 19, 2023
@electrum
Copy link
Member

electrum commented Apr 19, 2023

Looks like the AWS client has a bug or incompatibility with MinIO:

software.amazon.awssdk.services.s3.model.S3Exception: An error occurred when parsing the HTTP request PUT at '/' (Service: S3, Status Code: 400, Request ID: )
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:95)
	at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:270)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:171)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
	at software.amazon.awssdk.services.s3.DefaultS3Client.createBucket(DefaultS3Client.java:1176)
	at io.trino.plugin.exchange.filesystem.containers.MinioStorage.start(MinioStorage.java:65)
	at io.trino.plugin.deltalake.TestDeltaTaskFailureRecoveryTest.createQueryRunner(TestDeltaTaskFailureRecoveryTest.java:55)
	at io.trino.testing.BaseFailureRecoveryTest.createQueryRunner(BaseFailureRecoveryTest.java:109)
	at io.trino.testing.AbstractTestQueryFramework.init(AbstractTestQueryFramework.java:104)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:515)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:217)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:144)
	at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:169)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

@electrum
Copy link
Member

electrum commented Apr 20, 2023

Do you want to split out the Azure updates? I think we also need to test with secrets

@wendigo wendigo force-pushed the serafin/update-aws-azure-sdks branch from d9ebe60 to 5ada112 Compare April 20, 2023 18:45
@wendigo
Copy link
Contributor Author

wendigo commented Apr 20, 2023

@electrum I've fixed the configuration and it should work with the Minio (the problem was the lack of the s3 path style access)

@wendigo
Copy link
Contributor Author

wendigo commented Apr 21, 2023 •
edited
Loading

@electrum Can you run this with secrets?

@hashhar
Copy link
Member

hashhar commented Apr 21, 2023

/test-with-secrets sha=5ada11295297abd6ae8d1fdc337596ab0f13ec79

@github-actions
Copy link

github-actions bot commented Apr 21, 2023

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/4763998549

@wendigo
Copy link
Contributor Author

wendigo commented Apr 21, 2023

Failures are unrelated. PTAL @electrum

@losipiuk
Copy link
Member

losipiuk commented Apr 23, 2023

/test-with-secrets sha=5ada11295297abd6ae8d1fdc337596ab0f13ec79

@losipiuk
Copy link
Member

losipiuk commented Apr 23, 2023

@wendigo is this related?

@github-actions
Copy link

github-actions bot commented Apr 23, 2023

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/4777554151

@wendigo wendigo force-pushed the serafin/update-aws-azure-sdks branch from 5ada112 to 33f73cd Compare April 24, 2023 13:20
@wendigo
Copy link
Contributor Author

wendigo commented Apr 24, 2023

I've rebased this PR. Can you run it with secrets @ebyhr @hashhar ?

@hashhar
Copy link
Member

hashhar commented Apr 24, 2023

/test-with-secrets sha=33f73cd71edd3e27326b48479d4db23b4bd5b398

@github-actions
Copy link

github-actions bot commented Apr 25, 2023

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/4789114677

@wendigo
Copy link
Contributor Author

wendigo commented Apr 25, 2023

This is good to go. Failures are unrelated to this change.

@ebyhr @hashhar ptal

@kokosing kokosing merged commit 1ed247e into trinodb:master Apr 25, 2023
@github-actions github-actions bot added this to the 415 milestone Apr 25, 2023
@colebow colebow mentioned this pull request Apr 26, 2023
Merged
@wendigo wendigo deleted the serafin/update-aws-azure-sdks branch January 21, 2025 11:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@losipiuk losipiuk losipiuk approved these changes

@hashhar hashhar hashhar approved these changes

@kokosing kokosing Awaiting requested review from kokosing

+1 more reviewer

@linzebing linzebing linzebing approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

cla-signed delta-lake Delta Lake connector

Milestone

415

Development

Successfully merging this pull request may close these issues.

6 participants

@wendigo @electrum @hashhar @losipiuk @linzebing @kokosing