Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add S3ForcePathStyle support for s3 binding #1360

Merged
merged 2 commits into from
Dec 24, 2021

Conversation

rainfd
Copy link
Contributor

@rainfd rainfd commented Dec 4, 2021

Description

Add S3ForcePathStyle support for S3 API compatible products, like Minio.

Issue reference

#1322

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

@rainfd rainfd requested review from a team as code owners December 4, 2021 04:25
@ghost
Copy link

ghost commented Dec 4, 2021

CLA assistant check
All CLA requirements met.

s.uploader = s3manager.NewUploader(session)
s.s3Client = s3.New(session, cfg)
s.downloader = s3manager.NewDownloaderWithClient(s.s3Client)
s.uploader = s3manager.NewUploaderWithClient(s.s3Client)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason to change to NewDownloaderWithClient from NewDownloader ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

S3ForcePathStyle only can be set with s3 client or session. I couldn't find a way to set it up in NewDownloader.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

S3ForcePathStyle only can be set with s3 client or session. I couldn't find a way to set it up in NewDownloader.

@rainfd if it can be set in the session why not set it there, since session is already being used?

Copy link
Contributor Author

@rainfd rainfd Dec 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mukundansundar session get by GetClient in authentication/aws/aws.go, setting ForcePathStyle here is also ok, like
func GetClient(accessKey string, secretKey string, sessionToken string, region string, endpoint string, forcePathStyle bool) (*session.Session, error) {
but GetClient is used by other aws componment. Should I add modify here?
Or add more options to GetClient, like
func GetClient(accessKey string, secretKey string, sessionToken string, region string, endpoint string, opts...Option) (*session.Session, error)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks fir explaining that ... And no no need to make change in the common method.
Can you add a docs PR for this and also unit tests to make sure that the metadata is parsed correctly?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mukundansundar test and docs have already added.
dapr/docs#2054

@yaron2
Copy link
Member

yaron2 commented Dec 9, 2021

@mukundansundar please see comment and re-review.

@yaron2
Copy link
Member

yaron2 commented Dec 14, 2021

@rainfd rainfd force-pushed the s3-binding-path-style-access branch from d737ff8 to 3b36d0e Compare December 15, 2021 03:28
@rainfd
Copy link
Contributor Author

rainfd commented Dec 15, 2021

@yaron2 comment has been modified.

@codecov
Copy link

codecov bot commented Dec 19, 2021

Codecov Report

Merging #1360 (2a02745) into master (09fb60c) will decrease coverage by 0.06%.
The diff coverage is 24.30%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1360      +/-   ##
==========================================
- Coverage   35.02%   34.96%   -0.07%     
==========================================
  Files         148      149       +1     
  Lines       12825    12931     +106     
==========================================
+ Hits         4492     4521      +29     
- Misses       7853     7925      +72     
- Partials      480      485       +5     
Impacted Files Coverage Δ
authentication/azure/auth_amqp.go 0.00% <0.00%> (ø)
bindings/aws/kinesis/kinesis.go 2.61% <0.00%> (ø)
bindings/aws/s3/s3.go 15.06% <0.00%> (-0.11%) ⬇️
bindings/azure/eventgrid/eventgrid.go 3.84% <0.00%> (ø)
bindings/mqtt/mqtt.go 25.00% <0.00%> (-1.42%) ⬇️
configuration/redis/redis.go 37.22% <0.00%> (-1.12%) ⬇️
pubsub/rocketmq/rocketmq.go 0.00% <0.00%> (ø)
state/azure/cosmosdb/cosmosdb.go 13.91% <0.00%> (-1.35%) ⬇️
state/query/query.go 23.91% <ø> (ø)
authentication/azure/auth.go 40.35% <1.96%> (-15.75%) ⬇️
... and 12 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4918900...2a02745. Read the comment docs.

Copy link
Contributor

@mukundansundar mukundansundar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@yaron2 yaron2 merged commit 0f97b52 into dapr:master Dec 24, 2021
@yaron2
Copy link
Member

yaron2 commented Dec 24, 2021

@rainfd can you please open a docs PR that updates the metadata?

@rainfd
Copy link
Contributor Author

rainfd commented Dec 24, 2021

@rainfd can you please open a docs PR that updates the metadata?

dapr/docs#2054 What's missing?

@yaron2
Copy link
Member

yaron2 commented Dec 24, 2021

@rainfd can you please open a docs PR that updates the metadata?

dapr/docs#2054 What's missing?

Oh didn't see that, all good.

jigargandhi pushed a commit to jigargandhi/components-contrib that referenced this pull request Jan 21, 2022
* add S3ForcePathStyle support for s3 binding

Signed-off-by: rainfd <[email protected]>

* add S3ForcePathStyle unit test

Signed-off-by: rainfd <[email protected]>
Signed-off-by: jigargandhi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants