Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

iam role (access token) is refreshed for every s3 request #1117

Closed
fviard opened this issue Jun 27, 2020 · 1 comment · Fixed by #1118
Closed

iam role (access token) is refreshed for every s3 request #1117

fviard opened this issue Jun 27, 2020 · 1 comment · Fixed by #1118
Labels
to-be-fixed
Milestone
2.2.0

Comments

@fviard
Copy link
Contributor

fviard commented Jun 27, 2020

iam role (access token) is refreshed for every s3 request , and so, that can result in a lot slower operations as a request for credential has to be done each time, even if access token is not expired.

A nice fix would be to take into account the expiration date of the token, to only refresh it when it is needed.
@fviard fviard added this to the 2.2.0 milestone Jun 27, 2020
Patouche added a commit to Patouche/s3cmd that referenced this issue Jun 27, 2020
@Patouche
[s3tools#1117] Improve speed when running on EC2 instance
8b4de3d 
Fix http call on role endpoint to retrieve the token each time the role_refresh in invoke
@Patouche Patouche mentioned this issue Jun 27, 2020
Merged
fviard added a commit that referenced this issue Jun 27, 2020
@fviard
Merge pull request #1118 from Patouche/issues/1117
8d30061 
[#1117] Improve speed when running on EC2 instance
@fviard
Copy link
Contributor Author

fviard commented Jun 27, 2020

Here are the results of a benchmark I did with python3:
For around 16k files of a typical "/usr/share" directory:

  • With @Patouche fix: 17min 34s
  • Without: 20min 21s

👍

@Patouche Patouche mentioned this issue Jun 27, 2020
Merged
fviard added a commit that referenced this issue Jun 27, 2020
@fviard
Merge pull request #1119 from Patouche/fix/datetime-tz
a412966 
#1117 Fix timezone on old python version
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
to-be-fixed
Projects
None yet
Milestone
2.2.0
Development

Successfully merging a pull request may close this issue.

[#1117] Improve speed when running on EC2 instance Patouche/s3cmd
1 participant
@fviard