Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

InstanceProfileCredentials fail occasionally #913

Closed
phorsuedzie opened this issue Aug 28, 2015 · 2 comments
Closed

InstanceProfileCredentials fail occasionally #913

phorsuedzie opened this issue Aug 28, 2015 · 2 comments
Labels
feature-request A feature should be added or improved.

Comments

@phorsuedzie
Copy link

Every now and then (read: once or twice a month, running 5 EC2 instances 24/7), InstanceProfileCredentials fail to work as expected.

It looks more like a service problem than a genuine aws-sdk issue, but the SDK could try to handle that exceptional states nevertheless (as it does with retrying the network request) - maybe retry here too and hope for a better response ...

There have been two variants so far:

Incomplete JSON data

Aws::Json::ParseError: 795: unexpected token at ''
 [GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/aws-sdk-core/json.rb:28 :in `rescue in load`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/aws-sdk-core/json.rb:26 :in `load`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/aws-sdk-core/instance_profile_credentials.rb:68 :in `refresh`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/aws-sdk-core/refreshing_credentials.rb:20 :in `initialize`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/aws-sdk-core/instance_profile_credentials.rb:50 :in `initialize`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/aws-sdk-core/credential_provider_chain.rb:64 :in `new`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/aws-sdk-core/credential_provider_chain.rb:64 :in `instance_profile_credentials`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/aws-sdk-core/credential_provider_chain.rb:12 :in `block in resolve`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/aws-sdk-core/credential_provider_chain.rb:11 :in `each`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/aws-sdk-core/credential_provider_chain.rb:11 :in `resolve`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/aws-sdk-core/plugins/request_signer.rb:37 :in `block in <class:RequestSigner>`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:64 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:64 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:199 :in `block in resolve_defaults`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:57 :in `each`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:57 :in `each`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:198 :in `resolve_defaults`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:194 :in `value_at`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:183 :in `block in resolve`
/usr/local/lib/ruby/2.1.0/set.rb:263 :in `each_key`
/usr/local/lib/ruby/2.1.0/set.rb:263 :in `each`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:183 :in `resolve`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:171 :in `apply_defaults`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:144 :in `build!`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/base.rb:68 :in `build_config`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/base.rb:19 :in `initialize`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/base.rb:105 :in `new`
[GEM_ROOT]/gems/aws-sdk-resources-2.1.13/lib/aws-sdk-resources/resource.rb:169 :in `extract_client`
[GEM_ROOT]/gems/aws-sdk-resources-2.1.13/lib/aws-sdk-resources/resource.rb:15 :in `initialize`
...

Missing credentials in response

NoMethodError: undefined method `[]' for nil:NilClass
 [GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/aws-sdk-core/instance_profile_credentials.rb:70 :in `refresh`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/aws-sdk-core/refreshing_credentials.rb:20 :in `initialize`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/aws-sdk-core/instance_profile_credentials.rb:50 :in `initialize`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/aws-sdk-core/credential_provider_chain.rb:64 :in `new`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/aws-sdk-core/credential_provider_chain.rb:64 :in `instance_profile_credentials`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/aws-sdk-core/credential_provider_chain.rb:12 :in `block in resolve`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/aws-sdk-core/credential_provider_chain.rb:11 :in `each`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/aws-sdk-core/credential_provider_chain.rb:11 :in `resolve`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/aws-sdk-core/plugins/request_signer.rb:37 :in `block in <class:RequestSigner>`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:64 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:64 :in `call`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:199 :in `block in resolve_defaults`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:57 :in `each`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:57 :in `each`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:198 :in `resolve_defaults`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:194 :in `value_at`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:183 :in `block in resolve`
/usr/local/lib/ruby/2.1.0/set.rb:263 :in `each_key`
/usr/local/lib/ruby/2.1.0/set.rb:263 :in `each`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:183 :in `resolve`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:171 :in `apply_defaults`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/configuration.rb:144 :in `build!`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/base.rb:68 :in `build_config`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/base.rb:19 :in `initialize`
[GEM_ROOT]/gems/aws-sdk-core-2.1.13/lib/seahorse/client/base.rb:105 :in `new`
[GEM_ROOT]/gems/aws-sdk-resources-2.1.13/lib/aws-sdk-resources/resource.rb:169 :in `extract_client`
[GEM_ROOT]/gems/aws-sdk-resources-2.1.13/lib/aws-sdk-resources/resource.rb:15 :in `initialize`
@trevorrowe
Copy link
Member

Thank you for reporting these errors and stack traces. I agree that it should be more resilient and attempt to retry these errors. Let me take a look at this.

@trevorrowe
Copy link
Member

I've put together a patch that will catch both of these scenarios and expanded the unit tests to cover these as well. Thank you for your help tracking down these issues!

@diehlaws diehlaws added feature-request A feature should be added or improved. and removed enhancement labels Jan 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature-request A feature should be added or improved.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@trevorrowe @phorsuedzie @srchase @diehlaws