Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ArgumentError: no time information in "Expiration" when establishing a connection to DynamoDB #880

Closed
funglaub opened this issue Jul 22, 2015 · 6 comments
Labels
response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days.

Comments

@funglaub
Copy link

The described exception happens occasionally (very few times in the last four weeks in a busy production environment) when a connection to DynamoDB is being established.

The connection is established like that:

 @client = Aws::DynamoDB::Client.new({region: 'eu-west-1'})

Here's the relevant (gem specific) stacktrace of the exception:

/usr/local/lib/ruby/2.2.0/time.rb:252→ make_time
/usr/local/lib/ruby/2.2.0/time.rb:364→ parse
aws-sdk-core-2.0.48/lib/aws-sdk-core/instance_profile_credentials.rb:72→ refresh
aws-sdk-core-2.0.48/lib/aws-sdk-core/refreshing_credentials.rb:20→ initialize
aws-sdk-core-2.0.48/lib/aws-sdk-core/instance_profile_credentials.rb:49→ initialize
aws-sdk-core-2.0.48/lib/aws-sdk-core/credential_provider_chain.rb:63→ new
aws-sdk-core-2.0.48/lib/aws-sdk-core/credential_provider_chain.rb:63→ instance_profile_credentials
aws-sdk-core-2.0.48/lib/aws-sdk-core/credential_provider_chain.rb:11→ block in resolve
aws-sdk-core-2.0.48/lib/aws-sdk-core/credential_provider_chain.rb:10→ each
aws-sdk-core-2.0.48/lib/aws-sdk-core/credential_provider_chain.rb:10→ resolve
aws-sdk-core-2.0.48/lib/aws-sdk-core/plugins/request_signer.rb:37→ block in <class:RequestSigner>
aws-sdk-core-2.0.48/lib/seahorse/client/configuration.rb:64→ call
aws-sdk-core-2.0.48/lib/seahorse/client/configuration.rb:64→ call
aws-sdk-core-2.0.48/lib/seahorse/client/configuration.rb:199→ block in resolve_defaults
aws-sdk-core-2.0.48/lib/seahorse/client/configuration.rb:57→ each
aws-sdk-core-2.0.48/lib/seahorse/client/configuration.rb:57→ each
aws-sdk-core-2.0.48/lib/seahorse/client/configuration.rb:198→ resolve_defaults
aws-sdk-core-2.0.48/lib/seahorse/client/configuration.rb:194→ value_at
aws-sdk-core-2.0.48/lib/seahorse/client/configuration.rb:183→ block in resolve
/usr/local/lib/ruby/2.2.0/set.rb:283→ each_key
/usr/local/lib/ruby/2.2.0/set.rb:283→ each
aws-sdk-core-2.0.48/lib/seahorse/client/configuration.rb:183→ resolve
aws-sdk-core-2.0.48/lib/seahorse/client/configuration.rb:171→ apply_defaults
aws-sdk-core-2.0.48/lib/seahorse/client/configuration.rb:144→ build!
aws-sdk-core-2.0.48/lib/seahorse/client/base.rb:70→ build_config
aws-sdk-core-2.0.48/lib/seahorse/client/base.rb:21→ initialize
aws-sdk-core-2.0.48/lib/aws-sdk-core/client_stubs.rb:19→ initialize
aws-sdk-core-2.0.48/lib/seahorse/client/base.rb:107→ new

I'm using version 2.0.48 of the aws-sdk gem. For about 99.9% of the time, the connection can be established and everything works like it should.

The code is executed on an EC2 instance, authorization to DynamoDB is realised with an IAM Role linked to the EC2 instance. I think the whole progress of obtaining security credentials is described in the docs if I'm not mistaken.

@trevorrowe
Copy link
Member

Could you try applying the following patch to your code? That error would only happen if the value received from the instance metadata service for the "expiration" was the value "Expiration", i.e. Time.parse('Expiration') produces the error you are describing.

class Aws::InstanceProfileCredentials
  def refresh
    retry_count = 0
    begin
      json = get_credentials
      c = Json.load(json)
      @credentials = Credentials.new(
        c['AccessKeyId'],
        c['SecretAccessKey'],
        c['Token']
      )
      @expiration = c['Expiration'] ? Time.parse(c['Expiration']) : nil
    rescue ArgumentError => error
      puts "NO TIME INFORMATION: #{c.inspect} - #{json.inspect}"
      if retry_count < 3
        retry_count += 1
        retry
      else
        raise(error)
      end
    end
  end
end

You should replace the puts statement with something that can log to something more durable than standard out. This should:

  • Let us know if it is a temporal issue that resolves itself
  • Give us more information about the response that is generating this error

EDIT: Modified the snippet

@funglaub
Copy link
Author

The patch is online. I'll provide you with some information once the error occurs again.

@trevorrowe
Copy link
Member

Any update on this? Has the error reoccured yet?

@funglaub
Copy link
Author

Nope, I'm sorry. Still waiting for information on this.

@funglaub
Copy link
Author

Finally I can provide some feedback.

My current patch looks like that:

module Aws
  class InstanceProfileCredentials
    def refresh
      retry_count = 0
      begin
        credentials = MultiJson.load(get_credentials)
        @access_key_id = credentials['AccessKeyId']
        @secret_access_key = credentials['SecretAccessKey']
        @session_token = credentials['Token']
        if expires = credentials['Expiration']
          @expiration = Time.parse(expires)
        else
          @expiration = nil
        end
      rescue ArgumentError, MultiJson::ParseError => e
        airbrake_params = { get_credentials: get_credentials }
        Airbrake.notify(e, parameters: airbrake_params, cgi_data: ENV.to_hash)
        retry_count += 1
        retry if retry_count < 3
      end
    end
  end
end

In an error case the notification parameters were (the actual value returned from the Method get_credentials):

"get_credentials" => 
  "{\n  \"Code\" : \"Success\",\n  \"LastUpdated\" : \"2015-10-14T07:12:58Z\",\n  \"Type\" : \"AWS-HMAC\",\n  \"AccessKeyId\" : \"ASIAIRJ7CEAJFCFUQJ7A\",\n  \"SecretAccessKey\" : \"xxxxxxxxxxxxx\",\n  \"Token\" : \"xxxxxxxxx",\n  \"Expiration\" : \"2015-10-14T13:20:33Z\"\n}������������"

@trevorrowe
Copy link
Member

Hmm.. unfortunately, that error is unrelated to the other one. Without your patch applied, the current release of the SDK currently handles json parsing errors. There have been a few other changes to make parsing responses from the instance metadata service more robust. I'll be pushing another change shortly that will capture and retry the errors related to extracting the expiration time.

Thanks for collecting the information.

awood45 added a commit that referenced this issue Oct 22, 2015
@diehlaws diehlaws added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed information requested labels Jan 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days.
Projects
None yet
Development

No branches or pull requests

3 participants