Intermittent "NoCredentialProviders: no valid providers in chain" Errors

[danielwhelansb]

Currently in our production system we are intermittently getting the "NoCredentialProviders: no valid providers in chain" errors when trying to download a file via the S3 API (using the golang aws-sdk-go).

We did change the MTU on the instances to be 1500 instead of 9200 (default) because the machine was having issues downloading images from DockerHub. We are unsure if this caused it. We can't really put it back to 9200 to see if thats an issue because it fails to deploy our image at 9200 MTU and it is not very easy to reproduce.

I turned on verbose mode for the logging and got these errors

NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY not found in environment
SharedCredsLoad: failed to load shared credentials file
caused by: open /root/.aws/credentials: no such file or directory
UnknownError: unknown error

The environment would error because we don't set any environment variables with the keys. The shared creds would fail because we don't have a shared creds file setup but the final check which is the EC2 Role Provider fails with an UnknownError.
I tracked this error down to https://github.com/aws/aws-sdk-go/blob/master/aws/corehandlers/handlers.go#L101

This must mean we are getting some kind of error response code from the MetaData Service. I am already retrying our initial API calls (5 times, with a 1 second sleep between each retry) but it doesn't seem to be good enough.

Any suggestions?

[Shervanator]

This was our original ticket just for reference: #527

[jasdel]

Thanks for the extra information @Shervanator. I'm taking a look at this issue. If you're able to would you mind adding extra debug logging to your application? You can do this by setting the SDK's log level. This log level will give us the wire request/response data.

You can set the log level just for the EC2 instance role, and will only produce wire logs of the requests for credentials from the ec2metadata service.

sess := session.New()
sess.Config.Credentials = ec2rolecreds.NewCredentialsWithClient(
     ec2metadata.New(sess, aws.NewConfig().WithLogLevel(aws.LogDebugWithHTTPBody),
)

// use session within code
svc := s3.New(sess)

Alternatively you can set the logging globally add debug logging to your application's usage of AWS requests.

sess := session.New(aws.NewConfig().WithLogLevel(aws.LogDebugWithHTTPBody))

svc := s3.New(sess)
//...

[jasdel]

It looks like the EC2Metadata client is actually just discarding the body of the errored response. This should be a pretty simple change to add the body response as apart of the r.Error value

[LourdesMoran]

How did you finally fix the intermittent error? I am getting it too (also hard to replicate), but I haven't changed the MTU of my container.