"The request signature we calculated does not match the signature you provided." when uploading with Aws::S3::Object#upload_file

[BanzaiMan]

Travis CI has recently switched to use AWS SDK v2 for uploading artifacts to S3. The up-to-date code is found at https://github.com/travis-ci/dpl/blob/master/lib/dpl/provider/s3.rb

This works well for the most part, but a few isolated cases have been reported where the request signature does not match what's expected.

1. https://travis-ci.org/rebar/rebar3/jobs/79769044#L5696 (Aws::S3::Errors::SignatureDoesNotMatch)
2. https://travis-ci.org/octoblu/gateblu-forever/jobs/79543777#L409 (Aws::S3::MultipartUploadError)

The code is quite simple:

1. Initialize Aws::S3::Resource with appropriate options (https://github.com/travis-ci/dpl/blob/75f10f342cc3e5d31c6cd45a2997966233d1deb6/lib/dpl/provider/s3.rb#L10)
2. And upload with Aws::S3::Object#upload_file (https://github.com/travis-ci/dpl/blob/75f10f342cc3e5d31c6cd45a2997966233d1deb6/lib/dpl/provider/s3.rb#L56)

There is no custom signature generation from our code, so I am not sure what explains this exception.

I appreciate any help.

ref: https://github.com/travis-ci/travis-ci/issues/4776

[trevorrowe]

Off the cuff, I do not know what the cause would be. One very interesting piece of information comes from the second failed job linked. It was performing a multipart upload. It appears that all 7 parts failed with the same error.

Do you know if these are temporal errors that resolve with a retry?

[BanzaiMan]

https://travis-ci.org/octoblu/gateblu-forever/jobs/79709836 and https://travis-ci.org/octoblu/gateblu-forever/jobs/79896596 indicate that the issue persists. An interesting fact about this case is that the corresponding Linux build (https://travis-ci.org/octoblu/gateblu-forever/jobs/79709835), which had run earlier, was successful.

The "rebar" case seems a bit different, in that it only runs on Linux (the latest of which is https://travis-ci.org/rebar/rebar3/jobs/79951129#L5691), so the multiple retries do not come into play.

[trevorrowe]

It is difficult to say if these are related, but an issue was found in the classic S3 signature code that would cause a signature does not match error when the bucket name is stuttered as part of the S3 object key. For example, if I attempted the following:

s3 = Aws::S3::Resource.new
s3.bucket('bucket-name').object('bucket-name/key').upload_file('path')

Then it would raise an error when using the classic signature code, but not with signature version 4. The classic code is currently in use in most, but not all bucket regions. I'll put the fix up shortly, reference this issue. I suspect this will be released tomorrow. Hopefully this will resolve the issue.

[trevorrowe]

The fix was released yesterday with v2.1.21. Let me know when you get the chance to update and test this version.

[BanzaiMan]

@trevorrowe Thanks! There is at least one instance where 2.1.21 was able to upload (https://travis-ci.org/rebar/rebar3/jobs/80520689#L5710-L5731) where previously 2.1.20 threw an error.

I'm waiting to hear from other users. If I get a few more samples to confirm that it's working, we can close this. :-D

[trevorrowe]

Any more news on this? Was the patch successful?

[BanzaiMan]

Yes, I think so! Thank you for your help.

[BanzaiMan]

Hello. A user reports a regression (https://github.com/travis-ci/travis-ci/issues/4776#issuecomment-148918938) with 2.1.30. https://travis-ci.org/CentricWebEstate/pieisreal/builds/85906778#L5588

/home/travis/.rvm/gems/ruby-1.9.3-p551/gems/aws-sdk-core-2.1.30/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call': The request signature we calculated does not match the signature you provided. Check your key and signing method. (Aws::S3::Errors::SignatureDoesNotMatch)

Should I reopen this issue, or open a new one?

[trevorrowe]

@BanzaiMan Is it possible to find out what configuration options were used when uploading? There is potentially related issue here #949.

[BanzaiMan]

@trevorrowe

deploy:
  provider: s3
  ⋮
  bucket: pieisreal-sheets
  local-dir: release
  upload-dir: "$TRAVIS_BUILD_NUMBER"
  acl: public_read
  region: ap-southeast-2
  skip_cleanup: true
  on:
    repo: CentricWebEstate/pieisreal

It is sending to Sydney. I don't know how big the file is.

[trevorrowe]

@BanzaiMan Thanks. Lets open a new issue and reference this. Without more information I can't know if they are related, or simply different signature issues. I will need more information to debug this, so lets continue the discussion there.