generate_presigned_url no longer works, gives: SignatureDoesNotMatch

[Trogious]

import boto3,requests,os

def get_session():
    access_key = os.getenv('JAR_LAMBDA_ACCESS_KEY')
    secret_key = os.getenv('JAR_LAMBDA_SECRET_KEY')
    if None in [access_key, secret_key]:
        raise Exception('KEYs not set')
    session = boto3.Session(aws_access_key_id=access_key, aws_secret_access_key=secret_key, region_name='eu-central-1')
    return session

def print_url():
    key = 'mykey'
    bucket = 'mybucket'
    try:
        session = get_session()
        s3 = session.client('s3', config=boto3.session.Config(signature_version='s3v4'), region_name='eu-central-1')
        url = s3.generate_presigned_url(ClientMethod='get_object', Params={'Bucket': bucket, 'Key': key}, ExpiresIn=3600)
        print(url)
        resp = requests.get(url)
        print(resp)
    except Exception as e:
        print(e)

The browser gives a 403 and:

<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>
The request signature we calculated does not match the signature you provided. Check your key and signing method.
</Message>
...

Signing with:

aws s3 presign s3://mybuket/mykey --expires 1800 --profile presigner

works perectly. The presigner profile has the same keys as the python code above.

Versions:
Python 3.7.0 (had the same issue with 3.6.*)
boto3-1.7.70
botocore-1.10.70
s3transfer-0.1.13

Please also read this: https://stackoverflow.com/questions/50213740/aws-s3-presigned-urls-with-boto3-signature-mismatch

[kyleknap]

@Trogious I just have some questions to help narrow this down:

1. What does the HTTP request look like that is sent by the browser? More specifically what are the headers?

2. When you use the url from the presign command are you using it with the browser or some other tool?

I suspect that the headers being signed in the presigned url are not the same as the headers the browser may be sending.

[lu1s]

This bug is true. The presigned url is outdated and does not output the same values as other SDKs (for example the Ruby one).
This is a good one (not generated with boto3):
https://[bucketname].s3.us-west-2.amazonaws.com/[key]?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=[good-cred-string]&X-Amz-Date=20180821T180135Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=[good-signature]
And this is what boto3 client generates and no longer works:
https://[bucketname].s3-us-west-2.amazonaws.com/[key]?AWSAccessKeyId=[exposed-access-key-not-encoded]&Signature=[shorter-and-weird-signature]=&Expires=1534877951%27
As you can see, the expires contains a %2 char and not only numbers, and the other params are different.

[Trogious]

@lu1s in your example (the output from boto3 I mean), have you used v4 signature or v2 (the old one)?

[Trogious]

@kyleknap

1. See below.
2. Doesn't matter, I use with browser, python3.requests and curl. Same result on each.

Request headers:

GET /mykey.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIU3PEYH4NWLZWJMA%2F20180822%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20180822T141223Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=34989397276c5559893be86dfcc96123a931a27b6297e0196d07a1492ba0c0f1 HTTP/1.1
Host: testbucket1231231.s3.eu-central-1.amazonaws.com
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,pl;q=0.8

Query string params parsed:

X-Amz-Algorithm: AWS4-HMAC-SHA256
X-Amz-Credential: AKIAIU3PEYH4NWLZWJMA/20180822/eu-central-1/s3/aws4_request
X-Amz-Date: 20180822T141223Z
X-Amz-Expires: 3600
X-Amz-SignedHeaders: host
X-Amz-Signature: 34989397276c5559893be86dfcc96123a931a27b6297e0196d07a1492ba0c0f1

[lu1s]

@Trogious I'm using the v1 I guess. At the end is the only method that boto3 has. I installed boto3 from pip3 so its version is 1.7.82, yet I don't know about which signature version it is).

[imperio59]

After several hours of hitting this same issue, changing the addressing_style to 'path' made everything work for me, including custom headers...

See docs here: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3.html#changing-the-addressing-style

[alexivkin]

@imperio59 Fixed my issue, you da man! Does this mean the root of the issue is how DNS propogates through AWS for S3 buckets - and will go away on its own after all the propagation is done?

[imperio59]

@alexivkin I don't think so, I just think that's the URL style that works with this library :D

[hamx0r]

@imperio59 's answer helped, but I also needed to set signature_version='s3v4':

s3 = boto3.client('s3', aws_access_key_id={YOUR ACCESS KEY}, aws_secret_access_key={YOUR SECRET ACCESS},
                  config=Config(s3={'addressing_style': 'path'}, signature_version='s3v4'))

boto3 1.9.28 (1.9.34 is latest available)
botocore 1.12.28 (1.12.34 is latest available)
python 3.6.4 (MacOS)

[imperio59]

@hamx0r Yes I also had to set the signature_version to s3v4 to make it work :)

[Senitram666]

I'm having a similar problem, but I'm using django-storages library for interacting with boto3.
the main problem I had at first was the url generated was something like:
https://[bucketname].s3.amazonaws.com/media/2H83eeY.jpg
and the working url generated directly by S3 was
https://s3.us-east-2.amazonaws.com/[bucketname]/media/2H83eeY.jpg
but this could be corrected by the custom domain parameter.
But, for the files saved with acl=private, the custom domain must be false wich lead django storages to use
url = self.bucket.meta.client.generate_presigned_url('get_object', Params=params,
ExpiresIn=expire) (from https://github.com/jschneier/django-storages/blob/95fca7bef964178c9653c099e00267e9bd3846d7/storages/backends/s3boto3.py#L622 )
and the urls returned this way aren't working.

can someone help me correct this?

boto3=1.9.63
botocore=1.12.63
django-storages=1.7.1
python=3.6.7

[lu1s]

Hello @Senitram666 . Could you properly pre-sign the URL that it gets back to you with a key in it? It should be a long URL so it includes the key that will work as the pre-sign. Hope that makes sense. Cheers in advance.

[Senitram666]

Hello @lu1s
I couldn't, I'm not able to use any of the objects maked as private inside my bucket. Also, I don't know exactly if my problem is caused by django-storages(I followed the tutorial about it on SimpleIsBetterThanComplex) or by the boto3 recent upgrades, but the url generated by the library always results in an error...

[lu1s]

Hello @Senitram666 ,
Check first, please, the objects accesibility and your security groups. They need to be available for your API caller to execute the URL signature and retreival and that can be acheived by having proper configurations.
Also, check that if doing it without the session. Probably the new way to go is like in this gitst.
https://gist.github.com/heitorlessa/ea2450684b050ecc3cfe789db1717bb9

[joguSD]

@lu1s There's nothing inherently wrong with either of those URLs you posted previously, just that they're using two different signature versions.

It is possible that switching from virtual to path addressing style could fix pre-signed URLs signed using SigV4. For SigV4 the host is signed as part of the signature, which can cause problems for newly created buckets when virtual addressing style is being used. S3 will redirect to a different host for buckets that DNS haven't propagated for, which leads signature mismatch errors as the host is no longer correct. My hunch is that this is the case most people are running into.

It's also worth nothing that some regions only support SigV4 and you'll need to ensure that you're using it for those regions.

Using the latest version of boto3 I was able to successfully generate and use a presigned URL created with all permutations of SigV2/SigV4 and path/virtual addressing style for a bucket that existed for quite some time.

Let me know if that clears things up.

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[sejas]

Thanks @Trogious !
We had the same issue SignatureDoesNotMatch uploading an image from our React Native APP.

I fixed it passing the signature_version='s3v4' and the region_name

s3 = session.client('s3',
                    config=boto3.session.Config(signature_version='s3v4'),
                    region_name='eu-central-1'
)

[rshad]

Just In case none of the solutions above represents your case, however, I had a similar issue trying @sejas's solution, it did not work. In my case, I found that the Public Access Block was disabled for S3 bucket and just by enabling the Block it started working.

[marcmaxson]

I needed BOTH s3v4 and addressing_style:path in my Config() for it to work with boto3.

from botocore.config import Config
my_config = Config(
    region_name = 'us-east-2',
    signature_version = 's3v4',
    retries = {
        'max_attempts': 10,
        #'mode': 'standard'
    },
    s3={'addressing_style': 'path'},
)

... and then the functional part ...

client = boto3.client('s3', config=my_config,
    aws_access_key_id = AWS_ACCESS_KEY_ID,
    aws_secret_access_key = AWS_SECRET_ACCESS_KEY
)

bucket_name = <bucket name with no leading or trailing slashes>
key_name = <look up your `key` in the console for this S3 object, not the URL>

response = client.generate_presigned_url(
    'get_object',
    Params={'Bucket': bucket_name, 'Key': key_name},
    ExpiresIn=604800,
    HttpMethod=None
)

print(response)

Not sure how to make the URL last 7 days as requested. That has to do with the IAM I used, but working on it.

[wodCZ]

Without addressing_style: path I get this log:

Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x7f5cc39fccb0>
Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x7f5cc39fcc20>
Event before-parameter-build.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7f5cacb5bc90>>
Event before-parameter-build.s3.GetObject: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7f5caceac4d0>>
Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x7f5cc39fca70>
Event choose-signer.s3.GetObject: calling handler <function set_operation_specific_signer at 0x7f5cc39fc950>
Event before-sign.s3.GetObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7f5cacb61550>>
Defaulting to S3 virtual host style addressing with path style addressing fallback.
Checking for DNS compatible bucket for: https://s3.eu-central-1.amazonaws.com/bucket/file.txt
URI updated to: https://bucket.s3.amazonaws.com/file.txt
Calculating signature using v4 auth.
CanonicalRequest:
GET
/file.txt
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=MYKEY%2F20201119%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20201119T141500Z&X-Amz-Expires=7200&X-Amz-SignedHeaders=host
host:bucket.s3.amazonaws.com

host

Which generates this url:

https://bucket.s3.amazonaws.com/file.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=MYKEY%2F20201119%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20201119T141500Z&X-Amz-Expires=7200&X-Amz-SignedHeaders=host&X-Amz-Signature=signature

Which when opened redirects to this url:

https://bucket.s3.eu-central-1.amazonaws.com/file.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=MYKEY%2F20201119%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20201119T141500Z&X-Amz-Expires=7200&X-Amz-SignedHeaders=host&X-Amz-Signature=signature

Where CanonicalRequest changes the host too:

GET 
/file.txt 
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=MYKEY%2F20201119%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20201119T141500Z&X-Amz-Expires=7200&X-Amz-SignedHeaders=host
host:bucket.s3.eu-central-1.amazonaws.com

host
UNSIGNED-PAYLOAD

For some reason the code uses host:bucket.s3.amazonaws.com while S3 expects host:bucket.s3.eu-central-1.amazonaws.com.

When I initialize the client with region specified, the local canonical host is still the same: host:bucket.s3.amazonaws.com

Edit: got it working with config=Config(s3={"addressing_style": "virtual"})

[Thomas-McKanna]

Any reason why using s3v4 and setting addressing_style: path are not default behavior? Most of the time boto3 just seems to work, but presigned S3 URLs requires knowing about these special configurations.