AWS v4 authorization [JIRA: RCS-144]

[waxzce]

AWS S3 have a new protocol to sign request, called aws4

seems not supported on riak-cs

[reiddraper]

Further reading.

[randysecrist]

For further reference: aws4 implementation in rinamo:

https://github.com/basho-labs/rinamo/blob/master/src/rinamo_auth_keystone_v2.erl#L45-L47

[reiddraper]

Thanks @randysecrist!

[waxzce]

So you think about to port it in riakCS ?

[reiddraper]

@waxzce We'll consider this for Riak CS.

[shino]

s3cmd also implemented v4 auth and use it by default [1].
To access current Riak CS, one must add one configuration

signature_v2 = True

[1] s3tools/s3cmd#402

[Basho-JIRA]

TODO list after some reading of AWS docs and a little trial to execute v4 to AWS S3
https://gist.github.com/shino/447a57ee9a6c2af10823

_[posted via JIRA by Shunichi Shinohara]_

[shino]

temporary patch for boto
shino/boto@bb5e02b

Will be PRed or filed as issues to upstream repository.

[shino]

First round PR is #1062 .
Subsequent tasks are #1059 #1058 #1060 .

[brmzkw]

Currently, riak-cs returns a 500 error when v4 signature is used. Shouldn't it return a HTTP/400 instead?

[shino]

@brmzkw I thought I changed riak_cs to return 400 at #1062 (merged to current develop branch).
What version do you use? Prior 1.5.x, riak cs does not consider v4 auth.

[brmzkw]

I'm running version 1.5.4, so I guess I need to wait for the next release.

Returning a 500 in case of error is really disturbing, as it makes think to the client that riak-cs has some problems. A bad request should never raise HTTP/500.

Thanks for the response,

[shino]

Returning a 500 in case of error is really disturbing

I tested Riak CS (almost) 1.5.4 with s3cmd with v4 enabled. I got 403. This is because Authorization header starts with AWS4-HMAC- but Riak CS 1.5.4 expected AWS, then CS could not get access key ID and (thru authz path...) responded with 403.

[dragonfax]

This would be great to have. I use golang. Amazon has an official s3 client for golang out now, but it only contains v4 auth. Which means it doesn't work with Riak CS.

[kuenishi]

Although it is not generally available and is partial implementation, in 2.0.1 v4 authentication is included. Try adding {auth_v4_enabled, true} in advanced.config.

[dragonfax]

Ah, I tried their client with Riak CS 2.0.1 but didn't know about the config option. I'll try it again. Thanks.

[dragonfax]

Unfortunately this didn't work either. I'm not having much luck finding a workable solution for Riak CS and golang, together.

I've tried various client libraries, and combinations of v2 and v4 signing. I even patched them for #1166

But i just get crashes in the riak cs auth code. And my erlang isn't good enough to debug it.

No problems with using s3cmd to test these configurations and users.

[shino]

@dragonfax Thanks for information and effort! Agree with the subtleness of debugging auth phase 😓

Current implementation is tested against boto [1] as automated testing and against s3cmd by some manual testing.

If you kindly try more, a hint for debugging, which you may know already :) , is turning debug log on in riak cs, then it prints out CanonicalString and StringToSign. (Scope should be output but not ...) [2]
I guess well-implemented client library also have such functionality, for example, go sdk seems to have it [3].

[1] https://github.com/basho/riak_cs/tree/develop/client_tests/python/boto_tests
[2] https://github.com/basho/riak_cs/blob/develop/src/riak_cs_s3_auth.erl#L210-L219
[3] https://github.com/aws/aws-sdk-go/blob/master/internal/signer/v4/v4.go#L141-L165

[dragonfax]

Hey, Thanks for the interest guys. Its always heartwarming to be heard. I did actually get this to work with a bit more effort.

Adding a patch to aws-sdk-go (for issue #1166) that you can see in this commit. dragonfax/aws-sdk-go@cd6e377

I was able to get an example s3 client to work. https://gist.github.com/dragonfax/58d40bf9469cd06b6dc2 And it works quite well, too. Kudos.

If Riak CS could be patched up to accept that whitespace more flexibly. Then that might all you need for good golang support.

[shino]

@dragonfax Congrats! Also thanks a lot for sharing more information and suggestions!

[shino]

Cross note: the spaces after commas will be trimmed by the fix #1235.

[edwardotis]

I see this bug is still open 2.5 years later in 2017. As far as I can tell, the riaks project is not compatible with AWS S3 due to this bug. I cannot set a current aws sdk release to use signers older than v4.

[Basho-JIRA]

[~pbrewer] - This comment came in today. I wonder if this is something you or your team may be able to address?

[posted via JIRA by Derek Somogyi]

[Basho-JIRA]

[~pbrewer] - This comment came in today. I wonder if this is something you or your team may be able to address?

[posted via JIRA by Derek Somogyi]

[Basho-JIRA]

[~pbrewer] - This comment came in today. I wonder if this is something you or your team may be able to address?

[posted via JIRA by Derek Somogyi]

[waxzce]

Hi,
Does someone is working on this or contribution are welcome?

Best Regards,

[dragonfax]

My company dropped Riak CS for reasons such as this, and is moving away from Riak because of other various issues.