Aws::S3::Errors::InvalidArgument when specifying an alternative endpoint

[800a7b32]

Experiencing an issue with the ruby aws-sdk (v2) with Rails 4.2.5. When using a Ceph endpoint rather than S3 itself (specifically, we're using https://vault.ecloud.co.uk but have also tried https://objects.dreamhost.com in line with the Ceph docs: http://docs.ceph.com/docs/master/radosgw/s3/ruby/).

When using the following setup, we get the Aws::S3::Errors::InvalidArgument response.

Aws.config.update({
  endpoint: ENV['aws_endpoint'],
  access_key_id: ENV['aws_access_key_id'],
  secret_access_key: ENV['aws_secret_access_key'],
  force_path_style: true,
  region: ENV['aws_region']
})

If we drop endpoint and force_path_style and just use our standard S3 credentials, all works fine.

[awood45]

I'm actually not able to reproduce this. Can you give me your stack trace from this error, and maybe specific parameters (if possible) that create this issue?

Also, which version of the SDK are you using?

[800a7b32]

Gemfile.lock for versions:

aws-sdk (2.2.1)
      aws-sdk-resources (= 2.2.1)
    aws-sdk-core (2.2.1)
      jmespath (~> 1.0)
    aws-sdk-resources (2.2.1)
      aws-sdk-core (= 2.2.1)

Specific task (rake task run nightly): https://gist.github.com/andretanguy/f29a8302450a39e290b0

For params, I have the following: AWS_ENDPOINT: 'https://vault.ecloud.co.uk', AWS_REGION: 'default'

Stacktrace:

Uploading to backup servers...
rake aborted!
Aws::S3::Errors::InvalidArgument: 
/Users/andre/Documents/Web_Applications/VesselHQ/vhq-v3/vendor/bundle/ruby/2.2.0/gems/aws-sdk-core-2.2.1/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
/Users/andre/Documents/Web_Applications/VesselHQ/vhq-v3/vendor/bundle/ruby/2.2.0/gems/aws-sdk-core-2.2.1/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:18:in `call'
/Users/andre/Documents/Web_Applications/VesselHQ/vhq-v3/vendor/bundle/ruby/2.2.0/gems/aws-sdk-core-2.2.1/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
/Users/andre/Documents/Web_Applications/VesselHQ/vhq-v3/vendor/bundle/ruby/2.2.0/gems/aws-sdk-core-2.2.1/lib/seahorse/client/plugins/response_target.rb:21:in `call'
/Users/andre/Documents/Web_Applications/VesselHQ/vhq-v3/vendor/bundle/ruby/2.2.0/gems/aws-sdk-core-2.2.1/lib/seahorse/client/request.rb:70:in `send_request'
/Users/andre/Documents/Web_Applications/VesselHQ/vhq-v3/vendor/bundle/ruby/2.2.0/gems/aws-sdk-core-2.2.1/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
/Users/andre/Documents/Web_Applications/VesselHQ/vhq-v3/vendor/bundle/ruby/2.2.0/gems/aws-sdk-resources-2.2.1/lib/aws-sdk-resources/services/s3/multipart_file_uploader.rb:43:in `initiate_upload'
/Users/andre/Documents/Web_Applications/VesselHQ/vhq-v3/vendor/bundle/ruby/2.2.0/gems/aws-sdk-resources-2.2.1/lib/aws-sdk-resources/services/s3/multipart_file_uploader.rb:34:in `upload'
/Users/andre/Documents/Web_Applications/VesselHQ/vhq-v3/vendor/bundle/ruby/2.2.0/gems/aws-sdk-resources-2.2.1/lib/aws-sdk-resources/services/s3/file_uploader.rb:32:in `upload'
/Users/andre/Documents/Web_Applications/VesselHQ/vhq-v3/vendor/bundle/ruby/2.2.0/gems/aws-sdk-resources-2.2.1/lib/aws-sdk-resources/services/s3/object.rb:231:in `upload_file'
/Users/andre/Documents/Web_Applications/VesselHQ/vhq-v3/lib/tasks/backup.rake:32:in `send_to_storage'
/Users/andre/Documents/Web_Applications/VesselHQ/vhq-v3/lib/tasks/backup.rake:23:in `block (2 levels) in <top (required)>'
Tasks: TOP => pg:backup
(See full trace by running task with --trace)

[awood45]

Try modifying your code like so:

s3 = Aws::S3::Resource.new(http_wire_trace: true)
result = s3.bucket(app_name).object(backup_file_name).upload_file(backup_file_location)

Given your custom endpoint situation, and common request, I suspect that a server-side issue is likely. A wire trace would help to confirm.

[800a7b32]

Uploading to backup servers...
opening connection to vault.ecloud.co.uk:443...
opened
starting SSL for vault.ecloud.co.uk:443...
SSL established
<- "POST dump.sql?uploads HTTP/1.1\r\nContent-Type: \r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby2/2.2.1 ruby/2.2.3 x86_64-darwin14 resources\r\nX-Amz-Date: 20151212T203812Z\r\nHost: vault.ecloud.co.uk\r\nX-Amz-Content-Sha256: xxxx-removed-xxxx\r\nAuthorization: AWS4-HMAC-SHA256 Credential=xxxxxx-removed-xxxxxx/default/s3/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=xxxx-removed-xxxx\r\nContent-Length: 0\r\nAccept: */*\r\n\r\n"
-> "HTTP/1.1 400 Bad Request\r\n"
-> "x-amz-request-id: tx0000000000000003a260f-00566c85b6-1bfe76e-default\r\n"
-> "Content-Length: 81\r\n"
-> "Accept-Ranges: bytes\r\n"
-> "Content-type: application/xml\r\n"
-> "Date: Sat, 12 Dec 2015 20:38:14 GMT\r\n"
-> "\r\n"
reading 81 bytes...
-> "<?xml version=\"1.0\" encoding=\"UTF-8\"?><Error><Code>InvalidArgument</Code></Error>"
read 81 bytes
Conn keep-alive
rake aborted!

[awood45]

Without further information, or a reproducing example against a public S3 endpoint, this looks like an issue with the endpoint you're trying to call. The error you're getting is coming from the server, it's cast as Aws::S3::Errors::InvalidArgument because of the error code returned by the server.

I'd recommend at this stage that you reach out to the endpoint provider and cut a bug report with them. Providing the wire trace may help in that regard. Feel free to reach out if we can be of any more help.

[joraff]

Came across this today trying to use another Ceph installation. Pretty sure this is because Ceph versions < Jewel only supports aws2 signatures. DreamObjects is Hammer at the time of my posting.

[joraff]

FYI, Dreamhost just upgraded DreamObjects to the Jewel release. I switched to using this gem over fog on one app and so far so good. I'll report back if I discover any issues.