Support multiple HTTP client libraries -- maybe via Faraday?

[stevecj]

It would be nice to have a choice of what HTTP client library to use. Could get capability via the faraday gem (https://github.com/lostisland/faraday).

[lsegal]

We already have support for multiple HTTP libraries via a pluggable interface. We previously supported httparty, and we currently still support curb, as separate HTTP handlers. A handler is simply a class that responds to #handle with an HTTP request and response object. You can see the implementations in lib/aws/core/http.

You can replace the handler by specifying the :http_handler option either to the global config or to individual clients constructors with your handler object:

class MyHandler
  def handle(req, resp) ... end
end

AWS.config(http_handler: MyHandler.new)

# or

AWS::S3.new(http_handler: MyHandler.new)

See the AWS.config documentation for more.

[trevorrowe]

Another option would be to create a HTTP handler that uses Faraday for transport. Customers could configure the Faraday apapter.

[stevecj]

Ah -- I had thought that HTTP handlers were only used for insertion and not for replacement. I suppose, if you can insert, you can replace though.

[trevorrowe]

With the recently announced work on v2, we are going to put on hold requests requests like this to the basic client layer. We are looking into how we can make these things more pluggable in the future and will consider this suggestion.

[p7r]

It would appear that in fact, this is now impossible in v2 and you actively decided to not support such a feature.

We are having really poor performance throughput using Net::HTTP which Seahorse insists on, and can't see an easy way to get Typhoeus via Faraday plugged in. When building a micro-service architecture and wanting to publish SNS messages as fast as possible, this is kind of a blocker.

Is there any update on this issue?

[trevorrowe]

@p7r The v2 AWS SDK for Ruby supports pluggable HTTP clients. We currently only ship and support the Net::HTTP backed client because it is universally available. I can put together a proof-of-concept gist that should provide guidance on how to do this.

[trevorrowe]

@p7r I've added a gist with a sample implementation here: https://gist.github.com/trevorrowe/ae55df222d45cb424366c98bdc5e1e4e

[p7r]

@trevorrowe - thank you, that's really, really helpful. We could not see that at all, it wasn't obvious.

I think we might line up a typhoeus adapter and offer it up on pull request - might be worth documenting this up and providing examples with the SDK. I'll talk to the team next week.

[trevorrowe]

If you decide to go down that path and want to offer up a pull-request, reach out to me so we can talk about packaging. We may choose to not bundle it as part of the SDK, as we are trying our best to limit dependencies and ensure those we have are portable. I suppose we could ship it with a soft dependency, or potentially it could be made available as another gem, e.g. aws-sdk-typheous. I'm open to discussion on how that might work.

[rzaharenkov]

We are also struggling because of poor performance of net/http (limited by GIL).

I think faraday gem is de facto standard in ruby world. If everybody will re-implement his own handler for each HTTP client library it will be duplication of work that's already done in faraday. faraday is really light gem and brings only 1 additional dependency. I would highly recommend to consider this option. People will like it.