Streaming responses with HTTP.rb

[aka-nez]

Hi! I faced an interesting problem with HTTP.rb and Webmock.

I'm using HTTP.rb for streaming files from S3 through the Sinatra application. And want to test it with RSpec. Files for test reasons are located at localhost, so I allowed requests to localhost like this:

WebMock.disable_net_connect!(allow: 'storage:8080')

I want to stream each received chunk via Sinatra streaming API like this:

  get '/stream' do
    response = HTTP.get('https://postman-echo.com/stream/500')

    stream do |out|
      response.body.each do |chunk|
        out << chunk
      end
    end
  end

And, of course, I want to test it. But here I meet a problem:

Failures:

  1) Simple API /stream succesfully stream data
     Failure/Error: get '/stream'
     
     HTTP::StateError:
       body has already been consumed
     # /Users/nez/.rvm/gems/ruby-2.7.4/gems/http-5.1.1/lib/http/response/body.rb:67:in `stream!'
     # /Users/nez/.rvm/gems/ruby-2.7.4/gems/http-5.1.1/lib/http/response/body.rb:29:in `readpartial'
     # /Users/nez/.rvm/gems/ruby-2.7.4/gems/http-5.1.1/lib/http/response/body.rb:37:in `each'
     # ./app.rb:18:in `block (2 levels) in <class:SimpleAPI>'
     # /Users/nez/.rvm/gems/ruby-2.7.4/gems/sinatra-3.0.5/lib/sinatra/base.rb:509:in `block (2 levels) in stream'
     # /Users/nez/.rvm/gems/ruby-2.7.4/gems/sinatra-3.0.5/lib/sinatra/base.rb:702:in `with_params'
     # /Users/nez/.rvm/gems/ruby-2.7.4/gems/sinatra-3.0.5/lib/sinatra/base.rb:509:in `block in stream'
     # /Users/nez/.rvm/gems/ruby-2.7.4/gems/sinatra-3.0.5/lib/sinatra/base.rb:474:in `block in each'
     # /Users/nez/.rvm/gems/ruby-2.7.4/gems/sinatra-3.0.5/lib/sinatra/base.rb:453:in `defer'
     # /Users/nez/.rvm/gems/ruby-2.7.4/gems/sinatra-3.0.5/lib/sinatra/base.rb:472:in `each'
     # /Users/nez/.rvm/gems/ruby-2.7.4/gems/rack-2.2.6.2/lib/rack/response.rb:277:in `buffered_body!'
     # /Users/nez/.rvm/gems/ruby-2.7.4/gems/rack-2.2.6.2/lib/rack/mock.rb:191:in `initialize'
     # /Users/nez/.rvm/gems/ruby-2.7.4/gems/rack-test-2.0.2/lib/rack/test.rb:360:in `new'
     # /Users/nez/.rvm/gems/ruby-2.7.4/gems/rack-test-2.0.2/lib/rack/test.rb:360:in `process_request'
     # /Users/nez/.rvm/gems/ruby-2.7.4/gems/rack-test-2.0.2/lib/rack/test.rb:165:in `custom_request'
     # /Users/nez/.rvm/gems/ruby-2.7.4/gems/rack-test-2.0.2/lib/rack/test.rb:114:in `get'
     # ./spec/app_spec.rb:6:in `block (3 levels) in <top (required)>'
     # /Users/nez/.rvm/gems/ruby-2.7.4/gems/webmock-3.18.1/lib/webmock/rspec.rb:37:in `block (2 levels) in <top (required)>'

Finished in 2.14 seconds (files took 1.44 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./spec/app_spec.rb:5 # Simple API /stream successfully stream data

I looked at the source code, and probably found an issue.

body.to_s is it a shortcut to read whole body and I think it broke HTTP.rb streaming feature.

So, I want to find the best way to solve this long-term problem. Maybe someone can give me the right direction or any advice?

PS: I set a minimal application to show that case: https://github.com/aka-nez/sinatra-streaming-webmock-error

The related issue at HTTP.rb repo: httprb/http/issues/212

[bblimke]

@aka-nez Thank you for opening this issue, the detailed description. The provided sinatra app was very useful to debug the issue. You have correctly identified the root cause of the problem on WebMock side to be body.to_s call.
I have added a fix to the http_rb_adapter that fixed the problem: 216cad9

The fix is not ideal, as it replaces the original Response#body, but since original body cannot be reset for streaming, this is the best I came up with.

Thank you again for your contribution to identifying and helping me address this issue.

[bblimke]

The fix is now released in WebMock version 3.23.0