Performance issues due to non-blocking IO

[Byron]

The subject is the result of some debugging done while trying to trace down the source of my problems. These shall be explained in detail as I am sure the authors of this project and possibly hyper will know a workaround or fix.

When trying to upload a multi-gigabyte video to youtube, I notice that the transfer doesn't start until the entire input stream was read into memory. The call-trace looks like this:

Note the io::copy call coming from the hyper client, which I believe would block in case of a standard TCP connection.
A process trying to send a 1.1GB video (after loading it into memory) looks like this - note the 1.2GB of real memory:

A second observation is the high CPU usage. Even if the entire input is cached, I would have expected the send operation to saturate the 100Mb connection to the internet, yet the CPU seems to be bottlenecking, producing a mere ~2.5Mb per second.

When looking at the tcp send calls of the process, they will come in at about 4 to 5 calls per second, which seems to confirm that the CPU is not producing enough encrypted bytes.

Finally, this call-trace shows that the actual TCP send operation commences only when hyper attempts to end the operation.

The CPU bottleneck seems to be originating within OpenSSL, but that's just a guess of mine.

How to reproduce

Download the youtube3 executable from the Google APIs for Rust release page, then extract and run:

$ ./youtube3 videos insert -r snippet title=sometitle description=TBD  tags=test category-id=22 ..status public-stats-viewable=true privacy-status=private -u simple SOME_BIG_MOVIE_FILE.mov application/octet-stream"
# Once authenticated, it will attempt an upload to your youtube account. It fails in any way due to another reason (https://github.com/serde-rs/serde/issues/65), so it will mean no harm to try it.

Meta

• Using openssl 0.6.1 (registry+https://github.com/rust-lang/crates.io-index)

✗ rustc --version
rustc 1.0.0-nightly (f46c4e158 2015-04-20) (built 2015-04-20)

CC @seanmonstar @reem

Thanks for your help ! This is a major issue for me and I can help if you give me some hints on how to solve this.

[seanmonstar]

Here's what help I can give (I know 0 zero zip nil of libopenssl):

• When hyper finishes writing a request, it calls Request<Streaming>.end(), which will write the ending (such as if the request is chunked), and then call flush on the underlying stream.
• I see in rust-openssl that flush is when the bytes are copied from the Ssl buffer into the TcpStream: https://github.com/sfackler/rust-openssl/blob/master/openssl/src/ssl/mod.rs#L913
• Until flush has been called, the bytes seem to be put into a memory buffer on the Ssl object: https://github.com/sfackler/rust-openssl/blob/master/openssl/src/ssl/mod.rs#L645

Again, I do not understand the internals of openssl, but I'd imagine that it has support for flushing its buffer multiple times in a request. If so, it'd be possible to put a cap on the ssl buffer size, and flush each time the cap is reached (similar to std::io::BufferedWriter).

In the meantime, this could probably be forced by calling flush on the hyper::client::Request<Streaming> every so often.

[Byron]

Hi @seanmonstar , thanks for chiming in. I agree that calling flush after each write would certainly solve my problem, yet it would have to be something hyper would have to do, which in turn seems like an evil hack.

Here is some more information I have gathered previously:
From what I could see, openssl supports blocking and non-blocking IO naturally. There is a function called BIO_set_nbio(int) which should be set to zero to get blocking IO.
There are many other clients who use openssl who don't have this particular issue either.
Maybe there is a chance to review the way openssl-rust handles things, perhaps the option could be exposed, for example.

Something that caught my attention is the implementation of MemBio, which I would read as 'Memory Blocking IO' ... it makes no sense to me that it would have non-blocking semantics by default.

So far I couldn't find any hunk of code who would explicitly enable non-blocking IO, neither in hyper nor here. From what I see, openssl uses blocking-IO by default, as people usually ask to get non-blocking IO.

Hopefully @sfackler will find some time, as I hope we can get this fixed before 1.0 is released.

[sfackler]

I think this boils down to the lack of a call to write_through at the end of the Write implementation: https://github.com/sfackler/rust-openssl/blob/master/openssl/src/ssl/mod.rs#L901. It is definitely absurd to buffer gigabytes in memory before writing them out!

[Byron]

Great to hear you have pinpointed the problem's source ! I will be sitting tight and try it out once the issue is closed. Hopefully this will also fix the 'CPU is the bottleneck' issue I observed previously.

[sfackler]

I just pushed a commit up that will hopefully resolve this. Note that the indirection through MemBios is a bit of a bummer anyway - there's some in progress work to hook OpenSSL directly up to the underlying file descriptor of the TcpStream that's being wrapped to avoid some copies and silly retry logic.

I'll leave this open until you have a chance to see if this is actually fixed.

[Byron]

Thank you so much - I just verified it works for me.
See the screenshots to check the memory usage and CPU usage ...

... as well as the achieved bandwidth, which is limited to just 100Mb upstream:

This solves the issue for me, and I am looking forward to the additional improvements you are going to make (especially those who are allowing zero-copy).

[sfackler]

Great! Published v0.6.2 to crates.io with this fix.