-
-
Notifications
You must be signed in to change notification settings - Fork 749
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Performance issues due to non-blocking IO #208
Comments
Here's what help I can give (I know 0 zero zip nil of libopenssl):
Again, I do not understand the internals of openssl, but I'd imagine that it has support for flushing its buffer multiple times in a request. If so, it'd be possible to put a cap on the ssl buffer size, and flush each time the cap is reached (similar to In the meantime, this could probably be forced by calling |
Hi @seanmonstar , thanks for chiming in. I agree that calling flush after each write would certainly solve my problem, yet it would have to be something hyper would have to do, which in turn seems like an evil hack. Here is some more information I have gathered previously: Something that caught my attention is the implementation of So far I couldn't find any hunk of code who would explicitly enable non-blocking IO, neither in hyper nor here. From what I see, Hopefully @sfackler will find some time, as I hope we can get this fixed before 1.0 is released. |
I think this boils down to the lack of a call to |
Great to hear you have pinpointed the problem's source ! I will be sitting tight and try it out once the issue is closed. Hopefully this will also fix the 'CPU is the bottleneck' issue I observed previously. |
I just pushed a commit up that will hopefully resolve this. Note that the indirection through MemBios is a bit of a bummer anyway - there's some in progress work to hook OpenSSL directly up to the underlying file descriptor of the TcpStream that's being wrapped to avoid some copies and silly retry logic. I'll leave this open until you have a chance to see if this is actually fixed. |
Great! Published v0.6.2 to crates.io with this fix. |
The subject is the result of some debugging done while trying to trace down the source of my problems. These shall be explained in detail as I am sure the authors of this project and possibly hyper will know a workaround or fix.
When trying to upload a multi-gigabyte video to youtube, I notice that the transfer doesn't start until the entire input stream was read into memory. The call-trace looks like this:
Note the io::copy call coming from the hyper client, which I believe would block in case of a standard TCP connection.
A process trying to send a 1.1GB video (after loading it into memory) looks like this - note the 1.2GB of real memory:
A second observation is the high CPU usage. Even if the entire input is cached, I would have expected the send operation to saturate the 100Mb connection to the internet, yet the CPU seems to be bottlenecking, producing a mere ~2.5Mb per second.
When looking at the tcp send calls of the process, they will come in at about 4 to 5 calls per second, which seems to confirm that the CPU is not producing enough encrypted bytes.
Finally, this call-trace shows that the actual TCP send operation commences only when hyper attempts to end the operation.
The CPU bottleneck seems to be originating within OpenSSL, but that's just a guess of mine.
How to reproduce
Download the
youtube3
executable from the Google APIs for Rust release page, then extract and run:Meta
openssl 0.6.1 (registry+https://github.com/rust-lang/crates.io-index)
CC @seanmonstar @reem
Thanks for your help ! This is a major issue for me and I can help if you give me some hints on how to solve this.
The text was updated successfully, but these errors were encountered: