Add tokio-buf and a `BufStream` trait #611

carllerche · 2018-09-01T18:43:47Z

This is a work in progress.

The description will be updated over time.

TODO

SizeHint guarantees
Should consume_hint take a u64 or usize?
BufStream::limit() tests

carllerche · 2018-10-04T20:54:47Z

@seanmonstar I have updated size_hint documentation to say that the hint values must be accurate. This should work for Hyper to use the hint for the content-length value?

carllerche · 2018-10-04T20:55:46Z

The problem, however, is now size_hint is "useless" for gzip stream transformations. When compressing a stream, there is no guarantee as to the resulting size. The stream can even get larger, which means that an upper bound cannot be set.

carllerche · 2018-10-04T20:56:32Z

Perhaps, size_hint should have an approximate field that is between upper / lower that is used to reserve memory when collecting the stream?

seanmonstar · 2018-10-06T00:04:44Z

tokio-buf/src/buf_stream/mod.rs

+    ///
+    /// Once a stream is finished, i.e. `Ready(None)` has been returned, further
+    /// calls to `poll` may result in a panic or other "bad behavior".
+    fn poll(&mut self) -> Poll<Option<Self::Item>, Self::Error>;


Should this be poll_next like the upcoming change to Stream? Or perhaps a different name, since a type could conceivably implement both Stream and BufStream, and if both traits are in scope, so there is no conflict?

seanmonstar · 2018-10-06T00:05:03Z

tokio-buf/src/buf_stream/mod.rs

+    /// calls to `poll` may result in a panic or other "bad behavior".
+    fn poll(&mut self) -> Poll<Option<Self::Item>, Self::Error>;
+
+    /// Returns the bounds on the remaining length of the iterator.


Calls this an "iterator"

seanmonstar · 2018-10-06T00:15:54Z

tokio-buf/src/buf_stream/size_hint.rs

+pub struct SizeHint {
+    available: usize,
+    lower: usize,
+    upper: Option<usize>,


For use in hyper to set the content-length, this being a usize does mean that 32bit platforms can't set the content-length automatically if sending over 4GBs. Also, things like implementing BufStream for tokio::fs::File would cause problems there as well, since Metadata::len() returns a u64.

Changing size_hint to u64 has the interesting property of making collect() fallible.

carllerche · 2018-10-07T05:34:33Z

@seanmonstar I switched SizeHint to have u64 fields. Doing this exposed a flaw in collect() where large streams could result in overflow situations.

I resolved this by adding an error associated type to FromBufStream. See here.

This ended up complicating the FromBufStream implementation for Vec<u8> as I added handling for overflow. See here.

About half of those changes are handling potential overflow and the other half is factoring SizeHint::upper().

carllerche · 2018-10-07T05:45:16Z

Then the next question... should consume_hint be u64 or usize?

carllerche · 2018-10-07T05:59:54Z

I added a limit combinator here.

Now, to collect a body stream, you can do:

let body = body_stream
    .limit(mb_100)
    .collect();

jplatte

s/sneding/sending

carllerche · 2018-10-09T15:49:24Z

I think I'm going to get rid of SizeHint::available for now. It can be added later and it is not obviously needed.

seanmonstar · 2018-10-09T19:04:37Z

tokio-buf/src/buf_stream/mod.rs

+    /// if it could produce data at that point. If it chooses to return
+    /// `NotReady`, when `consume_hint` is called with a non-zero argument, the
+    /// task must be notified in order to respect the `poll_buf` contract.
+    fn consume_hint(&mut self, amount: usize) {


A possible alternative to this function is passing some hint as an argument to poll_buf. Like, stream.poll_buf(1024 * 16) or something...

The idea here (with h2) is this function would be called separately from reading. When a window update frame is received, it would call consume_hint w/ the total window size.

An argument to poll_buf would also work... but then it would be required to pass something. With a separate function consume_hint, calling it isn't required.

There are pros / cons to both, what are you thinking?

Another con is that middleware must remember to implement consume_hint as there is a default implementation.

I'm actually wondering if the default implementation should be omitted. Same with size_hint... I would expect it to be rarer to implement vs. use.

Is there an example of this providing value somewhere? I get the intention, but I'd personally prefer this left off until there's an implementation showing how it can be helpful.

seanmonstar · 2018-10-23T23:04:45Z

Notably, removing hyper's optimization for Payload::__hyper_full_data results in (on my machine) ~9% reduction in this pipeline benchmark. This is a private-ish method that allows skipping checking the size-hint and then polling and then checking is_end_stream, opting for just a single method call, when the Body already has the full data available.

carllerche · 2018-10-24T00:14:32Z

@seanmonstar what is the additional cost from?

carllerche · 2018-10-24T00:18:35Z

@seanmonstar it seems highly suspect that using BufStream would be slower given that calling size_hint on Once should be extremely fast and you can use the size_hint to only poll once and skip is_end_stream.

seanmonstar · 2018-10-24T19:32:09Z

So, some of the slow down I measured was that internally a optimized fast path was not being taken. Having refactored to basically this:

match body.poll_data()? {
    Async::Ready(Some(data)) => {
        if body.is_end_stream() {
            self.write_full_msg(head, data);
        }
    }
}

This has improved it somewhat, but there is still ~5% slowdown compared to when using body.__hyper__full_body. I suspect it's because when using __hyper_full_body, there's no branch to deal with an Err or NotReady, and no second check for is_end_stream.

carllerche · 2018-10-24T20:16:33Z

@seanmonstar that is pretty surprising and would be worth digging into.

carllerche · 2018-10-25T04:23:38Z

@seanmonstar I dug in a bit, and it looks like the comparison isn't quite fair. The "slow path" could probably be optimized quite a bit more by avoiding much work in poll_data when the body type is Once.

I did a bunch of experimentation myself, and I saw a 5ns difference between using poll_buf and size_hint as described compared with a similar optimization that you use. This should translate to, at most, an 80ns delta in your pipelined benchmark.

seanmonstar

Approving with the caveat that consume_hint should be looked at before publishing 0.1 to crates.io.

Import BufStream from tower-web

c9d9223

carllerche mentioned this pull request Sep 6, 2018

Consider adding Buf::into_bytes_mut() tokio-rs/bytes#228

Open

carllerche mentioned this pull request Sep 14, 2018

Could BufStream be flattened? carllerche/tower-web#111

Closed

srijs mentioned this pull request Sep 25, 2018

Feat/streaming body async read rusoto/rusoto#1153

Merged

carllerche added 3 commits October 4, 2018 12:22

Merge branch 'master' into tokio-buf

a94ff7c

Use released version of bytes

8bfa800

Update size_hint docs

d59b7d0

carllerche added 2 commits October 4, 2018 14:27

Make FromBufStream generic over T: Buf

9e7a735

Add some tests

3bc08b1

seanmonstar reviewed Oct 6, 2018

View reviewed changes

carllerche added 4 commits October 6, 2018 10:32

Rename BufStream::poll -> BufStream::poll_buf

2e37c04

Fix tests

382d163

stream, not iterator

15f24d1

Switch SizeHint fields to u64.

1468776

Add BufStream::limit combinator.

5b82b03

Add consume_hint documentation

a3b679e

jplatte reviewed Oct 8, 2018

View reviewed changes

Typo

ec5a6d7

carllerche added 2 commits October 9, 2018 09:59

A bunch of polish work

26cbfd2

More work

528353b

seanmonstar reviewed Oct 9, 2018

View reviewed changes

Merge branch 'master' into tokio-buf

11c1640

seanmonstar approved these changes Oct 29, 2018

View reviewed changes

carllerche merged commit 51e36e4 into master Oct 29, 2018

carllerche deleted the tokio-buf branch November 20, 2018 06:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add tokio-buf and a `BufStream` trait #611

Add tokio-buf and a `BufStream` trait #611

carllerche commented Sep 1, 2018 •

edited

Loading

carllerche commented Oct 4, 2018

carllerche commented Oct 4, 2018

carllerche commented Oct 4, 2018

seanmonstar Oct 6, 2018

seanmonstar Oct 6, 2018

seanmonstar Oct 6, 2018

carllerche Oct 6, 2018

carllerche commented Oct 7, 2018

carllerche commented Oct 7, 2018

carllerche commented Oct 7, 2018

jplatte left a comment

carllerche commented Oct 9, 2018

seanmonstar Oct 9, 2018

carllerche Oct 9, 2018

carllerche Oct 9, 2018

seanmonstar Oct 29, 2018

seanmonstar commented Oct 23, 2018

carllerche commented Oct 24, 2018

carllerche commented Oct 24, 2018

seanmonstar commented Oct 24, 2018

carllerche commented Oct 24, 2018

carllerche commented Oct 25, 2018

seanmonstar left a comment

Add tokio-buf and a BufStream trait #611

Add tokio-buf and a BufStream trait #611

Conversation

carllerche commented Sep 1, 2018 • edited Loading

TODO

carllerche commented Oct 4, 2018

carllerche commented Oct 4, 2018

carllerche commented Oct 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carllerche commented Oct 7, 2018

carllerche commented Oct 7, 2018

carllerche commented Oct 7, 2018

jplatte left a comment

Choose a reason for hiding this comment

carllerche commented Oct 9, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seanmonstar commented Oct 23, 2018

carllerche commented Oct 24, 2018

carllerche commented Oct 24, 2018

seanmonstar commented Oct 24, 2018

carllerche commented Oct 24, 2018

carllerche commented Oct 25, 2018

seanmonstar left a comment

Choose a reason for hiding this comment

Add tokio-buf and a `BufStream` trait #611

Add tokio-buf and a `BufStream` trait #611

carllerche commented Sep 1, 2018 •

edited

Loading