Streaming support

[xibz]

I started spending some time thinking about this and wanted to track this. Have you guys thought of a design on how you guys want this to work? I was thinking of looking into GRPC and taking it and molding it into something that is memory efficient for TTRPC. Was there anything I should look out for?

I also noticed there isn't any use of a transport which seems necessary for stream handling. What was the plan to handle multiplexing on a given stream? net.Conn is used to handle reading and writing data. In GRPC eventually the connection is used to create a transport. However, I do not see any logic surrounding transport in the package. Was this intentionally left out or was there plans to add it later?

Is there any other feature that needs to be implemented to support this?

[stevvooe]

So, we do have a transport that manages streams. The channeling protocol is defined in https://github.com/containerd/ttrpc/blob/master/channel.go#L44. It is roughly equivalent to what http/2 provides, minus the required server-side state management. The main complexity is simply managing the active streams. It shouldn't be too hard to add streaming support on top of that. I'd recommend studying how GRPC maps streaming requests to streams and work from there. In general, I would avoid using their code, as it is extremely inefficient and complex.

The protocol for GRPC is defined in https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md. I suspect that you might need to add some flags for ending streams to make it work but it may be as easy as just continuing to use the same stream id for the message passing on a streaming call.

[rcash]

Hi @stevvooe and @xibz I'm a student in UT Austin's Virtualization class, are you still looking for people to contribute to this? If so, could I get some more info about what would be needed to add streaming support?

[dmcgowan]

Merged in #107 , see new protocol doc for updates to protocol https://github.com/containerd/ttrpc/blob/main/PROTOCOL.md