Avoid multiple conversions between node streams and pull streams

[mcollina]

This module converts between streams and pull streams several times, increasing memory usage, latency and cpu usage on the hot path of receiving data.

The mplex.listener(stream) method require a pull stream. However, that pull stream is automatically converted into a node core stream in

js-libp2p-mplex/src/index.js

Line 11 in aa977c0

const stream = toStream(rawConn)

.
This is going to lead to adding two layers to abstractions that are not needed on Node.js, as mplex core multiplexer is implemented as a Node.js duplex (

js-libp2p-mplex/src/internals/index.js

Line 28 in aa977c0

class Multiplex extends stream.Duplex {

).

My recommendation is to move the internal multiplexer to a pull-stream only implementation, because that will also remove the conversion needed when emitting a new stream (

js-libp2p-mplex/src/muxer.js

Line 47 in aa977c0

catchError(toPull.duplex(stream)),

). In turn, this create even more overhead.

Overall, this module introduces 5/6 levels of data buffering, where 1 or 2 could suffice (each layer introduce latency. As a general rule, every time you pipe two streams it adds some data buffering and latency.

---

When doing this port to pull-stream, I recommend to remove one of the main buffering layer. Currently the module creates a Duplex stream which is piped back and fort (

js-libp2p-mplex/src/index.js

Line 21 in aa977c0

pump(stream, mpx, stream)

). Note that this duplex is only internal, and it is not exposed, so it's totally unnecessary. It's possible to use the native node core or pull stream methods to read and write to it, saving a buffering layer.

---

Another element of latency is introduced by the usage of duplexify, which adds one unneeded level of buffering. Duplexify is used to assemble shared streams in

js-libp2p-mplex/src/internals/index.js

Line 138 in aa977c0

return duplexify(this.createStream(name, Object.assign(opts, {lazy: true})), this.receiveStream(name, opts))

. Even if it's convenient, creating dupexes in this way is inefficient, and they should but created as duplex from the start.

[daviddias]

All shall be gone with #94

[vasco-santos]

It's done 🎉