Increasing network packet size

[Kubuxu]

Small packets are wasteful, Eth+IP+TCP is 66 bytes so everything smaller than that has more than 50% transfer overhead.

This was greatly improved by recent changes (we used send length in separate packet on few levels, which caused the number of packets to grow geometrically).

I would like to use this issue to show improvements that were made and also to set higher and higher aim.

Version 0.4.3-rc1:
Mean 110, Median 44

[dignifiedquire]

This was greatly improved by recent changes

could you reference those here?

[Kubuxu]

It was mostly: libp2p/go-libp2p-secio@fa92b06

I think there was something more but it is deep in dependency tree. @whyrusleeping might remember more.

[whyrusleeping]

These multistream changes were pretty significant: multiformats/go-multistream@d78b705

As well as the changes in bitswap and the dht that allow us to reuse streams to peers for multiple messages (avoiding the extra stream multiplexer framing for each message)

[Kubuxu]

0.4.4-dev as for August 30:
Median 48 bytes, Mean 112 bytes

One can see that 4 bytes packets were removed but there are a lot of 48 byte packets.

Can we try tracking it down? Secio is 36 bytes, is something bellow secio wrapping the data?
Or is it just higher level protocol? 12 bytes of effective data per packet with 102 bytes of overhead (66 TCP/IP/Eth + 36 secio) is still quite bad.

[whyrusleeping]

48 byte packets are dht provider rpcs

[Kubuxu]

Are DHT RPC calls just 12 bytes or did I do my math wrong about secio (SHA2 32bytes, length 4 bytes)?

[Kubuxu]

I think I found it: the 12 bytes frames we see are yamux frames: https://github.com/hashicorp/yamux/blob/master/spec.md . They are used for creation destruction of streams.

We already worked on decreasing their overhead by reusing DHT RPC streams (which are currently probably the main source of new yamux streams), but this change requires both sides to to support it, it was introduced in 0.4.3-rc1 and after official 0.4.3 release we might be seeing reduced number of 12 bytes frames.

[jbenet]

This would be useful to do with a tool that can inspect what the data is
directly. Muistream helps there to identify-- just need to either doable
secio or look into it somehow.

We could have a mode of --disable-transport-encryption that keeps secio
framing but doesn't do the cipher so we can see the multistream headers and
accurate sizes
On Wed, Aug 31, 2016 at 5:54 AM Jakub Sztandera [email protected]
wrote:

Are DHT RPC calls just 12 bytes or did I do my math wrong about secio
(SHA2 32bytes, length 4 bytes)?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#3155 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAIcobUOYNYkD5vWnMZz8qCRiZi6P3yEks5qlJH0gaJpZM4Jw3_q
.

[jbenet]

I still think a writer wrapping the manet.Conn that delays writes for up to
1ms (configurable and could be removed for latency sensitive users) will
probably be very useful. I'll experiment.
On Mon, Sep 19, 2016 at 8:54 AM Juan Benet [email protected] wrote:

This would be useful to do with a tool that can inspect what the data is
directly. Muistream helps there to identify-- just need to either doable
secio or look into it somehow.

We could have a mode of --disable-transport-encryption that keeps secio
framing but doesn't do the cipher so we can see the multistream headers and
accurate sizes
On Wed, Aug 31, 2016 at 5:54 AM Jakub Sztandera [email protected]
wrote:

Are DHT RPC calls just 12 bytes or did I do my math wrong about secio
(SHA2 32bytes, length 4 bytes)?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#3155 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAIcobUOYNYkD5vWnMZz8qCRiZi6P3yEks5qlJH0gaJpZM4Jw3_q
.

[Kubuxu]

I was investigating dumping not encrypted frames to the pcap format. So we can use wireshark and other tools to investigate them.

Using --disable-transport-encryption won't work that great as then we won't be connecting to the rest of the network.