Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increasing network packet size #3155

Open
Kubuxu opened this issue Aug 30, 2016 · 10 comments
Open

Increasing network packet size #3155

Kubuxu opened this issue Aug 30, 2016 · 10 comments
Labels
kind/enhancement A net-new feature or improvement to an existing feature status/deferred Conscious decision to pause or backlog

Comments

@Kubuxu
Copy link
Member

Kubuxu commented Aug 30, 2016

Small packets are wasteful, Eth+IP+TCP is 66 bytes so everything smaller than that has more than 50% transfer overhead.

This was greatly improved by recent changes (we used send length in separate packet on few levels, which caused the number of packets to grow geometrically).

I would like to use this issue to show improvements that were made and also to set higher and higher aim.

Version 0.4.3-rc1:
Mean 110, Median 44
sizes_full
sizes_200

@Kubuxu Kubuxu added the kind/enhancement A net-new feature or improvement to an existing feature label Aug 30, 2016
@dignifiedquire
Copy link
Member

This was greatly improved by recent changes

could you reference those here?

@Kubuxu
Copy link
Member Author

Kubuxu commented Aug 30, 2016

It was mostly: libp2p/go-libp2p-secio@fa92b06

I think there was something more but it is deep in dependency tree. @whyrusleeping might remember more.

@whyrusleeping
Copy link
Member

These multistream changes were pretty significant: multiformats/go-multistream@d78b705

As well as the changes in bitswap and the dht that allow us to reuse streams to peers for multiple messages (avoiding the extra stream multiplexer framing for each message)

@Kubuxu
Copy link
Member Author

Kubuxu commented Aug 30, 2016

0.4.4-dev as for August 30:
Median 48 bytes, Mean 112 bytes

sizes-all
sizes-sub200

One can see that 4 bytes packets were removed but there are a lot of 48 byte packets.

Can we try tracking it down? Secio is 36 bytes, is something bellow secio wrapping the data?
Or is it just higher level protocol? 12 bytes of effective data per packet with 102 bytes of overhead (66 TCP/IP/Eth + 36 secio) is still quite bad.

@whyrusleeping
Copy link
Member

48 byte packets are dht provider rpcs

@Kubuxu
Copy link
Member Author

Kubuxu commented Aug 30, 2016

Are DHT RPC calls just 12 bytes or did I do my math wrong about secio (SHA2 32bytes, length 4 bytes)?

@Kubuxu
Copy link
Member Author

Kubuxu commented Aug 30, 2016

I think I found it: the 12 bytes frames we see are yamux frames: https://github.com/hashicorp/yamux/blob/master/spec.md . They are used for creation destruction of streams.

We already worked on decreasing their overhead by reusing DHT RPC streams (which are currently probably the main source of new yamux streams), but this change requires both sides to to support it, it was introduced in 0.4.3-rc1 and after official 0.4.3 release we might be seeing reduced number of 12 bytes frames.

@whyrusleeping whyrusleeping added the status/deferred Conscious decision to pause or backlog label Sep 14, 2016
@jbenet
Copy link
Member

jbenet commented Sep 19, 2016

This would be useful to do with a tool that can inspect what the data is
directly. Muistream helps there to identify-- just need to either doable
secio or look into it somehow.

We could have a mode of --disable-transport-encryption that keeps secio
framing but doesn't do the cipher so we can see the multistream headers and
accurate sizes
On Wed, Aug 31, 2016 at 5:54 AM Jakub Sztandera [email protected]
wrote:

Are DHT RPC calls just 12 bytes or did I do my math wrong about secio
(SHA2 32bytes, length 4 bytes)?


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#3155 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAIcobUOYNYkD5vWnMZz8qCRiZi6P3yEks5qlJH0gaJpZM4Jw3_q
.

@jbenet
Copy link
Member

jbenet commented Sep 19, 2016

I still think a writer wrapping the manet.Conn that delays writes for up to
1ms (configurable and could be removed for latency sensitive users) will
probably be very useful. I'll experiment.
On Mon, Sep 19, 2016 at 8:54 AM Juan Benet [email protected] wrote:

This would be useful to do with a tool that can inspect what the data is
directly. Muistream helps there to identify-- just need to either doable
secio or look into it somehow.

We could have a mode of --disable-transport-encryption that keeps secio
framing but doesn't do the cipher so we can see the multistream headers and
accurate sizes
On Wed, Aug 31, 2016 at 5:54 AM Jakub Sztandera [email protected]
wrote:

Are DHT RPC calls just 12 bytes or did I do my math wrong about secio
(SHA2 32bytes, length 4 bytes)?


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#3155 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAIcobUOYNYkD5vWnMZz8qCRiZi6P3yEks5qlJH0gaJpZM4Jw3_q
.

@Kubuxu
Copy link
Member Author

Kubuxu commented Sep 19, 2016

I was investigating dumping not encrypted frames to the pcap format. So we can use wireshark and other tools to investigate them.

Using --disable-transport-encryption won't work that great as then we won't be connecting to the rest of the network.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement A net-new feature or improvement to an existing feature status/deferred Conscious decision to pause or backlog
Projects
None yet
Development

No branches or pull requests

4 participants