[GossipSub 1.2] IDONTWANT control message #548

Nashatyrev · 2023-05-30T06:51:06Z

This PR introduces a new GossipSub version 1.2

Co-authored with @Menduist

New messages

The new IDONTWANT control message is added which notifies mesh peers to suspend sending back the full publish message based on its messageId

Simulation results

Various simulations demonstrated ~30% traffic reduction and ~3-5% message delivery latency reduction

@Menduist results involving the full TCP stack(the feature is named choke there) : https://hackmd.io/X1DoBHtYTtuGqYg0qK4zJw#4---BBR
@Nashatyrev results (short cutting TCP stack): https://docs.google.com/spreadsheets/d/15tbZHBbWZmsDIY_xm9swNe9A2pcQ-gp4yE42iGDCJB4/edit#gid=1815594575

Related work

Relates to #413

vyzo · 2023-05-30T07:42:34Z

That seems very reasonable; I like IDONTWANT more for the name.

@AgeManning any thoughts?

vyzo · 2023-05-30T07:51:41Z

Actually, this might make some sense to implement with IHAVE, assuming we keep track of peer IHAVEs and avoid sending those messages to the peer.

However, this will conflate with heartbeat generated IHAVEs.
So overall, i think it is better to implement with an IDONTWANT message.

We can also effect some positive scoring on this action (if it pertains to messages we have seen or see shortly after)

Another consideration: if we sending the message to the peer already, would that be useful?
It is kind of implicit.

PS:
@Nashatyrev do you want to join our episub group in telegram?

Menduist · 2023-05-30T08:37:17Z

pubsub/gossipsub/gossipsub-v1.2.md

+
+| Parameter               | Description                                                      | Reasonable Default |
+|-------------------------|------------------------------------------------------------------|--------------|
+| `max_dontsend_messages` | The maximum number of `DONTSEND` messages per heartbeat per peer | ???  |


Is that necessary? We can just descore peer if they send duplicate DONTSENDs, or we don't eventually see the message id
GossipSub already has too many parameters

or we don't eventually see the message id

Sounds like a good idea 👍

or we don't eventually see the message id

Considering DONTSEND is allowed to be sent before validation, a peer can be downscored if a message DONTSEND has been sent for appears to be invalid

Do we have numbers on how much we can gain by allowing to send DONTSENDs before validation? (ie, how long the validation is in practice)

I think we can avoid the downscoring by keeping a bounded cache, that gets overfilled to /dev/null.

We probably dont need a parameter for this, each peer can configure appropriately according to the expected message rate.

If we do want to downscore excessive rates of IDONTWANT, then we should validate first or else we open the door for a spam attack.

Note that validation in some networks can be slow, so there is real benefit by sending early.

pubsub/gossipsub/gossipsub-v1.2.md

vyzo · 2023-05-30T08:44:11Z

Thinking a bit more about this, I am not entirely convinved we need any of this.

We can simply start keep tracking messages we have received from a mesh peer recently; if they have send us a message, we should supress it. Otherwise we just send the message and the IDONTWANT is implicit/redundant.

Menduist · 2023-05-30T08:48:23Z

@vyzo the DONTSEND becomes useful when the messages get big enough that it actually takes time to transmit them

ie, a 0.5mb message over a 25mbps line will take 160ms to be sent to 8 peers, so sending a small DONTSEND before sending the full message gives us 160 ms more to avoid duplicates
Also, when more sophisticated relaying technique are used (for instance, sending to one peer after the other), this becomes even more useful

vyzo · 2023-05-30T08:50:33Z

Ok, fair enough.
We should still do the supress thing however, for interacting with older peers.

Nashatyrev · 2023-05-30T09:08:31Z

Actually, this might make some sense to implement with IHAVE, assuming we keep track of peer IHAVEs and avoid sending those messages to the peer.

However, this will conflate with heartbeat generated IHAVEs. So overall, i think it is better to implement with an IDONTWANT message.

Yes, we discussed that option, and came to agreement that those messages have slightly different semantics which might make the difference under some conditions. For example a peer may send IDONTWANT prior to message validation which would be different from IHAVE as the message may appear to be invalid finally.

I like IDONTWANT more for the name.

Yes I'm voting for IDONTWANT too 👍

Another consideration: if we sending the message to the peer already, would that be useful?

Yes, the basic strategy would be to broadcast corresponding IDONTWANT prior to broadcasting the message

Menduist · 2023-05-30T13:42:25Z

pubsub/gossipsub/gossipsub-v1.2.md

+}
+
+message ControlDontSend {
+    required bytes messageID = 1;


Just for discussion, what would be the pros/cons of also including the topic?
I think it's the first time that we reference messages only by their id on the wire, so needs some consideration

I guess the only con of include the topic is more bandwidth usage

maybe an optional field?
Agreed about bandwidth usage, we should aim to keep this lean.

IHAVE also has an optional topic field. But it looks like it is utilized by one implementation only (not sure which one exactly). I couldn't find any reasonable usage of topic for IDONTWANT tbh.

If no one is opposed or has any ideas on usage scenarios I would keep it without optional topic field to be more explicit

If no one is opposed or has any ideas on usage scenarios I would keep it without optional topic field to be more explicit

The usage can be #548 (comment)

pubsub/gossipsub/gossipsub-v1.2.md

vyzo · 2023-05-30T13:53:54Z

@Menduist @mkalinin note the existence of the episub working group in telegram; our little Lair is open, reach out to me in Telegram if you want to join.

pubsub/gossipsub/gossipsub-v1.2.md

arnetheduck · 2023-06-01T09:06:36Z

Ok, fair enough.

for context, here are two scenarios we're addressing with this upgrade, for large messages in particular:

ppopth · 2023-06-26T17:37:59Z

@vyzo Can I also join the episub group? FYI, I am the network person of the EF research and have been working with @Nashatyrev, @Menduist, and others.

vyzo · 2023-06-26T18:39:16Z

Of course! Reach out on telegram to add you.

vyzo · 2023-07-14T18:37:08Z

Let's target this to #560

# Conflicts: # pubsub/gossipsub/gossipsub-v1.2.md

djrtwo · 2023-12-14T14:25:43Z

Looks good to me!

vyzo

Just a small comment, otherwise lgtm.

vyzo · 2023-12-19T17:48:56Z

pubsub/gossipsub/gossipsub-v1.2.md

+
+When the peer receives the first message instance it immediately broadcasts 
+(not queue for later piggybacking) `IDONTWANT` with the `messageId` to all its mesh peers. 
+This could be performed prior to the message validation to further increase the effectiveness of the approach.    


Concerns about spam attacks triggering amplified IDONTWANT spam?

This doesn't look like a feasible attack vector to me:

IDONTWANT is primarily intended for larger messages, so the cumulative size of resulting IDONTWANT messages is expected to be significantly smaller than the original message

If an attacker is sending invalid messages to initiate IDONTWANT spamming it would be pretty quickly banned due to negative scoring

And we have max_idontwant_messages limit as the last resort

pubsub/gossipsub/gossipsub-v1.2.md

…VE and IWANT messages Co-authored-by: Pop Chunhapanya <[email protected]>

AgeManning · 2024-02-27T04:55:53Z

pubsub/gossipsub/gossipsub-v1.2.md

+
+| Parameter                | Description                                                      | Reasonable Default |
+|--------------------------|------------------------------------------------------------------|--------------|
+| `max_idontwant_messages` | The maximum number of `IDONTWANT` messages per heartbeat per peer | ???  |


I guess this probably needs to be per-topic per heartbeat, rather than a total per heartbeat.

It seems it could be tied in with the scoring for mesh message delivery rate. I.e the more messages we are expecting per topic, the more IDONTWANT messages we would expect to receive.

One thought would be to add a behaviour penalty, similar to broken promises, if the number of IDONTWANT messages received from a peer exceeds the mesh message delivery rate.

We intend to implement this fairly soon. Perhaps we can leave the scoring penalty here for a future PR if we dont want to specify it now.

Tracking IDONTWANT by topics would be perfect, however you don't know a message topic by its ID unless you already received this message.
IDONTWANT message is almost semantically equivalent to IHAVE message, so probably should have a similar anti-spam protection mechanism?
Probably it could make sense to set the 'reasonable default' to maxIHaveLength * maxIHaveMessages ?

however you don't know a message topic by its ID unless you already received this message.

Is it a good reason to include topics to IDONTWANT and not care what IHAVE and IWANT are doing?

Is it a good reason to include topics to IDONTWANT and not care what IHAVE and IWANT are doing?

Including a topic would increase IDONTWANT traffic 2-3 times (messageID is 20 bytes, topic could be up to around 40 bytes). Not sure if it's worth just to enable more advanced spam protection...

Please also consider that IHAVE is sent on heartbeat with a batch of messages which may be grouped by topics and the topic overhead here could not be that significant.
IDONTWANT on the other hand is intended to be flushed immediately and would most likely contain just a single messageID so the topic overhead would be more significant

I suspect it might be valuable to specify it - ie effectively, after mcache expires, the message should no longer exist and implementations should be able to rely on them being "resent" if they resurface after that time - this more faithfully keeps the protocol consistent in this aspect

I don't see why using a weak hash function for unwanted is a problem.

@ppopth I was meaning this HashDoS attack.
The messageIDs in IDONTWANT are not validated: neither upon receive nor later (as for IHAVE). Thus an adversary may generate and send significant amount of messageIDs which yields the same hashCode in the context of unwanted hash set. That would result in a DoS on the receiver's side.
Actually this attack vector could be addressed on the implementation side pretty easily. I just wanted to add a 'warning note' for implementers into the spec

@Nashatyrev Got it. Thank you so much.

not sure it's been mentioned, but because the message id:s are not validated

@arnetheduck Good point!
I believe as far as the messageId function is application specific, the validation of a messageId is left for an application responsibility. I believe Ethereum clients should all have such validation

I believe Ethereum clients should all have such validation

I'm not convinced this is the case, ie at least in nimbus, we don't validate message id:s (only messages). There's a custom message id generation feature, but this again does not validate message id:s themselves.

This PR represents the first time we receive message id:s that we're expected to store / keep track of - all others are either generated from actual messages or ephemeral.

pubsub/gossipsub/gossipsub-v1.2.md

Co-authored-by: João Oliveira <[email protected]>

AgeManning · 2024-04-15T01:13:15Z

Hey all.

We are very interested in testing this. What are our thoughts on merging this. We can do a future PR to handle scoring, or leave it up to implementors to handle their own dos prevention strategies.

We want to start testing on live networks using the 1.2 protocol id. My concern is that if we do this, without this PR being merged and someone decides to change the protobuf, we will have then polluted the protocol-id space with an incompatible version.

We will go with a -beta in the meantime, however it would be good to merge down to get some consensus.

arnetheduck · 2024-05-01T06:22:14Z

fwiw, nimbus/nim-libp2p has implemented this as an extension to 1.1: https://github.com/vacp2p/nim-libp2p/blob/2b5319622c997ce1c80bc62c863e30f3349ee0d7/libp2p/protocols/pubsub/gossipsub/behavior.nim#L266 - it would indeed be nice to get this merged and properly give it a version number

vyzo

Soooo, lets merge it?
Any objections?

arnetheduck

lgtm, unless someone wants to edit in the various security recommendations

## GossipSub v1.2 implementation Specification: libp2p/specs#548 ### Work Summary Sending IDONTWANT Implement a smart queue Add priorities to the smart queue Put IDONTWANT packets into the smart priority queue as soon as the node gets the packets Handling IDONTWANT Use a map to remember the message ids whose IDONTWANT packets have been received Implement max_idontwant_messages (ignore the IDONWANT packets if the max is reached) Clear the message IDs from the cache after 3 heartbeats Hash the message IDs before putting them into the cache. More requested features Add a feature test to not send IDONTWANT if the other side doesnt support it ### Commit Summary * Replace sending channel with the smart rpcQueue Since we want to implement a priority queue later, we need to replace the normal sending channels with the new smart structures first. * Implement UrgentPush in the smart rpcQueue UrgentPush allows you to push an rpc packet to the front of the queue so that it will be popped out fast. * Add IDONTWANT to rpc.proto and trace.proto * Send IDONTWANT right before validation step Most importantly, this commit adds a new method called PreValidation to the interface PubSubRouter, which will be called right before validating the gossipsub message. In GossipSubRouter, PreValidation will send the IDONTWANT controll messages to all the mesh peers of the topics of the received messages. * Test GossipSub IDONWANT sending * Send IDONWANT only for large messages * Handle IDONTWANT control messages When receiving IDONTWANTs, the host should remember the message ids contained in IDONTWANTs using a hash map. When receiving messages with those ids, it shouldn't forward them to the peers who already sent the IDONTWANTs. When the maximum number of IDONTWANTs is reached for any particular peer, the host should ignore any excessive IDONTWANTs from that peer. * Clear expired message IDs from the IDONTWANT cache If the messages IDs received from IDONTWANTs are older than 3 heartbeats, they should be removed from the IDONTWANT cache. * Keep the hashes of IDONTWANT message ids instead Rather than keeping the raw message ids, keep their hashes instead to save memory and protect again memory DoS attacks. * Increase GossipSubMaxIHaveMessages to 1000 * fixup! Clear expired message IDs from the IDONTWANT cache * Not send IDONTWANT if the receiver doesn't support * fixup! Replace sending channel with the smart rpcQueue * Not use pointers in rpcQueue * Simply rcpQueue by using only one mutex * Check ctx error in rpc sending worker Co-authored-by: Steven Allen <[email protected]> * fixup! Simply rcpQueue by using only one mutex * fixup! Keep the hashes of IDONTWANT message ids instead * Use AfterFunc instead implementing our own * Fix misc lint errors * fixup! Fix misc lint errors * Revert "Increase GossipSubMaxIHaveMessages to 1000" This reverts commit 6fabcdd. * Increase GossipSubMaxIDontWantMessages to 1000 * fixup! Handle IDONTWANT control messages * Skip TestGossipsubConnTagMessageDeliveries * Skip FuzzAppendOrMergeRPC * Revert "Skip FuzzAppendOrMergeRPC" This reverts commit f141e13. * fixup! Send IDONWANT only for large messages * fixup! fixup! Keep the hashes of IDONTWANT message ids instead * fixup! Implement UrgentPush in the smart rpcQueue * fixup! Use AfterFunc instead implementing our own --------- Co-authored-by: Steven Allen <[email protected]>

vyzo · 2024-08-16T15:17:40Z

The go implementation in libp2p/go-libp2p-pubsub#553 has merged.

Add draft DONTSEND Gossip control message spec

c75e81a

Nashatyrev marked this pull request as draft May 30, 2023 06:51

Menduist reviewed May 30, 2023

View reviewed changes

pubsub/gossipsub/gossipsub-v1.2.md Outdated Show resolved Hide resolved

Menduist reviewed May 30, 2023

View reviewed changes

pubsub/gossipsub/gossipsub-v1.2.md Show resolved Hide resolved

Menduist reviewed May 30, 2023

View reviewed changes

mkalinin reviewed May 30, 2023

View reviewed changes

pubsub/gossipsub/gossipsub-v1.2.md Outdated Show resolved Hide resolved

mxinden reviewed May 31, 2023

View reviewed changes

pubsub/gossipsub/gossipsub-v1.2.md Outdated Show resolved Hide resolved

pubsub/gossipsub/gossipsub-v1.2.md Outdated Show resolved Hide resolved

Rename DONTSEND to IDONTWANT

4fafe66

Nashatyrev changed the title ~~[GossipSub 1.2] DONTSEND control message~~ [GossipSub 1.2] IDONTWANT control message May 31, 2023

Use the proper SHOULD and MAY wordings

c6cfac9

gossipsub v1.2 scaffolding

9f9314c

vyzo mentioned this pull request Jul 14, 2023

gossipsub v1.2 #560

Draft

Nashatyrev added 2 commits July 17, 2023 12:21

Merge branch 'gossipsub/v1.2' into gossip-dontsend

9ba2b9c

# Conflicts: # pubsub/gossipsub/gossipsub-v1.2.md

Add authors and interest group members

7064203

Nashatyrev changed the base branch from master to gossipsub/v1.2 July 17, 2023 08:22

Menduist mentioned this pull request Jul 24, 2023

GossipSub: IDontWant vacp2p/nim-libp2p#934

Merged

arnetheduck mentioned this pull request Aug 7, 2023

post-devcon 4844 spec updates ethereum/consensus-specs#3043

Closed

Remove potential use cases which may potentially lead to amplification

31706a8

Nashatyrev marked this pull request as ready for review November 30, 2023 04:32

realbigsean mentioned this pull request Dec 14, 2023

Consensus-layer Call 124 ethereum/pm#922

Closed

p-shahi requested a review from vyzo December 19, 2023 17:35

vyzo reviewed Dec 19, 2023

View reviewed changes

diegomrsantos reviewed Dec 28, 2023

View reviewed changes

pubsub/gossipsub/gossipsub-v1.2.md Outdated Show resolved Hide resolved

Change SHOULD to MUST according to discussion in comments

b9202a8

ppopth reviewed Feb 22, 2024

View reviewed changes

pubsub/gossipsub/gossipsub-v1.2.md Outdated Show resolved Hide resolved

ppopth reviewed Feb 22, 2024

View reviewed changes

pubsub/gossipsub/gossipsub-v1.2.md Outdated Show resolved Hide resolved

Make the ControlIDontWant.messageIDs plural to be consistent with IHA…

1fdaeb8

…VE and IWANT messages Co-authored-by: Pop Chunhapanya <[email protected]>

AgeManning reviewed Feb 27, 2024

View reviewed changes

jxs mentioned this pull request Mar 17, 2024

Implement gossipsub IDONTWANT sigp/lighthouse#5422

Merged

jxs reviewed Mar 18, 2024

View reviewed changes

pubsub/gossipsub/gossipsub-v1.2.md Outdated Show resolved Hide resolved

AgeManning mentioned this pull request Mar 20, 2024

Consensus-layer Call 130 ethereum/pm#987

Closed

Change filed name to lower case for consistency with original spec

c7dfbe0

Co-authored-by: João Oliveira <[email protected]>

ppopth mentioned this pull request Apr 29, 2024

GossipSub v1.2: IDONTWANT control message and priority queue. libp2p/go-libp2p-pubsub#553

Merged

8 tasks

AgeManning approved these changes May 15, 2024

View reviewed changes

vyzo approved these changes May 15, 2024

View reviewed changes

vyzo changed the base branch from gossipsub/v1.2 to master May 15, 2024 11:46

arnetheduck approved these changes May 15, 2024

View reviewed changes

vyzo merged commit e8b3234 into libp2p:master May 15, 2024

arnetheduck mentioned this pull request May 24, 2024

Support gossipsub 1.2.0 vacp2p/nim-libp2p#1102

Closed

Wondertan mentioned this pull request Jun 14, 2024

Dynamic Optimal Graph (DOG) gossip protocol cometbft/cometbft#3263

Open

StefanBratanov mentioned this pull request Jul 25, 2024

Implement gossipsub IDONTWANT libp2p/jvm-libp2p#371

Closed

nisdas mentioned this pull request Sep 6, 2024

Update Pubsub Library To Use Gossipsub 1.2 prysmaticlabs/prysm#14428

Merged

Nashatyrev mentioned this pull request Sep 23, 2024

[GossipSub 1.2] Add IDONTWANT support libp2p/jvm-libp2p#374

Merged

[GossipSub 1.2] IDONTWANT control message #548

[GossipSub 1.2] IDONTWANT control message #548

Conversation

Nashatyrev commented May 30, 2023 • edited Loading

New messages

Simulation results

Related work

vyzo commented May 30, 2023

vyzo commented May 30, 2023

Menduist May 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vyzo May 30, 2023 • edited Loading

Choose a reason for hiding this comment

vyzo commented May 30, 2023 • edited Loading

Menduist commented May 30, 2023

vyzo commented May 30, 2023

Nashatyrev commented May 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vyzo commented May 30, 2023

arnetheduck commented Jun 1, 2023

ppopth commented Jun 26, 2023 • edited Loading

vyzo commented Jun 26, 2023

vyzo commented Jul 14, 2023

djrtwo commented Dec 14, 2023

vyzo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Nashatyrev Mar 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Nashatyrev May 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Nashatyrev May 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AgeManning commented Apr 15, 2024

arnetheduck commented May 1, 2024 • edited Loading

vyzo left a comment

Choose a reason for hiding this comment

arnetheduck left a comment

Choose a reason for hiding this comment

vyzo commented Aug 16, 2024

Nashatyrev commented May 30, 2023 •

edited

Loading

Menduist May 30, 2023 •

edited

Loading

vyzo May 30, 2023 •

edited

Loading

vyzo commented May 30, 2023 •

edited

Loading

ppopth commented Jun 26, 2023 •

edited

Loading

Nashatyrev Mar 13, 2024 •

edited

Loading

Nashatyrev May 7, 2024 •

edited

Loading

Nashatyrev May 7, 2024 •

edited

Loading

arnetheduck commented May 1, 2024 •

edited

Loading