PubSub & GossipSub configuration proposal

[aschmahmann]

Context

PubSub spec

The PubSub spec is mostly limited to the wire protocol which is pretty barebones and as a result is open to multiple implementations.

The only "configuration" mentioned in the spec is regarding the Topic_Descriptor and the potential for enabling authentication and encryption on the channel. However, the Topic_Descriptor protobuf is never actually sent over the wire in the reference Go implementation and authentication and encryption are both left not implemented. As a result it is not clear if or how this type of configuration is really intended to be used.

go-libp2p-pubsub

While the PubSub spec doesn't have much specified in the way of configurations, the go-libp2p-pubsub implementation does in fact enable a few configurations. These configurations apply to all channels of a given PubSub instance and include:

• Blacklisting
• Message signing
• Strict signature validation
• Message routing strategy

Of these configuration options the only one deemed important enough to effect the protocol/contract between nodes is the routing strategy (i.e. "floodsub", "meshsub", etc.). While it makes sense that blacklisting nodes does not need to be communicated at the protocol level other configurations might be relevant (e.g. message signature/verification, persistence, encryption, etc.)

Additionally the common router implementations, such as FloodSubRouter and GossipSubRouter have a grand total of zero configurable parameters. This means that a slightly modified version of GossipSubRouter essentially involves copy-pasting the source into a new router.

Description of Change

1. I really think that at the very least the GossipSubRouter should be extensible enough to allow developers to utilize the underlying protocols for managing the mesh while implementing their own logic for how, when, and which messages get sent along the mesh.

   While it may seem like this logic could be layer on top of GossipSubRouter in fact gossipsub's insistence on (unreliably) fowarding all application messages to peers in its mesh exactly once is too cumbersome for some applications.

2. We should start working on how we'd like to enable protocol extensions to PubSub and GossipSub (and actually implement one)

3. Allow for subscribing to a PubSub topic with one or more configuration options

Motivation

The motivations for each of the changes above are described in the same order below:

1. There have been requests for a persistent PubSub option for a while. For example, IPNS-over-PubSub is not any faster than IPNS-over-DHT for initial content resolution because PubSub has no notion of persistence. While I could copy-paste GossipSubRouter into LastWriterWinsPersistentGossipSubRouter and make some changes this feels like a huge maintainability problem. This becomes even more obvious when you take into account that if I wanted to create a scheme for synchronizing multiwriter DAGs over PubSub this would result in yet another copy-paste-modify router.

2. I'm not dead set on enabling protocol extensions immediately because there exists an alternative, which is the developer creating a new protocol ID for each combination of protocol + extensions. However, this does lead to the overhead of dealing with a number of protocol IDs that's exponential in the number of options, so it's something we probably should deal with.

3. I think the DX would likely be better passing config options to Subscribe() then creating new PubSub instances for every configuration. Also, the system does not behave well if you Subscribe to the same topic with multiple PubSub instances that support the same protocols (e.g. FloodSubRouter and GossipSubRouter both support the floodsub protocol), so being able to limit the possiblity of this happening would be good. Finally, there's lost efficiency if we have multiple gossipsub-style routers that can't group their messages together because they are managed by different instances.

   Because GossipSubRouter is backwards compatible with FloodSubRouter and message signing/verification are pretty good I doubt people have been needing multiple PubSub instances. However, in a world where there are multiple valid configurations and one isn't "better" than another this is bound to happen.

Proposed Design

Specs: No need to change anything. However, it would be nice to have either of:

1. A bare bones gossipsub spec that focused on the mesh connectivity and left the decisions on when and which messages to forward up to the implementations/higher level protocols. This could make it easier to create new specs that build off the same basic premise just as the abstractions help build reusable code.
2. An extended gossipsub spec that includes a persistence option. The persistence option would have to be triggered at a higher level (e.g. adding a map of router parameters the SubOpts RPC, including them in the protocol ID, etc.). The persistence spec could build off of my existing work at https://github.com/libp2p/go-libp2p-pubsub/blob/3f326121f7228a96827e740dbbd1fc9cc5cc7b62/gossiplww.go

go-libp2p-pubsub: See PR at #171

1. Modify PubSubRouter interface from Join(topic) to Join(topic, protocolID) and appropriately modify the existing go-libp2p-pubsub PubSubRouter implementations.
2. Create a SubOpt allowing users to Subscribe to a topic with a particular protocol ID
3. Modify GossipSubRouter to take in multiple configurations that are associated with particular protocol IDs

Proposed Next Steps

1. Get feedback and approval for configuration PR to allow for more experimentation (potentially leaving out the LWW implementation)
2. Publish LWW persistent gossipsub spec for community review and improvement
3. Get community feedback on whether anyone else is interested in building off of gossipsub and if there's more configuration to be added going forward
4. Get community feedback on whether we'd like more optionality built into gossipsub or just have lots of protocol IDs

[raulk]

Thank you so much for this detailed write-up. To make sure I'm not missing the forest for the trees, I'm going to restate a summary of what you are proposing:

1. Gossipsub implementations should allow users to locally attach configuration parameters to topics. There is no concept of mesh-wide topic configuration, which would require some level of consensus.
2. You want to modularise Gossipsub to make it a dependable and versatile foundational layer to build higher-level behaviours. I think is in tune with the original spirit, although I suspect the swappability is only available for the routing strategy right now. But I'd generally agree to a modular pubsub system.
3. One of such behaviours is what you call LWW (last writer wins), which is similar to the "last image caching" subscription recovery policy of ActiveMQ, where you retain only the newest message so that new subscribers (or ones that missed the message) can fetch it.

BTW – I can easily trace several of the points you propose to features that have been available for decades in enterprise messaging systems like ActiveMQ, RabbitMQ, etc.

At one point I'd like us to do a review of the state-of-the-art in terms of P2P pubsub and parity with enterprise messaging systems, and approach this more holistically.

[aschmahmann]

Right on the money with 2 and 3.

Regarding 1: There is no "gossipsub the protocol" concept for mesh-wide configurations. However, there is now a concept for mesh-wide configurations in "GossipSub the go implementation" -- that concept is to use Protocol IDs (which is the current solution used by FloodSub, RandomSub, and GossipSub).

However, as I mentioned above:

this does lead to the overhead of dealing with a number of protocol IDs that's exponential in the number of options, so it's something we probably should deal with.

[aschmahmann]

After talking with @raulk at DTN it sounds like we may be interested in combining this configuration proposal with a great refactor of go-libp2p-pubsub.

The main issue with go-libp2p-pubsub that requires a refactor independent of the configuration proposal above is that the PubSubRouters all depend on private variables in PubSub which makes it essentially impossible for users to create their own routers without forking this library.

Additionally, @raulk mentioned that he did not agree with #171 (comment) and thought that some of the particular configurations that I am working on (e.g. Last-Writer-Wins Persistence and DAG Synchronization) should be available across routing systems instead of only implementing them for the Gossipsub based router.

@vyzo @raulk please let me know if I've misrepresented any of our discussion and let me know if you have any thoughts on the direction of the refactor. I'm also happy to do a video call to try and hammer out some of the details.

One concern I'd like to point out is that IPNS-over-PubSub's improvements are dependent on LWW persistence being available, at least in Gossipsub. As a result, I would not want to combine the configuration and refactor proposals if we expect the results to take a long time to produce. I think it's likely we could get away with two separate refactors here as long as we don't break the NewFloodSub and NewGossipSub functions because the use of private PubSub variables in routers means nobody else will be affected by these changes anyway.

[raulk]

Synthesising notes of my discussions with @aschmahmann in DTN'19. I'm mostly concerned with how not to special-case this.

• I believe retroactive subscriptions could be a facility available in the pubsub base layer, not just on gossipsub.

• Right now base, the code for gossipsub and floodsub in go-libp2p is kinda tangled. We should separate these in layers with clear public APIs to truly deliver on the promise of extensibility.

• Peers should be able to apply QoS attributes to topics freely (e.g. message retention, retroactive subscriptions, batching, persistence, etc.) These should surface in the protocol as topic metadata. On GRAFT/SUBSCRIBE time, peers should be able to assert QoS attributes they seek, and peers should reject their requests to subscribe if they can't honour them.

• Acquiring network-level consensus on the QoS applied to a given topic may seem tempting (e.g. all peers meshing on topic X will provide you with the last message on subscription), but it's huge pandora box and definitely out of scope.

[aschmahmann]

retroactive subscriptions

I can see both the view of adding this functionality for all routers, and the points made by @vyzo and @hsanjuan in #42 about wanting to keep a basic pubsub implementation simple.

We should separate these in layers with clear public APIs to truly deliver on the promise of extensibility

Absolutely!

These should surface in the protocol as topic metadata

That's one way to do it, and it's fine by me as long as the topic metadata can be arbitrary strings (effectively sub-protocol IDs) instead of a preset group of flags.

It's worth pointing out that classic "most recent X message" semantics are not good enough for our case because we are in a distributed systems environment and have no concept of time. We can ask for "best X messages" for some definition of "best" (e.g. largest version number with tie breaker based on hash of content, smallest message value, from peer with largest ID ...).

Acquiring network-level consensus on the QoS applied to a given topic may seem tempting

We actually already have something worse than this which is network-level consensus on which routing protocol is the best. To illustrate, let's say we have 3 peers that support protocols: P0: gossipsub, P1,P2: gsA, gossipsub. In order for pubsub (really the host + Pubsub together) to know what connection to establish between peers there needs to be a priority ordering on the protocols for each peer (e.g. gossipsub, gsA -> gossipsub).

Additionally given procotolA is higher priority than protocols 0...N, protocolA must also support sending messages to all protocols 0...N or messages may not propagate through the network. In our case above all peers support gossipsub and P1 and P2 support gsA, if gsA does not send messages to gossipsub (perhaps because the implementer of the gsA library didn't know about gossipsub) then when P1 and P2 send messages to each other they will never propagate to P0.

We've now fragmented the ecosystem on the developer end since all new routing protocol developers need to know about each other so that episub can support sending to gossipsub and floodsub, and gossipsub to floodsub.

All this is to say that we're already staring pretty squarely at Pandora's box right now and it's probably something we need to deal with. The complexity here is much larger than would be required by a scheme like prepending QoS parameters to a topic name and using those as hints.