Partial Messages Extension by MarcoPolo · Pull Request #631 · libp2p/go-libp2p-pubsub

MarcoPolo · 2025-07-15T01:31:43Z

Implements libp2p/specs#685

Commits are meaningful. Best to review commit by commit.

extensions.go

partialmessages/partialmsgs.go

MarcoPolo · 2025-09-24T17:53:49Z

thanks for the review @ppopth, friendly reminder you can see the latest changes with git range-diff master 7fc7eb495bd6 578e12f4afcd

The bitmap is probably the common case, so add library support for it.

This allows a peer's local partsMetadata to merge with their remote partsMetadata. As long as the underlying PartsMetadata type supports CRDT-like addition, we can avoid sending the peer duplicates. Take the following example (credit Sukun): > Say: We have parts 1, 2, 5, 6 > > 1. Peer sends a message saying I have part 1, 2 > 2. We send parts 5, 6 and update the peerstate in PublishMessage to (1, 2, 5, 6) > 3. Concurrently with 2, we receive message from the peer saying I now have parts (1, 2, 3, 4) With this change we can now correctly store the fact that this peer should have parts 1,2,3,4,5,6. And in a way that does not assume a specific representation of the PartsMetadata.

and fix a bug where we would skip sending them full messages if they requested partial messages and we disabled partial messages.

if they requested partial messages on a given topic.

When a topic join happens concurrently with adding a new peer, we may send the topic join announcement before the hello packet which contains our extensions. This is a protocol violation. This change starts sending rpcs on new stream only after we've enqued a hello packet. We use UrgentPush to ensure that it's the first one sent.

instead of requiring a message object

We need some way to give peer feedback to gossipsub if the application is using partial messages. This adds a new PeerFeedback method that lets an application modify the peer score.

This lets us distinguish who can support a partial message request versus who requests partial message on a per topic basis.

Creates a mixed network where only one peer request partial, and the rest either support or don't

A peer initiated group state is a state that is initiatlized due to receiving a partial message RPC for which we have not (yet) published to. It's useful to have this state so we can remember what parts a peer has if we do publish to that group. Previously we limited the total number of peer initiated group states in a topic. This was agnostic to how many groups a single peer initiated, which meant that a single peer could exhaust the whole limit. This change tracks how many group states each peer has initialized and limits each peer to a configurable number of these states. The default total and per peer limits of 255 8 are likely good enough for most use cases, but the options exist for users to change this if they need to.

changes partialMessageStatePerTopicGroup to partialMessageStatePerGroupPerTopic to better reflect its usage.

## Summary - Replace `github.com/libp2p/go-libp2p` with `github.com/celestiaorg/go-libp2p` fork based on v0.46.0 - The fork makes `SetDeadline`, `SetReadDeadline`, and `SetWriteDeadline` return nil on mocknet streams instead of an error ## Context go-libp2p-pubsub added `SetWriteDeadline` calls in [libp2p/go-libp2p-pubsub@6587e8e](libp2p/go-libp2p-pubsub@6587e8e) (part of libp2p/go-libp2p-pubsub#631). Mocknet streams use `io.Pipe` which doesn't support deadlines, so `SetWriteDeadline` returns an error. The pubsub code treats this as fatal — resets the stream and never sends the hello packet. This prevents gossipsub peers from discovering each other's subscriptions in any mocknet-based test. Upstream PR: https://github.com/libp2p/go-libp2p/pull/new/fix/mocknet-deadline-noop

MarcoPolo changed the base branch from master to marco/extensions July 17, 2025 21:01

MarcoPolo force-pushed the marco/extensions branch 2 times, most recently from 222428a to 5299023 Compare August 29, 2025 19:56

MarcoPolo force-pushed the marco/partial-messages-2 branch from 6a50510 to 8162df6 Compare August 29, 2025 20:44

MarcoPolo force-pushed the marco/partial-messages-2 branch from 8162df6 to 74b015e Compare September 10, 2025 21:21

MarcoPolo force-pushed the marco/extensions branch 3 times, most recently from 7402459 to 939defb Compare September 15, 2025 18:43

MarcoPolo force-pushed the marco/partial-messages-2 branch 2 times, most recently from a1d0543 to c5773aa Compare September 15, 2025 19:33

MarcoPolo force-pushed the marco/extensions branch 2 times, most recently from 4e4faae to 857c8cc Compare September 15, 2025 19:37

Base automatically changed from marco/extensions to master September 15, 2025 20:32

MarcoPolo force-pushed the marco/partial-messages-2 branch 2 times, most recently from a76890d to 96339bb Compare September 15, 2025 22:32

MarcoPolo marked this pull request as ready for review September 15, 2025 22:32

MarcoPolo requested a review from sukunrt September 15, 2025 22:32

jxs mentioned this pull request Sep 16, 2025

feat: implement gossipsub partial messages extension sigp/rust-libp2p#577

Closed

haxxpop reviewed Sep 23, 2025

View reviewed changes

extensions.go Outdated Show resolved Hide resolved

haxxpop reviewed Sep 23, 2025

View reviewed changes

partialmessages/partialmsgs.go Show resolved Hide resolved

MarcoPolo force-pushed the marco/partial-messages-2 branch 3 times, most recently from 818f40d to 7fc7eb4 Compare September 23, 2025 21:05

haxxpop reviewed Sep 24, 2025

View reviewed changes