gossipsub #5373

vyzo · 2018-08-12T10:53:35Z

Adds the option to use gossipsub as the pubsub protocol.

This is handled by adding an Pubusub.Router config option, which allows the user to specify the pubsub router to use.
The possible values are

floodsub (or empty), for the default floodsub protocol
gossipsub for the gossipsub protocol

Requires:

Blocked:

Gx updates and fixes to use new cid.Builder interface. #5375

vyzo · 2018-08-12T11:03:48Z

Ugh, it seems that the configuration repo has been extracted from go-ipfs in the last day, which breaks my patch.
The option will have to be added to the extracted package ...

vyzo · 2018-08-12T11:27:38Z

Updated go-ipfs-config to include the necessary option and rebased.

Stebalien · 2018-08-13T00:45:05Z

package.json

      "name": "go-libp2p-pubsub-router",
-      "version": "0.3.9"
+      "version": "0.3.10"


FYI, I had to re-publish so this'll have to be re-updated.

ok, will do.

Unfortunately this pulled a bunch of new deps, so lots of things need to be updated recursively... sigh

Ok, this update touches too many things -- we'll have to wait for #5375 I think, as there are also code changes involved.

Stebalien · 2018-08-13T00:47:57Z

test/sharness/t0180-pubsub-gossipsub.sh

+
+test_expect_success "enable gossipsub" '
+  for x in $(seq 0 4); do
+    ipfsi 0 config Pubsub.Router gossipsub


Should be ipfsi $x .... FYI, you can also use iptb for-each ipfs config ....

It's a bit odd that the tests are still passing.

oops, good catch!
It's not odd that the test still pass, gossipsub is backwards compatible with floodsub.

Ah. Good to know.

vyzo · 2018-08-13T08:59:23Z

We are blocked on #5375 as it interleaves a big update with a competing update to go-libp2p-pubsub-router.

We will have to wait for those updates to get merged (and rebase) before we can proceed, as it's out of scope to include a cascade of irrelevant package touches in this pr, let alone that it would create conflicts.

Stebalien · 2018-08-13T16:16:21Z

@vyzo update merged.

vyzo · 2018-08-13T16:41:41Z

rebased.

vyzo · 2018-08-13T16:44:03Z

hrm, some merge errors -- fixing.

vyzo · 2018-08-13T16:48:07Z

should be ok now.

Stebalien · 2018-08-14T18:34:08Z

Rebased.

magik6k · 2018-08-14T22:56:06Z

test/sharness/t0180-pubsub-gossipsub.sh

+'
+
+# this is just a copy of t0180-pubsub; smell.
+startup_cluster $NUM_NODES --enable-pubsub-experiment


You should be able to wrap this into a function in the pubsub test and call it with different setup, like https://github.com/ipfs/go-ipfs/blob/master/test/sharness/t0170-dht.sh#L10

I kind of wanted to keep the two tests separate, but maybe they should be one.
We could also have a library script that contains the test logic in a function, and then source that from the two tests...

Up to you. I'm usually not a stickler on DRYing up tests, especially integration tests. We should never have to change these anyways ("don't break userspace").

yeah, me neither... there is some merit in having a separate test for gossipsub so that the two can vary and perhaps have tests for protocol specific behaviour.

so if anyone has strong feelings about it, i can merge into a single test -- but i am otherwise inclined to keep it as is.

magik6k · 2018-08-14T22:57:28Z

docs/experimental-features.md

@@ -43,6 +43,11 @@ experimental, default-disabled.
 run your daemon with the `--enable-pubsub-experiment` flag. Then use the
 `ipfs pubsub` commands.

+### gossipsub
+
+You can enable the new experimental gossipsub protocol via configuration:


I'd add a short line explaining the difference between the default floodsub and gossipsub.

hrm, maybe, although that's a topic in itself. There will be a gossipsub spec soon [TM] and perhaps we could link to that instead.

For now, I'd just go with:

Gossipsub is a new, experimental routing protocol for pubsub that should waste less bandwidth than the current default routing protocol (floodsub). It's backwards compatible with floodsub so enabling this feature shouldn't break compatibility with existing IPFS nodes.

That should tell users everything they need to know (for now).

that sounds good -- will add.

License: MIT Signed-off-by: vyzo <[email protected]>

in particular, this enabls gossipsub when configured so. License: MIT Signed-off-by: vyzo <[email protected]>

just a copy of t0180-pubsub with gossipsub enabled for now License: MIT Signed-off-by: vyzo <[email protected]>

License: MIT Signed-off-by: vyzo <[email protected]>

License: MIT Signed-off-by: Steven Allen <[email protected]>

License: MIT Signed-off-by: vyzo <[email protected]>

gossipsub This commit was moved from ipfs/kubo@99df463

vyzo requested a review from Kubuxu as a code owner August 12, 2018 10:53

ghost assigned vyzo Aug 12, 2018

ghost added the status/in-progress In progress label Aug 12, 2018

vyzo requested review from Stebalien and whyrusleeping August 12, 2018 10:54

vyzo mentioned this pull request Aug 12, 2018

Add pubsub configuration options ipfs/go-ipfs-config#3

Merged

vyzo force-pushed the feat/gossipsub branch 2 times, most recently from f3f640f to 2753fd8 Compare August 12, 2018 11:26

vyzo force-pushed the feat/gossipsub branch from 2753fd8 to e0ed3aa Compare August 12, 2018 11:30

Stebalien reviewed Aug 13, 2018

View reviewed changes

vyzo force-pushed the feat/gossipsub branch from 60568d4 to 18aa320 Compare August 13, 2018 07:40

vyzo force-pushed the feat/gossipsub branch from 18aa320 to 803291a Compare August 13, 2018 16:41

vyzo force-pushed the feat/gossipsub branch from 803291a to 1edacc1 Compare August 13, 2018 16:47

ghost assigned Stebalien Aug 13, 2018

Stebalien approved these changes Aug 13, 2018

View reviewed changes

Stebalien force-pushed the feat/gossipsub branch from caa4a2a to 23a4097 Compare August 14, 2018 18:33

Stebalien added RFM and removed status/in-progress In progress labels Aug 14, 2018

Stebalien requested a review from magik6k August 14, 2018 21:32

magik6k reviewed Aug 14, 2018

View reviewed changes

vyzo added 2 commits August 15, 2018 10:02

update go-floodsub to 0.9.24

ca0c3ba

License: MIT Signed-off-by: vyzo <[email protected]>

update go-ipfs-config

e002de9

License: MIT Signed-off-by: vyzo <[email protected]>

vyzo and others added 5 commits August 15, 2018 10:02

instantiate pubsub router based on configuration

8a7a261

in particular, this enabls gossipsub when configured so. License: MIT Signed-off-by: vyzo <[email protected]>

gossipsub sharness test

884df03

just a copy of t0180-pubsub with gossipsub enabled for now License: MIT Signed-off-by: vyzo <[email protected]>

document Pubsub.Router option to enable gossipsub

643522e

License: MIT Signed-off-by: vyzo <[email protected]>

fix gossipsub test initialization

42a9cdc

License: MIT Signed-off-by: vyzo <[email protected]>

fix gx duplicate dep by updating iptb

9300997

License: MIT Signed-off-by: Steven Allen <[email protected]>

Stebalien force-pushed the feat/gossipsub branch from 23a4097 to 9300997 Compare August 15, 2018 17:03

ghost added the status/in-progress In progress label Aug 15, 2018

Stebalien added need/author-input Needs input from the original author and removed RFM labels Aug 15, 2018

better documentation for gossipsub

f8725d8

License: MIT Signed-off-by: vyzo <[email protected]>

magik6k approved these changes Aug 15, 2018

View reviewed changes

Stebalien added RFM and removed status/in-progress In progress need/author-input Needs input from the original author labels Aug 15, 2018

Stebalien merged commit 99df463 into master Aug 15, 2018

Stebalien deleted the feat/gossipsub branch August 15, 2018 18:45

hacdias pushed a commit to ipfs/boxo that referenced this pull request Jan 27, 2023

Merge pull request ipfs/kubo#5373 from ipfs/feat/gossipsub

0d7e854

gossipsub This commit was moved from ipfs/kubo@99df463

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gossipsub #5373

gossipsub #5373

vyzo commented Aug 12, 2018 •

edited by Stebalien

Loading

vyzo commented Aug 12, 2018

vyzo commented Aug 12, 2018

Stebalien Aug 13, 2018

vyzo Aug 13, 2018

vyzo Aug 13, 2018

vyzo Aug 13, 2018

Stebalien Aug 13, 2018

Stebalien Aug 13, 2018

vyzo Aug 13, 2018

Stebalien Aug 13, 2018

vyzo commented Aug 13, 2018 •

edited

Loading

Stebalien commented Aug 13, 2018

vyzo commented Aug 13, 2018

vyzo commented Aug 13, 2018

vyzo commented Aug 13, 2018

Stebalien commented Aug 14, 2018

magik6k Aug 14, 2018

vyzo Aug 15, 2018

Stebalien Aug 15, 2018

vyzo Aug 15, 2018

vyzo Aug 15, 2018

magik6k Aug 14, 2018

vyzo Aug 15, 2018 •

edited

Loading

Stebalien Aug 15, 2018

vyzo Aug 15, 2018

vyzo Aug 15, 2018

gossipsub #5373

gossipsub #5373

Conversation

vyzo commented Aug 12, 2018 • edited by Stebalien Loading

vyzo commented Aug 12, 2018

vyzo commented Aug 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vyzo commented Aug 13, 2018 • edited Loading

Stebalien commented Aug 13, 2018

vyzo commented Aug 13, 2018

vyzo commented Aug 13, 2018

vyzo commented Aug 13, 2018

Stebalien commented Aug 14, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vyzo Aug 15, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vyzo commented Aug 12, 2018 •

edited by Stebalien

Loading

vyzo commented Aug 13, 2018 •

edited

Loading

vyzo Aug 15, 2018 •

edited

Loading