Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

namesys/pubsub: pubsub Publisher and Resolver #4047

Merged
merged 1 commit into from
Nov 30, 2017

Conversation

vyzo
Copy link
Contributor

@vyzo vyzo commented Jul 7, 2017

Implements a namesys publisher and resolver with pubsub.

@vyzo vyzo force-pushed the ipns-pubsub branch 6 times, most recently from dbfbf81 to 61b2613 Compare July 10, 2017 09:29
@vyzo vyzo changed the title [WIP] namesys/pubsub: pubsub Publisher and Resolver namesys/pubsub: pubsub Publisher and Resolver Jul 10, 2017
@vyzo vyzo force-pushed the ipns-pubsub branch 23 times, most recently from 7d7d693 to a477208 Compare July 11, 2017 17:50
@vyzo
Copy link
Contributor Author

vyzo commented Nov 18, 2017

Forced pushed the last commit to restart jenkins -- errored out with some system-specific gremlin.
Rebased on master.

@vyzo vyzo force-pushed the ipns-pubsub branch 3 times, most recently from 7ee2ec4 to 08b519e Compare November 19, 2017 09:45
@vyzo
Copy link
Contributor Author

vyzo commented Nov 19, 2017

Updated for changes in the command interface.

@vyzo
Copy link
Contributor Author

vyzo commented Nov 19, 2017

it's not clear why jenkins failed -- all the not ok's are marked as known breakages.
I also can't restart the build through the interface for some reason.

@keks
Copy link
Contributor

keks commented Nov 19, 2017

@vyzo I restarted both the broken travis build and the jenkins build. Both fail, but at different places.

Travis:
in test/sharness/t0183-namesys-pubsub.sh

expecting success: 
    echo -n "" > expected &&
    ipfsi 1 name pubsub subs > subs1 &&
    ipfsi 2 name pubsub subs > subs2 &&
    test_cmp expected subs1 &&
    test_cmp expected subs2

> diff -u expected subs1
--- expected	2017-11-19 13:20:38.000000000 +0000
+++ subs1	2017-11-19 13:20:38.000000000 +0000
@@ -1 +0,0 @@
--n 

not ok 18 - check subscriptions

It looks like echo -n prints -n on the Travis Mac builders.

Jenkins:
in test/sharness/t0088-repo-stat-symlink.sh

expecting success: 
  export IPFS_PATH="sym_link_target" &&
  reposize_direct=$(ipfs repo stat | grep RepoSize | awk '{ print $2 }') &&
  export IPFS_PATH=".ipfs" &&
  reposize_symlink=$(ipfs repo stat | grep RepoSize | awk '{ print $2 }') &&
  test $reposize_symlink -ge $reposize_direct

not ok 4 - 'ipfs repo stat' RepoSize is correct with sym link

@vyzo
Copy link
Contributor Author

vyzo commented Nov 19, 2017

It looks like echo -n prints -n on the Travis Mac builders.

wow, that's very unfortunate. I wonder if there is a version of echo that works correctly, otherwise i'll have to use touch with a unique filename.

in test/sharness/t0088-repo-stat-symlink.sh

that's completely unrelated to this PR.

@vyzo
Copy link
Contributor Author

vyzo commented Nov 19, 2017

Changed the sharness test to use rm && touch instead of echo -n.

@vyzo
Copy link
Contributor Author

vyzo commented Nov 19, 2017

Another jenkins failure, similar to the first one -- all the not ok are in known breakage.
What's up with our jenkins setup, why is it being flaky?

@vyzo
Copy link
Contributor Author

vyzo commented Nov 19, 2017

note that circle and travis were successful.
@keks can you give jenkins another restart to see if the pattern repeats?

@whyrusleeping
Copy link
Member

@vyzo we use printf instead of echo -n generally. Its the most cross platform thing i've found

@whyrusleeping
Copy link
Member

@Stebalien could you check out the gx deps here? I think that the go-testutil dep is a bit wonky (other things are importing the old go-testutil)

@Stebalien
Copy link
Member

Dammit. I did not mean to update that package. Let's try this again from the bottom.

@Stebalien
Copy link
Member

On the bright side, #4403 should fix that issue. Of course, it'll probably introduce a bunch of ever so fun merge conflicts.

Commits:
namesys: pubsub Publisher and Resolver
namesys/pubsub: pacify code climate.
namesys/pubsub: timeout for rendezvous
namesys/pubsub: filter self in bootstrap connections
namesys/pubsub: Publish to the correct topic

License: MIT
Signed-off-by: vyzo <[email protected]>

namesys/pubsub: unit  test

Commits:
namesys/pubsub: test
namesys/pubsub_test: pacify code climate
namesys/pubsub: update test to use extant mock routing

License: MIT
Signed-off-by: vyzo <[email protected]>

namesys/pubsub: integrate namesys pubsub

namesys: integrate pubsub resolvers
namesys/pubsub_test: tweak delays
- trying to make travis happy.
namesys/pubsub: fix duplicate bootstraps
- subscription key is topic, not ipnskey.
namesys/pubsub: no warning needed on cancellation
namesys/pubsub: warning for receive errors
- and more informative error messages at that.
namesys/pubsub_test: smaller test
- make it work with seemingly low fdlimits in travis/macosx.
  also, more informative test failures.
namesys/pubsub: add delay to let pubsub perform handshake
namesys/pubsub: update gx imports
namesys/pubsub_test: preconnect publisher, reduce delays
- preconnects the publisher to the receivers in order to avoid bootstrap
  flakiness with connectivity problems in travis.
  reduces sleeps to 1s for flood propagation (3s seems excessive with 5 hosts).
namesys/pubsub: drop named return values in resolveOnce
- per review comment.
namesys/pubsub: check errors
namesys/pubsub: store bytes in resolver datastore
namesys/pubsub: resolver Cancel
- for canceling subscriptions, pre whyrusleeping's request.
namesys/pubsub: fix resolution without /ipns prefix
- also improve the logging a bit.
namesys/pubsub: don't resolve own keys through pubsub
namesys/pubsub: signal ErrResolveFailed on resolution failure
namesys/pubsub: use sync datastore, resolver lock only for subs
namesys/pubsub_test: coverage for Cancel

License: MIT
Signed-off-by: vyzo <[email protected]>

namesys/pubsub: parallelize dht and pubsub publishing

Commits:
namesys/pubsub: code cosmetics
namesys: parallelize publishing with dht and pubsub
namesys/pubsub: periodically reprovide topic rendezvous
namesys/pubsub: cancelation for rendezvous goroutine
namesys/pubsub: log ipns record seqno on publish

License: MIT
Signed-off-by: vyzo <[email protected]>

namesys/pubsub: error checking

License: MIT
Signed-off-by: vyzo <[email protected]>

namesys/pubsub: --enable-namesys-pubsub option and management

Commits:
package.json: update go-libp2p-blankhost
namesys: fix stale package imports
update go-testutil
namesys/pubsub: reduce bootstrap provide period to 8hr
namesys/pubsub: try to extract the key from id first
option to enable ipns pubsub: --enable-namesys-pubsub
ipfs name pubsub management subcommands
corehttp/gateway_test: mockNamesys needs to implement GetResolver
pacify code climate

License: MIT
Signed-off-by: vyzo <[email protected]>

namesys/pubsub: pubsub sharness test

test/sharness: test for ipns pubsub
namesys/pubsub: return boolean indicator on Cancel
package.json: remove duplicate entry for go-testutil
update gx deps, testutil to 1.1.12
fix jenkins failure: use tabs in t0183-namesys-pubsub
t0183: use 4 spaces for tabification

License: MIT
Signed-off-by: vyzo <[email protected]>

namesys/pubsub: update for new command interface

License: MIT
Signed-off-by: vyzo <[email protected]>

namesys/pubsub: fix sharness test for broken MacOS echo

echo -n "" should print -n, but hey it's a mac.

License: MIT
Signed-off-by: vyzo <[email protected]>
@whyrusleeping
Copy link
Member

I squashed and rebased, the dependency issues should be fixed now

@whyrusleeping
Copy link
Member

Strange ipns related failure on jenkins, copied the failure logs here: https://gist.github.com/whyrusleeping/9f4bef1726577e14a67568b9cd997f5a

Any chance this is related to the pubsub changes @vyzo ?

cc @Stebalien as maybe this is a known random failure.

@vyzo
Copy link
Contributor Author

vyzo commented Nov 22, 2017

the tests seem to have a 300ms timeout -- maybe that's the cause?
I haven't seen this failing before, but normal ipns resolution shouldn't be affected

@vyzo
Copy link
Contributor Author

vyzo commented Nov 22, 2017

@whyrusleeping can you rerun jenkins to see if the problem is persistent?
it might be something post squash, or it might be a gremlin (or simply 300ms not being enough). I still don't have rebuild permissions on jenkins.

@whyrusleeping
Copy link
Member

@victorbjelkholm can you trigger this one to be rebuilt? And also tell me how to do it myself?

@whyrusleeping whyrusleeping merged commit 66bde41 into ipfs:master Nov 30, 2017
@ghost ghost removed the status/in-progress In progress label Nov 30, 2017
@vyzo vyzo deleted the ipns-pubsub branch November 30, 2017 13:05
hacdias pushed a commit to ipfs/boxo that referenced this pull request Jan 27, 2023
namesys/pubsub: pubsub Publisher and Resolver

This commit was moved from ipfs/kubo@66bde41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic/ipns Topic ipns
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants