ipfs pubsub for not directly connected nodes

[klueq]

Afaik, the current implementation of pubsub expects all the nodes interested in the topic to be directly connected with each other. This means that if two random nodes that haven't talked before subscribe to the same topic and one node publishes a message, the other node won't get it. It would be nice to enable this, though.

[Stebalien]

This should work, usually. We (go-ipfs) use the DHT to rendezvous. However, this happens in the background and may take a while. There's also a race: if both peers start at the same time, they may not find each other.

Have you actually run into this issue?

[klueq]

I haven't been able to make pubsub working recently. name publish is very slow for me too. My statement was also based on reading sources of js-ipfs which checked if a peer is also interested in the topic before sending a message to it. However js-ipfs doesn't have DHT implemented (afaik), so this might explain why their pubsub is written this way.

I left ipfs pubsub sub chat running in my console. Could you try sending me a message? I see nothing there, despite "chat" should be a popular test topic.

[Stebalien]

So, I can't seem to connect to any nodes listening on chat. You may be behind a NAT.e We have some NAT traversal features but they're far from complete.

But yeah, I'll admit that pubsub rendezvous needs to be significantly more reliable.

[klueq]

Not sure if I'm behind a NAT. I've checked this on the WebRTC Trickle ICE test app, which sends a request to a STUN server, and it returned a single IPv4 address in the 50.x.x.x range. This must be a public address.

[Stebalien]

(resurrecting this thread)

So, that doesn't mean you're not behind a NAT. It just means that either you or your NAT has a public IP address (required to use the internet).

What's the output of ipfs id?

[klueq]

user1@u320:~$ ipfs id
{
	"ID": "QmS2AgwrBGQsmsuGdYtpEyMcAMaztT4xm9Fy33c8Fjv83M",
	"PublicKey": "CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC8WIaMf8F1AuC+H7XveTH5Ry8npUaiY/LBhRRI0o+nRs9cBULtF7/ySrlM67d88upQbHuyeE2m7h4/DqB/f6VwBuerz5Yes98hN2ZPtWouPj/A2iOn+wJ9vjHNhO8OFOQGOBJW54Ue0R4fnQcq66hrSSnh1wPLiMfSg4obtkn9yYiGsH/6FTfYd1lafvz6RbmFzOpD9etx7xnc2/rN1HY6q+IZLHouCSctiFb3CJFdgDc73wzMBP7py7gT1q6yUo1D8PRNq2Wy1TeHZ5Zw03ddIQgCmVqv37LdLwKBqmxw+Bbt0I9jNBO5NnWJjelaFL4RnwvhBKf3GHiOTl2FBvORAgMBAAE=",
	"Addresses": null,
	"AgentVersion": "go-ipfs/0.4.17/",
	"ProtocolVersion": "ipfs/0.1.0"
}

[Stebalien]

Is your node online? It doesn't have any addresses.

[klueq]

Now it's online:

user1@u320:~$ ipfs id
user1@u320:~$ ipfs id
{
	"ID": "QmS2AgwrBGQsmsuGdYtpEyMcAMaztT4xm9Fy33c8Fjv83M",
	"PublicKey": "CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC8WIaMf8F1AuC+H7XveTH5Ry8npUaiY/LBhRRI0o+nRs9cBULtF7/ySrlM67d88upQbHuyeE2m7h4/DqB/f6VwBuerz5Yes98hN2ZPtWouPj/A2iOn+wJ9vjHNhO8OFOQGOBJW54Ue0R4fnQcq66hrSSnh1wPLiMfSg4obtkn9yYiGsH/6FTfYd1lafvz6RbmFzOpD9etx7xnc2/rN1HY6q+IZLHouCSctiFb3CJFdgDc73wzMBP7py7gT1q6yUo1D8PRNq2Wy1TeHZ5Zw03ddIQgCmVqv37LdLwKBqmxw+Bbt0I9jNBO5NnWJjelaFL4RnwvhBKf3GHiOTl2FBvORAgMBAAE=",
	"Addresses": [
		"/ip4/127.0.0.1/tcp/4001/ipfs/QmS2AgwrBGQsmsuGdYtpEyMcAMaztT4xm9Fy33c8Fjv83M",
		"/ip4/50.x.x.x/tcp/4001/ipfs/QmS2AgwrBGQsmsuGdYtpEyMcAMaztT4xm9Fy33c8Fjv83M",
		"/ip6/::1/tcp/4001/ipfs/QmS2AgwrBGQsmsuGdYtpEyMcAMaztT4xm9Fy33c8Fjv83M",
		"/ip4/50.x.x.x/tcp/4001/ipfs/QmS2AgwrBGQsmsuGdYtpEyMcAMaztT4xm9Fy33c8Fjv83M"
	],
	"AgentVersion": "go-ipfs/0.4.17/",
	"ProtocolVersion": "ipfs/0.1.0"
}

[VandeurenGlenn]

Hi @Stebalien
I'm having the same issue, it sometimes receives the message but mostly not, even after connecting to the node.
Just so you know, most nodes are running on the same machine (up till 30 sometimes) plus one on a VPS and it's a private network.
I recently updated to the new version, going to test on previous versions once I have the time for it.

@klueq I don't know your use case, but if your able, running a private network fixes the slow publish times.

Windows 10 x64
go-ipfs v0.4.18
nodejs v11.4.0

[VandeurenGlenn]

I played around a bit and when I start 5 nodes, only two receive a message, strangely enough after restarting all, the same two nodes continue to receive messages.

[Stebalien]

Can you try go-ipfs master? We had some issues tracking connected peers that were recently fixed.

[VandeurenGlenn]

So, compiled master from source, no change.

Didn't test on network, currently upgrading/replacing machines, once done I'll try again with physical ones, expect some follow up end of the week.

[Stebalien]

Wait, what ports are your nodes running on?

[VandeurenGlenn]

On the local machine it starts from 5001, 4001 and increases by one every node.

[Stebalien]

Got it. I wanted to make sure you weren't reusing ports. Unfortunately, because we're intentionally setting the REUSEPORT option, multiple daemons running as the same user can end up taking the same port.

[ryco117]

Hi @Stebalien , sorry if this is considered a dead thread but I've been developing an application using the docker image for go-ipfs version 0.4.18 and have been experiencing this issue. Both nodes on my network have several hundred peers in their swarm peers list, yet only if I manually swarm connect them do they actually see the other's messages. And even after being manually connected they often lose eachother as peers and are unable to see the other again.

Are there any recent thoughts or developments on this? Or any way to force my nodes to mainain a direct connection for the meantime? Thanks

[Stebalien]

Yes, actually. There has been some work on integrating the peer discovery system deeper into pubsub to make sure we're always connected to some set of peers (libp2p/go-libp2p-pubsub#184).

[Stebalien]

Fixed in 0.5.0