In terms of WireGuard functionality, a peer can be behind a firewall/NAT and reach out to a known and accessible host and keep the connection alive using the PersistentKeepalive option (eg PersistentKeepalive=25). This new node will now be directly accessible.

I would guess that wesher should be able to easily use this by simply requiring the new "hidden" node to initiate a connection to any accessible peer. Once the connection initiated, then this "hidden" node would get information on more wesher nodes and continue adding mesh connections.

However, I don't think that any two nodes that are hidden from each other could create a direct mesh connection, however, they can route to each other through WireGuard through any of the nodes that they can connect directly to. Several hidden nodes might visible to each other so they would continue meshing as possible.

So you should easily be able to have scalable cluster with a automatic partial meshing even for hidden nodes.

If for some reason, your meshing communication needs to initiate in both directions. The new "hidden" nodes reaching out to connect to a see node could use a pre-defined private/public key set and IP simply for the initial communication which would occur over WireGuard, then reconnect with the final configuration.

I would guess that wesher should be able to easily use this by simply requiring the new "hidden" node to initiate a connection to any accessible peer. Once the connection initiated, then this "hidden" node would get information on more wesher nodes and continue adding mesh connections.

The problem is: wesher does not use the wireguard connection for mesh establishment. It uses memberlist.
I don't currently see a clean way of running the memberlist communication on top of wireguard, since it would be a bit of a chicken-and-egg situation.

However, I don't think that any two nodes that are hidden from each other could create a direct mesh connection

Yes, we'd need at least one open node. An "introductor" to the mesh.

however, they can route to each other through WireGuard through any of the nodes that they can connect directly to

This would require a more complicated topology than the current simple full-mesh. We'd have to keep track of indirect routes. It might be the direction we have to go anyway, but I'd like to avoid it if possible.

Understood. You mean https://godoc.org/github.com/hashicorp/memberlist :

    
    // Configuration related to what address to advertise to other
    // cluster members. Used for nat traversal.
    AdvertiseAddr string
    AdvertisePort int
    

If I am understanding correctly, this should already work out of the box for NAT traversal?

If I am understanding correctly, this should already work out of the box for NAT traversal?

Kinda. Unless I'm overlooking something, these are only there to allow working behind NATs, but it doesn't do any of the actual heavy lifting. You - as a memberlist user - still are responsible for discovering the actual external NAT address/port (presumably because memberlist doesn't want to dictate how you set up your network: you could use fixed address/port NAT or dynamic with uPnP+IGD or something else).

So we still have to implement this part in wesher. I would have no problem whatsoever if you wanna start tackling this 😉

If a new hidden wesher node can call out to at least one visible memberlist address, and since the memberlist is gossiping, all your need is that the visible member returns all other peers to the hidden node through open call from the new hidden node (NAT traversal).

The new hidden node should then be able to call out to each node in the list and the visible ones will return directly the info required to handshake WireGuard. From reading the functions, this should work.

I guess the big question, which I don't know the answer because I don't know memberlist well, is can the hidden node push it's public key through to the gossiping via a visible node? Technically this can be done because the hidden node is calling a visible node and could send it's info and the visible node can respond with a know memberlist. Both sides need to share all known public keys and IPs (hidden node shares it's only public key, visible node shares all gossiped public keys).

You can also consider to continue the memberlist gossiping within the newly establised WireGuard subnet and continue gossiping outside the WireGuard subnet also), but skipping WireGuard subnet addresses when establishing new WireGuard peer-to-peer connections (since they are already meshed).

"still are responsible for discovering the actual external NAT address/port": WireGuard will do this as long as the hidden node can call out to a visible node and both sides have each other's public keys. Once two peers have meshed, you could even get them to share a pre-shared key within this new tunnel and they could then self-harden their specific mesh link.

https://godoc.org/github.com/hashicorp/memberlist#Memberlist.SendReliable
https://godoc.org/github.com/hashicorp/memberlist#Memberlist.Members

I will try testing soon once I have time to dig into this.

@realcarbonneau ,
Any update on your investigation into nat reachability of hidden nodes? This topic interests me -- instead of using tailscale, I'd rather try wesher because I'll have full control of my nodes and isnt a hosted service

any update ? Its very good feature