Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve checking for p2p connection limits #2985

Merged
merged 6 commits into from
Aug 21, 2019

Conversation

j01tz
Copy link
Member

@j01tz j01tz commented Jul 31, 2019

Connection limiting can be improved by adding a check against maximum allowed peer count immediately after a connection is accepted.

Copy link
Member

@antiochp antiochp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just had a thought about this - our seed nodes currently accept connection requests and send "peer lists" back to the recently connected peers. Seed nodes will eventually notice they have too many peers and will randomly pick some to shutdown. I don't think we look at the age of the connection when picking some to close.

This change will alter this behavior - potentially making it harder for new peers joining the network to obtain peer lists from the initial seeds. If a seed is at capacity then it will simply close any new connection before it has chance to send any msgs to the new peer.

Edit: What if checked against peer+max_count + 10% or some other softer limit?

p2p/src/serv.rs Outdated Show resolved Hide resolved
@DavidBurkett
Copy link
Contributor

There's been lots of discussion about this in the past. See: #2726
#2725

I still believe that anytime we're dropping a new peer immediately after accepting the connection, it's imperative that we send a peer list first.

@j01tz
Copy link
Member Author

j01tz commented Jul 31, 2019

I still believe that anytime we're dropping a new peer immediately after accepting the connection, it's imperative that we send a peer list first.

My initial thinking is that we want to consume as few resources as possible until we know the new connection is desirable (not from a banned peer, not from a known peer and not if it would exceed our maximum configured connection limit).

Otherwise it would allow "undesirable" connections to consume more resources than necessary (like sending peer lists), even after a ban. I don't think it makes sense to send a peer list to a banned peer for example?

@DavidBurkett
Copy link
Contributor

That's fine, we can check if they're banned before sending a peer list. But if we're disconnecting simply because we have too many incoming connections, we could have a real bootstrapping problem if we don't provide alternative peers.

@j01tz
Copy link
Member Author

j01tz commented Jul 31, 2019

But if we're disconnecting simply because we have too many incoming connections, we could have a real bootstrapping problem if we don't provide alternative peers.

Agreed it would potentially impact bootstrapping ability with seed nodes in current form.

Do you see this remaining an issue if the seed nodes don't maintain the proposed rule of shutting down new incoming connections while connection limit is reached without sending a peer list?

Are there cases where non-seed nodes with default connection limits following this rule would likely impact bootstrapping?

@DavidBurkett
Copy link
Contributor

Seed nodes would probably be enough, but what constitutes a seed node? We don't currently have a config setting for that, so we would have to add one.

@j01tz j01tz changed the title Improve checking for p2p connection limits [WIP] Improve checking for p2p connection limits Aug 13, 2019
@j01tz
Copy link
Member Author

j01tz commented Aug 13, 2019

After some more thought and further consideration of #2726:

Goals:

  • limit accepting incoming connections (async cleanup not sufficient)
  • allow network to grow even if all peers are already at max peer limit (maintain some flexibility)
  • improve clarity around distinction between incoming and outgoing connections

Changes:

  • replace PEER_MAX_COUNT with PEER_MAX_INBOUND_COUNT and PEER_MAX_OUTBOUND_COUNT to clarify and make configurable incoming and outgoing connection limits
  • improve check at network level for tcp listener to avoid accepting too many incoming connections
    • include a buffer PEER_LISTENER_BUFFER_COUNT to exceed this limit if necessary, but not by much (default value is 8)
    • make this configurable for use with seed nodes etc
  • increase minimum desired outgoing connections from 4 to 8
    • reduces chance of getting partitioned from the network
    • can improve block/tx propagation
    • makes eclipse attacks more expensive
  • limit outgoing connections to 8
    • prevents exhausting potential incoming connection slots
      • not all nodes allow incoming connections
      • want to make good use of available incoming connections for the network
      • don't need to take up slots with many outgoing connections that could be used for incoming
      • reduces resources on nodes that don't allow incoming connections

I'm updating the PR to [WIP] as this probably deserves more thought and testing, especially around limiting outgoing connections to 8.

The changes above may be too much and can be rolled back in favor of simply having the check in the listener with a configurable 8 peer buffer.

@antiochp
Copy link
Member

antiochp commented Aug 14, 2019

I actually like what is being proposed here. Making a more explicit distinction between incoming and outgoing connection handling makes a lot of sense. They are different and probably should be handled differently.

All nodes have outbound connections (I guess technically some nodes may not have any but that would be unusual). Only "public" nodes have inbound connections.

  • Nodes with outbound connections need to maintain a healthy number of them.
  • Nodes that support inbound connections need to balance conflicting needs around allowing connections and not allowing DoS via too many connections.

My only comment is this may be overkill short-term. We may want to tackle this is two stages -

  • this PR plus the proposed "8 peer temporary buffer"
  • then tackle the larger inbound vs outbound suggestion proposed here

Edit: Ahh I thought you were just talking about future changes. I did not realize you had actually made them...
In that case its probably not overkill - let me take another look but 👍 on the proposal.

Copy link
Contributor

@hashmap hashmap left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, will test it. I like the clear separation of inbound and outbound connections, it was suggested recently because bitcoin uses the same approach

#peer_max_inbound_count = 117

#maximum number of outbound peer connections
#peer_max_outbound_count = 8
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

#maximum number of peers
#peer_max_count = 125
#maximum number of inbound peer connections
#peer_max_inbound_count = 117
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should just leave this as 125 or maybe even 128 for a nice round number.
It was arbitrary to begin with, definitely don't need to tweak them to add up to 125 in total.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Increased the limit to 128 here.

p2p/src/types.rs Outdated
const PEER_MAX_INBOUND_COUNT: u32 = 117;

/// The max outbound peer count
const PEER_MAX_OUTBOUND_COUNT: u32 = 8;

/// min preferred peer count
const PEER_MIN_PREFERRED_COUNT: u32 = 8;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this one be inbound/outbound specific as well?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed PEER_MIN_PREFERRED_COUNT in favor of PEER_MIN_PREFERRED_OUTBOUND_COUNT

p2p/src/types.rs Outdated
@@ -268,11 +287,24 @@ impl P2PConfig {
}
}

/// return peer_max_count
/// return total maximum peer count (incoming + outgoing)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we still need this total max count anywhere?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was still used when broadcasting compact blocks and transactions for Peers but should be able to be removed in favor of making incoming and outgoing connections more explicit. Please see the unresolved question posted here #2985 (comment)

Copy link
Member

@antiochp antiochp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great. 👍

@j01tz
Copy link
Member Author

j01tz commented Aug 14, 2019

Changes in 7bfb54e:

  • increase peer_max_inbound_count from 117 to 128
  • remove peer_min_preferred_count in favor of peer_min_preferred_outbound_count
  • consolidate redundant enough_peers() with healthy_peers_mix()
  • remove peer_max_count
  • better distinguish outbound connections in p2p and servers

Potential unresolved question:
- Do we want to broadcast compact blocks and transactions only to our outgoing connected peers (limited to 8) or temporarily exceed that limit to broadcast to all currently connected peers?
- previously to broadcast we iterate through connected_peers() up to the max_peer_count() to broadcast to as many peers as we are connected to up to the maximum limit
- this was kept the same in spirit so far, as the connection directionality is still ignored in Peers::broadcast() and we still attempt to broadcast up to the theoretical maximum connection limit

@DavidBurkett
Copy link
Contributor

We definitely don't want to just broadcast to outgoing peers. Non-public nodes would never receive new blocks or transactions.

@j01tz
Copy link
Member Author

j01tz commented Aug 14, 2019

We definitely don't want to just broadcast to outgoing peers. Non-public nodes would never receive new blocks or transactions.

That's where I ended up. I wasn't certain if nodes would request those on their own from peers or not (even if so the propagation delay could be ugly). It sounds like we definitely want to broadcast to all connected peers.

@DavidBurkett
Copy link
Contributor

I wouldn't say definitely broadcast to all. Figuring out the optimal number is more a graph theory problem. @tromp could probably tell us how to figure that out. We'd probably have to take into account expected percentage of public nodes, average number of incoming and outgoing peers per public node, and average number of outgoing peers per private node.

@j01tz
Copy link
Member Author

j01tz commented Aug 14, 2019

Agreed in the long term that a more complex mechanism will get us a more desirable result. My goal for this PR for now is to add a basic limit to incoming p2p connections by distinguishing incoming and outgoing connections without changing too much. I think as is, the broadcasting should behave the same as before, but the question in general could probably use more thought.

@antiochp
Copy link
Member

I think as is, the broadcasting should behave the same as before, but the question in general could probably use more thought.

Yeah I was actually wondering (briefly) about just broadcasting to outbound but then realized the same thing - some nodes only have outbound connections and they would never receive anything.

We should not distinguish between inbound/outbound when broadcasting (at least in the short-term) and just broadcast to max peers.

@antiochp
Copy link
Member

👍

#until we get to at least this number
#peer_min_preferred_count = 8
#maximum number of outbound peer connections
#peer_max_outbound_count = 8
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the point of this? If this is greater than peer_min_preferred_outbound_count, do we continue to connect to more outbound peers?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

peer_max_outbound_count lets us put a configurable limit on how many outgoing connections with peers we can have and by extension all connections. We continue to try to connect to more outbound peers until peer_min_preferred_outbound_count is reached (a "healthy peers mix"). By default these values are the same (we try to maintain connections on all available outgoing slots)

Copy link
Contributor

@DavidBurkett DavidBurkett Aug 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But let's say you have peer_min_preferred_outbound_count of 8, and peer_max_outbound_count of 16, how would you ever end up with more than 8 outbound connections?

Copy link
Member Author

@j01tz j01tz Aug 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may not after future refactoring to better clean up distinguishing incoming and outgoing connections. For now it happens when we broadcast compact blocks for example. There are probably still cases where outgoing connections are opened for one reason or another, exceeding the peer_max_outbound_count at least until clean_peers() is run to clean up any potential excess connections. In this case if peer_max_outbound_count was set to 16, I think we would hang on to those connections instead of dropping them when cleaning up peers.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😕 You're saying we open outbound connections when broadcasting compact blocks? Where in the code does that take place, and what for?

Copy link
Member Author

@j01tz j01tz Aug 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After taking a closer look new outbound connections are not open when broadcasting, my mistake. During testing there were cases where logic allowed opening new connections beyond the outgoing limit which got closed shortly after during clean_peers(). You can set some test limits and watch logs to see the outgoing connections be temporarily exceeded. Hopefully as we better define incoming and outgoing connections in future PRs these cases can all be thoughtfully handled (as in the broadcast scenarios)

@@ -300,11 +303,14 @@ fn listen_for_addrs(
return;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can remove the peers.peer_count() > peers.peer_outbound_count() check on line 302, since healthy_peers_mix() covers it.

Copy link
Member Author

@j01tz j01tz Aug 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense. They don't quite check the same thing but I think we only want the healthy_peers_mix() check here as you said. I'm pretty sure we don't care to check here that the total connected peers exceeds the outbound connected peers. This would require (I think) that we have inbound connections, and I'm pretty sure we want to return here any time our outbound connections >= our configured preferred minimum outbound connections ("a healthy number of outbound peers").

Removed in fb9b06e unless there is a good reason to keep this as is.

p2p/src/peers.rs Outdated
pub fn enough_peers(&self) -> bool {
self.peer_count() >= self.config.peer_min_preferred_count()
}

/// We have enough peers, both total connected and outbound connected
pub fn healthy_peers_mix(&self) -> bool {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd advise changing the name of this now, since "healthy_peers_mix" doesn't really represent what it's checking. Possible options are something like meets_preferred_outbound, or preferably reverse the logic and call it needs_outbound_connections. The logic for managing connections was already built piecemeal through a series of defect corrections, so it's probably best to try to keep it from getting anymore complicated/confusing.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The naming here wasn't immediately intuitive to me either. My understanding is that it is a result of needing to maintain a healthy mix of outbound connections for dandelion to allow each peer to select a random outbound relay to send txs along the stem. Without ensuring a "healthy peers mix" apparently some nodes were not having enough slots for outbound connections.

With that context it makes a little more sense to me- a healthy peers mix is one
that has at least the minimum number of preferred outbound connections. So while I agree it doesn't strictly represent what is checked (outbound connections >= configured min preferred outbound connections), I think for now it does an ok job of capturing what we are checking for (that we have a healthy peers mix of outgoing connections).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, I agree that's where the name came from, but you've now just changed the functionality, so my claim is the name no longer makes sense. Before, it used to compare inbound and outbound count so it made sense, but it's no longer about having a healthy mix, and is now just about making sure we have a minimum number of outbound.

Copy link
Member Author

@j01tz j01tz Aug 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, it could make sense to use a more descriptive name about functionality rather than a prescriptive name about health. Updated in d06a8e1

@DavidBurkett
Copy link
Contributor

Agreed in the long term that a more complex mechanism will get us a more desirable result. My goal for this PR for now is to add a basic limit to incoming p2p connections by distinguishing incoming and outgoing connections without changing too much. I think as is, the broadcasting should behave the same as before, but the question in general could probably use more thought.

It wouldn't be a more complex mechanism, I don't think. It's a complex formula to figure out by hand what the optimal number is, but then it's just a matter of broadcasting to n random peers. I'm ok with pushing this off though, just know that our network will be a whole lot noisier than it needs to be.

@DavidBurkett
Copy link
Contributor

I spoke with @tromp about this, and he found: https://www.quora.com/How-many-peers-is-a-Bitcoin-node-connected-to-at-all-times
It seems bitcoin chooses 8 random peers when broadcasting. I recall igno at one point also tossing out 8 as the magic number.

@antiochp
Copy link
Member

antiochp commented Aug 15, 2019

I think I originally introduced the "broadcast to 8 of our peers regardless of total peer connections" logic and only for headers. We never did this for transactions as far as I'm aware.

Currently we implement the following -

  • headers - broadcast to peer_min_preferred_count
  • compact blocks (initial broadcast from miner) - broadcast to peer_max_count
  • transactions - broadcast to peer_max_count

But I think there is a lot of misinformation and misunderstanding out there on the internet regarding this. And some of the confusion comes from the inbound/outbound terminology.
The "bitcoin node has 8 peers" thing is really old and I don't think its been like that for a long time. My understanding is bitcoin nodes connect to many more peers than 8.
And the documentation out there around exactly which peers get relayed to is confusing and ambiguous.

This documentation implies nodes send to all their peers for block relay -

https://bitcoin.org/en/p2p-network-guide#term-unsolicited-block-push
https://bitcoin.org/en/p2p-network-guide#term-standard-block-relay

And this documentation suggests transactions are sent to all peer as well -

https://en.bitcoin.it/wiki/Network#Standard_relaying

Standard relaying
When someone sends a transaction, they send an inv message containing it to all of their peers. Their peers will request the full transaction with getdata. If they consider the transaction valid after receiving it, they will also broadcast the transaction to all of their peers with an inv, and so on. Peers ask for or relay transactions only if they don't already have them. A peer will never rebroadcast a transaction that it already knows about, though transactions will eventually be forgotten if they don't get into a block after a while. The sender and receiver of the transaction will rebroadcast, however.
Anyone who is generating will collect valid received transactions and work on including them in a block. When someone does find a block, they send an inv containing it to all of their peers, as above. It works the same as transactions.

I'm not entirely sure which what the correct interpretation of bitcoin behavior is, but I'm less convinced that some magic 8 peers is used anywhere (even if we did this for header first broadcast, which I think may be wrong behavior).

@DavidBurkett
Copy link
Contributor

@antiochp Ok, I went to the actual source code, since documentation is inconsistent. Here's what I found.

For compact blocks, looks like bitcoin actually only relays to up to 3 peers, but the whole system is different than ours (btc nodes request to receive compact blocks from specific peers). https://github.com/bitcoin/bips/blob/master/bip-0152.mediawiki and https://github.com/TheBlueMatt/bitcoin/blob/48efec82f3a18364c019577495914fcebb425e35/src/main.cpp#L470

* For nodes which have sufficient inbound bandwidth, sending a sendcmpct message with the first integer set to 1 to up to 3 peers is RECOMMENDED. If possible, it is RECOMMENDED that those peers be selected based on their past performance in providing blocks quickly (eg the three peers which provided the highest number of the recent N blocks the quickest), allowing nodes to receive blocks which come from those peers in only 0.5*RTT.
* Nodes MUST NOT send such sendcmpct messages to more than three peers, as it encourages wasting outbound bandwidth across the network.

For transactions, looks like it announces the availability of txs to all peers (similar to our tx kernel hash message): https://github.com/bitcoin/bitcoin/blob/67be6d7a177cb14ca8a68b9c6361dfaae9e64f2b/src/net_processing.cpp#L1311

It looks like it's the same for headers as well, but I could be reading the code wrong.

One difference we should note is bitcoin keeps track of each peer's "Inventory", so it doesn't send transactions and headers to nodes that are known to already have them. So based on all of this, my recommendation is:

  • Continue broadcasting headers and peers to all nodes for now.
  • In the future, add the ability to keep track of inventory of each connected peer
  • Limit how many peers we request a compact block from (in cases where a well-connected node receives dozens of header messages at once)

@antiochp
Copy link
Member

One difference we should note is bitcoin keeps track of each peer's "Inventory", so it doesn't send transactions and headers to nodes that are known to already have them.

We do something similar at the peer level, tracking hashes of blocks and transactions that have passed through that particular peer connection. It is implemented in the TrackingAdapter and tracks the 30 most recent hashes (tx and block). I believe this is pretty much entirely undocumented...

@DavidBurkett
Copy link
Contributor

I had no idea. Ok, seems like we're good then.

@j01tz j01tz changed the title [WIP] Improve checking for p2p connection limits Improve checking for p2p connection limits Aug 15, 2019
@j01tz
Copy link
Member Author

j01tz commented Aug 15, 2019

Made a few cleanups based on feedback and removed the [WIP] tag. I think if we want to make more substantial changes beyond here we may want to revert back to the listener check and buffer only for this PR and move the incoming and outgoing connection changes to a separate PR.

@antiochp
Copy link
Member

@j01tz This is good to go right? I'll merge if so.

@j01tz
Copy link
Member Author

j01tz commented Aug 20, 2019

Yes I think so, although haven't heard from @hashmap since he started testing it

@hashmap
Copy link
Contributor

hashmap commented Aug 21, 2019

Works on my machine!

@antiochp antiochp merged commit 24f0a52 into mimblewimble:master Aug 21, 2019
@antiochp
Copy link
Member

antiochp commented Aug 21, 2019

Just merged this.
Checked out latest master and running a node locally on my laptop with default config.
I'm seeing it hit the DNS seeds but I'm not seeing my node actually connect to any peers.

Can someone else see if it works for them (from a fresh node, no local peers db)?

@hashmap @j01tz

I'm wondering if this is an issue with current seeds not accepting connections currently (and not directly related to this PR).


I see the following in my logs -

20190821 13:09:28.527 DEBUG grin_servers::grin::seed - Retrieving seed nodes from dns mainnet.seed.grin.icu
20190821 13:09:28.528 DEBUG grin_servers::grin::seed - Retrieving seed nodes from dns mainnet.seed.713.mw
20190821 13:09:28.530 DEBUG grin_servers::grin::seed - Retrieving seed nodes from dns mainnet.seed.grin.lesceller.com
20190821 13:09:28.531 DEBUG grin_servers::grin::seed - Retrieving seed nodes from dns mainnet.seed.grin.prokapi.com
20190821 13:09:28.534 DEBUG grin_servers::grin::seed - Retrieving seed nodes from dns grinseed.yeastplume.org
20190821 13:09:28.536 DEBUG grin_servers::grin::seed - Retrieved seed addresses: [PeerAddr(V4(35.247.33.125:3414)), PeerAddr(V4(95.216.163.175:3414)), PeerAddr(V4(46.4.91.48:3414)), PeerAddr(V4(167.99.3.23:3414)), PeerAddr(V4(198.245.50.26:3414)), PeerAddr(V4(5.9.152.75:3414)), PeerAddr(V4(145.239.4.27:3414)), PeerAddr(V4(35.224.46.151:3414)), PeerAddr(V4(139.162.168.18:3414)), PeerAddr(V4(173.230.154.243:3414)), PeerAddr(V4(40.71.184.87:3414)), PeerAddr(V4(45.118.135.254:3414)), PeerAddr(V4(157.230.142.11:3414)), PeerAddr(V4(37.187.124.202:3414)), PeerAddr(V4(94.130.229.193:3414)), PeerAddr(V4(67.207.70.122:3414)), PeerAddr(V4(159.65.11.32:3414)), PeerAddr(V4(35.181.35.93:3414)), PeerAddr(V4(157.230.48.217:3414)), PeerAddr(V4(34.207.234.131:3414)), PeerAddr(V4(18.204.166.78:3414)), PeerAddr(V4(5.135.184.54:3414)), PeerAddr(V4(159.69.37.136:3414)), PeerAddr(V4(109.74.202.16:3414))]
...
20190821 13:09:28.622 DEBUG grin_p2p::peer - connect: handshaking with Ok(V4(167.99.3.23:3414))
20190821 13:09:30.542 DEBUG grin_servers::grin::seed - monitor_peers: on 0.0.0.0:3414, 0 connected (0 most_work). all 0 = 0 healthy + 0 banned + 0 defunct
20190821 13:09:34.554 DEBUG grin_servers::grin::seed - monitor_peers: on 0.0.0.0:3414, 0 connected (0 most_work). all 0 = 0 healthy + 0 banned + 0 defunct
20190821 13:09:42.583 DEBUG grin_servers::grin::seed - monitor_peers: on 0.0.0.0:3414, 0 connected (0 most_work). all 0 = 0 healthy + 0 banned + 0 defunct
20190821 13:09:58.625 DEBUG grin_servers::grin::seed - monitor_peers: on 0.0.0.0:3414, 0 connected (0 most_work). all 0 = 0 healthy + 0 banned + 0 defunct
20190821 13:10:18.669 DEBUG grin_servers::grin::seed - monitor_peers: on 0.0.0.0:3414, 0 connected (0 most_work). all 0 = 0 healthy + 0 banned + 0 defunct
20190821 13:10:38.742 DEBUG grin_servers::grin::seed - monitor_peers: on 0.0.0.0:3414, 0 connected (0 most_work). all 0 = 0 healthy + 0 banned + 0 defunct

@antiochp
Copy link
Member

antiochp commented Aug 21, 2019

Current master appears to work fine if the node already has peers.
Previous release appears to work fine from empty peer db.

So it looks like there is an issue here with this PR merged to master where we are not successfully getting peer connections after hitting the initial DNS seed list.

If someone else can reproduce we should investigate and fix (still not entirely convinced this is not a temporary issue with our seeds).

@antiochp
Copy link
Member

Going to hold off opening an issue for this until we can verify this is actually a problem.
And we should probably revert these changes on master and investigate on a branch if we do verify this as an issue.

antiochp added a commit to antiochp/grin that referenced this pull request Aug 21, 2019
@j01tz
Copy link
Member Author

j01tz commented Aug 21, 2019

@antiochp has anything changed with the seeds? I've been testing this for a few days and only started seeing this issue today. Something is definitely wrong though, I can reproduce the same conditions you are seeing now.

@lehnberg
Copy link
Collaborator

can confirm that building with this PR included had me connecting to 0 peers,
and reverting and building without the PR had me connected to 14 peers within a few seconds,
so it feels highly likely that it's something in the PR that is causing this.

@hashmap
Copy link
Contributor

hashmap commented Aug 21, 2019

Hm, this PR still works for me, got peers (I reverted the revert), but much slower than usual, with 10-20 secs delay. It feels like we are slower in opening outbound connections, perhaps waiting for some delay.

@antiochp
Copy link
Member

antiochp commented Aug 21, 2019

See #2993 for proposed fix and explanation (we are slower in opening outbound connections).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants