p2p: Add 2 outbound block-relay-only connections

[sdaftuar]

Transaction relay is optimized for a combination of redundancy/robustness as well as bandwidth minimization -- as a result transaction relay leaks information that adversaries can use to infer the network topology.

Network topology is better kept private for (at least) two reasons:

(a) Knowledge of the network graph can make it easier to find the source IP of a given transaction.

(b) Knowledge of the network graph could be used to split a target node or nodes from the honest network (eg by knowing which peers to attack in order to achieve a network split).

We can eliminate the risks of (b) by separating block relay from transaction relay; inferring network connectivity from the relay of blocks/block headers is much more expensive for an adversary.

After this commit, bitcoind will make 2 additional outbound connections that are only used for block relay. (In the future, we might consider rotating our transaction-relay peers to help limit the effects of (a).)

[gmaxwell]

I agree with the concept. I wish we could announce our own addresses to these peers, but I think doing so results in a topology leak. :( The reason I'd like this is that if an attacker eclipses a node except for its blocks only links it will eventually eclipse its blocks-only links too through control of the addrman state.

[sdaftuar]

This approach was motivated by the TxProbe paper. Fundamentally, transaction relay is going to leak information about our node's peers (as long as we care about not wasting bandwidth). Rather than combat transaction-relay based inferences, which I think is a lost cause in the long-run even if there are improvements we can make in the short-term, I figure we can directly address why we care about keeping the graph private.

Some considerations I had while working on this:

• Is 2 the right number of blocks-only connections? From a robustness standpoint, more connections like this are better; however there's a tradeoff with resource utilization. Two resources that occurred to me to worry about were the memory used by each CNode object (hence the refactor commits that allow the transaction-relay data structures to not be instantiated for the blocks-only peers), and the number of connection slots on the network (I figure going from 8 to 10 is not that bad?). I also did some simulations to verify that a random graph of 10000 peers would be fully connected if each peer makes 2 outbound connections to random other peers, which I thought was a good justification for 2 being a useful number of outbounds to add.

• How should addrman interactions work on these blocks-only connections? My approach attempts to leave addrman unchanged as a result of these connections, neither processing addr messages nor sending any, to avoid leaking information via addr messages that we relay to other peers. But I have not given a lot of thought to what kind of attacks are possible using addrman, nor what the consequences are to not updating addrman with information from our blocks-only peers.

[gmaxwell]

I also did some simulations to verify that a random graph of 10000 peers would be fully connected if each peer makes 2 outbound connections to random other peers,

Can you run these simulations assuming x% of the 10,000 peers are black holes (don't connect the graph), for various values of x%?

2 might be fine for now, but ultimately I'd like to optimize memory usage for both inbound and outbound blocksonly links, then at least double the inbound connection limit for blocks only links, and make 8 blocks only links. (I suspect running that simulation assuming reasonable number of black hole peers will show that 2 is not really enough).

[instagibbs]

I figure we can directly address why we care about keeping the graph private

To the uninformed p2p reader this reason is?

[DrahtBot]

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #16762 (Rust-based Backup over-REST block downloader by TheBlueMatt)
• #16702 (p2p: supplying and using asmap to improve IP bucketing in addrman by naumenkogs)
• #16698 ([WIP] Mempool: rework rebroadcast logic to improve privacy by amitiuttarwar)
• #16682 ([WIP] p2p: Disconnect peer that send us tx INVs when we opted out of tx relay by jnewbery)
• #16507 (feefilter: Compute the absolute fee rather than stored rate by instagibbs)
• #15502 (Speed up initial connection to p2p network by ajtowns)
• #14033 (p2p: Drop CADDR_TIME_VERSION checks now that MIN_PEER_PROTO_VERSION is greater by Empact)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

[gmaxwell]

To the uninformed p2p reader this reason is?

(b) Knowledge of the network graph could be used to split a target node or
nodes from the honest network (eg by knowing which peers to attack in order to
achieve a network split).

[sdaftuar]

Can you run these simulations assuming x% of the 10,000 peers are black holes (don't connect the graph), for various values of x%?

This table shows my simulation results for various black hole rates, with various values of how many outbound connections we make (k=2 is what I've implemented in this PR). This is on a 10,000 node graph with no non-listening nodes. Each table entry shows the success count (where success is defined as "the subgraph of non-black-hole nodes is connected") out of 1000 simulations.

black hole rate	k=2	k=4	k=6	k=8
0	1000	1000	1000	1000
5%	30	998	1000	1000
10%	0	976	1000	1000
15%		867	999	1000
20%		598	994	1000
25%		229	978	1000
30%		33	928	998
35%		2	812	992

2 might be fine for now, but ultimately I'd like to optimize memory usage for both inbound and outbound blocksonly links, then at least double the inbound connection limit for blocks only links, and make 8 blocks only links.

This is roughly what I have in mind as well. I think we need a p2p protocol change to allow negotiating blocks-only links (so that the peer on the other side of our connection knows that the inbound peer won't turn transaction relay on later), so I figured we could start small for now.

[practicalswift]

@sdaftuar Thanks for sharing the results of the analysis! Would you be willing to share the simulation scripts?

[sdaftuar]

@practicalswift My simulation code is unreadable, and also may have bugs, so I'd prefer that anyone interested in this would independently corroborate my results from scratch.

[naumenkogs]

separating block relay from transaction relay;

This is indeed a good idea. I also see it as a small step towards peer rotation, I think this suggestion makes it easier to reason about rotation.

I'm not sure if this is strongly related to the data you provided, but I'm also curious how many nodes have to be attacked (suspended?) to split network in halves etc.

[luke-jr]

Wouldn't this break all the benefits of compact blocks? (Ignoring that with the current implementation, block relay would likely still occur on the normal connections.)

[sdaftuar]

@luke-jr I don't think so. The idea is that nodes are still expected to do transaction relay with other peers, and my belief is that transaction propagation is good enough that compact block relay works just fine on these blocksonly links. I believe my testing of this branch confirms that as well, but I'd welcome additional testers to corroborate those results in case I'm overlooking something.

[practicalswift]

Concept ACK: I like the idea

[promag]

Maybe commits up to 7f2ce0a could be in a separate pull request.

[promag]

c36939c

Could explain why the new mutex? This commit isn't just refactor.

[sdaftuar]

I think one issue is that I wasn't sure how to do the GUARDED_BY annotations with a variable that didn't live in the class. But also I thought it might make sense to explicitly lock things in this class with its own mutex, as that seems like a clearer design for making this code modular.

The reason I included this refactor commit at all was in preparation for being able to optionally not instantiate this data structure, if there was concern about minimizing resource utilization as much as possible for these additional blocksonly connections. Since I ended up not doing that (I believe I concluded that it probably wasn't all that much additional memory?), I could also drop this commit instead.

[promag]

Thanks for the explanation, I'll review again with that in mind.

[jonasschnelli]

Concept ACK.
I'm still trying to understand this more detailed: does that mean that with 2 block-only connection and an assumed 5% black-hole-nodes rate, we only have a 3% chance to succeed? Since we relay, I guess there is no chance we could identify (and thus disconnect) black-hole peers?

[Empact]

nit: m_addr_relay_peer could be private

[sdaftuar]

Meh, going to leave this as-is.

[dongcarl]

Trying to understand the logic here...

According to the commit description, bitcoind will make 2 additional outbound connections, but with this logic, if connOptions.nMaxOutbound <= 6 then there will be no additional outbound connections: https://www.wolframalpha.com/input/?i=min(2,+max(x-6,+0))

From elsewhere in this commit, it seems like we consider the accounting for nMaxOutboundBlocksOnly as separate from that of nMaxOutbound, so perhaps just

 connOptions.nMaxOutboundBlocksOnly = MAX_BLOCKS_ONLY_CONNECTIONS;

would do?

[EthanHeilman]

This - 6 is a little confusing and requires that you look up the default value of connOptions.nMaxOutbound to understand what this is doing. Also echoing @dongcarl comments that this doesn't work well when connOptions.nMaxOutbound <= 6.

Given that this fails for particular values and you only want 2 block only outbound why not change this to:

connOptions.nMaxOutboundBlocksOnly = MAX_BLOCKS_ONLY_CONNECTIONS

[sdaftuar]

Perhaps the accounting logic should be improved to be clearer, but I believe the behavior here is at least (roughly) correct. The way this works is that if a user sets -maxconnections very low, that will be a hard limit on the number of inbound, outbound and outbound-blocks-only connections. Before this PR, it would take the user setting their max connections value to less than 8 to change the number of outbound connections we make (we first reduce all the inbound we take and only then reduce the outbound).

This PR adds new blocks-only connections and preferences those connections over regular outbound connections (whenever we make an outbound connection, we check to see if we are below our blocks-only target, and if so we initiate the connection as blocks-only). As a result, I thought it didn't make sense to have any blocks-only links if we have too few regular outbound connections, because I don't think it makes sense to have blocks-only links at the expense of transaction relay.

In particular, the problem with the suggestion of just setting nMaxOutboundBlocksOnly to 2 is that if the user passes -maxconnections=2, then they'll only have 2 blocksonly peers and won't receive any transactions.

So I believe the way I implemented things here is that we try to reserve 6 connections as regular outbound, and then if we add additional connections we will have those be blocks-only until we get to 2, and then any further connections are regular outbounds up to 8, and then finally any additional ones are used for feelers and inbounds.

Of course we could change this to be more conservative and only allocate blocks-only connections if we have all 8 regular outbounds. However I don't think we should be less conservative and risk having fewer regular outbounds.

[TheBlueMatt]

I think just do std::min(MAX_BLOCKS_ONLY_CONNECTIONS, std::max(connOptions.nMaxConnections - MAX_OUTBOUND_CONNECTIONS, 0)).

[sdaftuar]

@TheBlueMatt I pushed a commit that changes the behavior so that the first 8 outbounds are reserved to be regular full-relay peers, and only then do we allocate up to 2 blocksonly peers.

[sdaftuar]

Rebased and addressed review comments.

[sdaftuar]

I'm still trying to understand this more detailed: does that mean that with 2 block-only connection and an assumed 5% black-hole-nodes rate, we only have a 3% chance to succeed? Since we relay, I guess there is no chance we could identify (and thus disconnect) black-hole peers?

@jonasschnelli With 2 blocks-only connections, and if we assume 5% of nodes are useless black holes, then my simulation suggests we have a 3% chance of the network graph being connected by only the blocks-only connections. For additional robustness, we will want to add more blocks-only connections in the future -- but for resource utilization reasons I don't want to do that until we add better support at the p2p protocol layer for establishing these connections.

Note that we are still going to relay blocks on the existing 8 outbound connections, so this PR is purely additive robustness, at the cost of a little more resource utilization. We now will have 10 outbound peers, rather than 8, which are able to provide blocks to us. (We just aren't going to do tx relay on the two additional connections.)

One issue I'd like to flag for reviewers is how these new blocksonly peers interact with outbound peer disconnection (introduced in #11560 and #11490). I believe I've implemented this so that the new blocksonly peers will not be subject to other forms of disconnection, because non-tx-relay outbounds are excluded from being outbound disconnection candidates. However, it's worth verifying the implementation is correct and worth thinking about whether this behavior is the best choice.

[JeremyRubin]

I think that this should be correct code in case someone wants to set it up to tabulate against Suhas's simulation.

Edit: I've rewritten it to be much much better. Scipy is so cool!

 import random
 import copy
 import scipy
 import scipy.sparse
 import scipy.linalg
 import numpy as np
 
 N = 1000
 def biased_coin(bias):
     return random.random() <= bias
 def make_graph(n_outbound, p_blackhole, n_total = N):
     graph = np.zeros((n_total, n_total))
     for i in range(n_total):
         if not biased_coin(p_blackhole):
             for j in range(n_outbound):
                 graph[i][j] = 1
             # rejection sample no self-edges
             while True:
                 np.random.shuffle(graph[i])
                 if not graph[i][i]:
                     break
     return graph
 
def is_connected(graph):
      return scipy.sparse.csgraph.connected_components(graph, directed=True, return_labels=False) == 1

 print(sum(is_connected(make_graph(8, 0.05, 1000)) for x in range(100)))

My results were, for 100 samples, 92. This suggests Suhas's simulation was not correct.

edit: Nevermind, Suhas's simulation was on n=10,000. will re-run

[JeremyRubin]

Improved the code for the above sim, in case anyone's following by email-only.

[JeremyRubin]

Deep apologies for the noise; I missed Suhas's earlier note about the connectivity of the non black hole nodes, not the total connectivity. The below should work for that.

import random
 import copy
 import scipy
 import scipy.sparse
 import scipy.linalg
 import numpy as np
 
 N = 1000
 def biased_coin(bias):
     return random.random() <= bias
 def make_graph(n_outbound, p_blackhole, n_total = N):
     graph = np.zeros((n_total, n_total))
     count = 0
     for i in range(n_total):
         if not biased_coin(p_blackhole):
             count += 1
             for j in range(n_outbound):
                 graph[i][j] = 1
             # rejection sample no self-edges
             while True:
                 np.random.shuffle(graph[i])
                 if not graph[i][i]:
                     break
     return (graph,count)
 
 def is_connected(graph, count):
     first_non_blackhole = None
     for (i,row) in enumerate(graph):
         if any(r != 0 for r in row):
             first_non_blackhole = i
             break
     if first_non_blackhole is None:
         return False
     n, labels = scipy.sparse.csgraph.connected_components(graph, directed=True, return_labels=True)
     label = labels[first_non_blackhole]
     size = sum(l == label  for l in labels)
     return size >= count
 
 
 
 print(sum(is_connected(*make_graph(8,0.05,10000)) for _ in range(10)))

[taurus228]

src/init.cpp