Good ConnectionLimits for a Filecoin node

[LesnyRumcajs]

I'm looking for a piece of advice on reasonable defaults for ConnectionLimits for a Filecoin node.

Right now, in Forest, they are pretty arbitrary. Most of the time, they work fine, but they're likely on the low side.

let connection_limits = connection_limits::Behaviour::new(
    connection_limits::ConnectionLimits::default()
    .with_max_pending_incoming(Some(10))
    .with_max_pending_outgoing(Some(30))
    .with_max_established_incoming(Some(config.target_peer_count)) // target_peer_count is 75 by default
    .with_max_established_outgoing(Some(config.target_peer_count))
    .with_max_established_per_peer(Some(5)),
);

Not setting limits may expose the node to malicious actors creating peers and ultimately OOMing the node (or eclipse attacks, as noted in the documentation). That is a difference between the rust-libp2p and go-libp2p defaults, but I may need to be corrected.

[mxinden]

Cross-posting response on Slack here:

but they're likely on the low side.

Agreed. Take e.g. the kademlia-exporter project runs on a medium size VM (~30 EUR) handling couple of thousands connections simultaneously.

kademlia-exporter.max-inden.de/

Our latest rust-libp2p based IPFS bootstrap node handles around 30_000 connections concurrently.

ipfs/ipfs-blog#601

I assume that not setting any limits may expose the node to malicious actors creating peers and ultimately OOMing the node?

Correct.

Today you can only set static limits via ConnectionLimits. In the future I would like these limits to be dynamic based on the available memory. Let me know in case you would like to contribute such a mechanism.

See line below on #3078 for dynamic connection limits based on memory.

In the future, with generic connection management, this can be dynamic, e.g. based on your system memory #2824

[LesnyRumcajs]

Thanks!

Today you can only set static limits via ConnectionLimits. In the future I would like these limits to be dynamic based on the available memory. Let me know in case you would like to contribute such a mechanism.

We have some capacity to do it; how much effort would you consider it to be? Is it mostly grabbing the approach from go-libp2p or something from scratch? cc @lemmih @hanabi1224

[thomaseizinger]

The implementation for static connection limits is pretty simple: https://github.com/libp2p/rust-libp2p/blob/master/misc/connection-limits/src/lib.rs

All you we do is check against static counters for new connections. The approach that @mxinden is talking about is making an implementation of NetworkBehaviour that uses a more sophisticated algorithm to make the decision whether it allows a new NetworkBehaviour.

I don't think the implementation itself would be particularly difficult. Instead, what I believe is tricky working out the algorithm. An approach based on available memory needs to know how much memory one connection consumes. That is pretty application specific and I think can't be determined generically. That is one of the reasons we haven't attempted such an implementation.

Also, one would have to agree on a policy on how much memory the application is allowed to consume. All that is available on the system? Again, this highly depends on the application and the production environment (think server vs an app running on desktop computers).

[thomaseizinger]

The bottom line is that I think such a NetworkBehaviour would probably live out-of-tree but I am happy to be convinced otherwise :)

[mxinden]

how much effort would you consider it to be?

@LesnyRumcajs I would roughly estimate around 16h.

We have some capacity to do it;

Wonderful. Note that this would be an unpaid open source contribution.

Is it mostly grabbing the approach from go-libp2p or something from scratch?

From scratch.

---

@thomaseizinger I would experiment with a simpler approach:

• Define a critical memory reached limit. E.g. 70% of the available memory on the system (i.e. VM, or memory namespace in the container).
• When reached, don't allow any additional connections.

I know this will not work for all use-cases. E.g. what if another component within the same process or on the same system eagerly uses memory up to 80% for ephemeral storage e.g. caches. Then libp2p would not allocate a single connection, even though technically one would prefer a connection over some ephemeral storage.

That said, I consider it a good starting point. E.g. it would be ideal for https://github.com/mxinden/rust-libp2p-server/ where the majority of memory usage comes from libp2p and memory usage is proportional to the number of connections. Here a static limit is hard to get right, i.e. estimating how many connections fit the memory of a system is hard to estimate.

@LesnyRumcajs and @thomaseizinger let me know what you think.

[hanabi1224]

@mxinden I've made a draft PR #4281 to implement the proposed approach, please let me know if it's on the right path, if so I will go ahead and make it a formal PR