Improve transmission propagation

[vicsn]

Motivation

Currently, it can take up to a minute on mainnet for a transaction to be confirmed or rejected. One major part of a transaction lifecycle, is how fast it is included into a batch proposal.

A core feature of Narwhal is that transmissions may be included in multiple proposals. The current setup of snarkOS has an overreliance on safety, because clients and validators propagate valid seen transmissions to all of their peers.

However, there's currently no good reason to play it this safe. This PR proposes to reduce the propagation to "only" 3 other validators. Transmissions will still eventually reach all validators, but they are more likely to be certified by the other validators try to include it in their own proposal.

When running a load test with 6000 transmissions on reference hardware and no propagation at all, sometimes all transmissions would land immediately, but typically only around 5750 would land immediately, which is an indication some older certificates get left behind and we need at least some propagation.

Note for more context this section from the Narwhal paper:

Further, Narwhal relies on clients to re-submit a transac-
tion if it is not sequenced in time, due to the leader being
faulty. An expensive alternative is to require clients to sub-
mit a transaction to 𝑓 +1 authorities, but this would divide
the bandwidth of authorities by 𝑂(𝑛). This is too high a price
to pay, since a client submitting a transaction to a fixed 𝑘
number of authorities has a probability of including a cor-
rect one that grows very quickly in 𝑘, at the cost of only
𝑂(1)overhead. Notably, Mir-BFT uses an interesting trans-
action de-duplication technique based on hashing which we
believe is directly applicable to Narwhal in case such a fea-
ture is needed. Ultimately, we relegate this choice to system
designers using Narwhal.

Test Plan

• Deploy a devnet with txcannons, observe adequate transaction inclusion time

[vicsn]

New graphs:

[vicsn]

Closing in favour of: #3962