[WIP] Rebase peras-staging onto main #1748
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Update to use newest cardano-crypto-class with unsound pure KES implementation - Use mlocked KES - Add KES agent connectivity - Rebase cleanup - Handle drop-key messages from KES Agent - Provide KESAgentClientTrace to BlockForging - Revert change to MockCrypto and require DSIGN only when running the KES agent - Bump kes-agent SRP to remove SerDoc dependency
…Source to make it unusable
…then releases the resource)
…locates resources required for the BlockForging
Use the new queryPoolState from ledger. Update golden files to the latest LedgerState representation. The LedgerState changed because the representation of PState has changed. Tx Size now is a Word32 instead of an Integer.
- Update the SRP. - Use the new `queryPoolState` query from ledger for `GetPoolState`. - Update all golden files for `LedgerState` since in `cardano-ledger` the representation of `PState` has changed. - `Tx Size` is changed from `Integer` to `Word32`.
This PR supercedes #1487 includes the following squashed commit messages: - Update to use newest cardano-crypto-class with unsound pure KES implementation - Use mlocked KES - Add KES agent connectivity - Rebase cleanup - Handle drop-key messages from KES Agent - Provide KESAgentClientTrace to BlockForging - Revert change to MockCrypto and require DSIGN only when running the KES agent - Bump kes-agent SRP to remove SerDoc dependency # Description Please include a meaningful description of the PR and link the relevant issues this PR might resolve. Also note that: - New code should be properly tested (even if it does not add new features). - The fix for a regression should include a test that reproduces said regression.
This tiny PR documents in a bit more details how the arguments to `db-synthesizer` program look like. In particular, it wasn't obvious what the "bulk credentials" structure was and I got it only after looking at the example. HTH.
- Removes the ledger SRP from cabal.project - Removes the plutus-related constraints from cabal.project - Removes allow-newer for QuickCheck from cabal.project - Use explicit imports rather than `hiding` to support QuickCheck-2.15 without `-wdodgy-imports` firing
This was removed in #1535 but we forgot to update the spec.
Conflicts: ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Block/SupportsPeras.hs ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/MiniProtocol/ObjectDiffusion/ObjectPool/PerasCert.hs ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Storage/ChainDB/API.hs ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Storage/PerasCertDB/API.hs ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Storage/PerasCertDB/Impl.hs ouroboros-consensus/test/consensus-test/Test/Consensus/MiniProtocol/ObjectDiffusion/PerasCert/Smoke.hs ouroboros-consensus/test/storage-test/Test/Ouroboros/Storage/PerasCertDB/Model.hs ouroboros-consensus/test/storage-test/Test/Ouroboros/Storage/PerasCertDB/StateMachine.hs
- Add `PerasRoundLength` - introduce the `PerasEnabled` datatype to track values are only used when Peras is enabled - HFC: translate between Peras rounds and slots Conflicts: ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Block/SupportsPeras.hs
In the presence of EBBs, block numbers can be very misleading, eg the tip block number of a shorter chain can have a higher block number than that of a longer one. To avoid test failures due to this peculiar behavior, we do not look at block numbers at all for the `WeightedSelectView`, and instead measure the length of the fragment (relative to its anchor). Concretely, this change fixes test failures in the ChainDB q-s-m test when testing with eg `k=5` instead of `k=2` (as different candidates can then actually contain *multiple* EBBs). When EBBs are not used (which has been the case on mainnet for >5 years), this change has no semantic impact.
Since the Peras boost per certificate will likely become a protocol parameter, we proactively avoid exposing the current hardcoded value, replacing it with an instantiation of (currently trivial) the PerasCfg builder. In the special cases where it's interesting to vary the boost dynamically (ChainDB q-s-m), validated Peras certs now contain randomly generated boost weights.
Extends the ChainDB model with generator state to support carrying gap blocks in state machine tests. This increases the chances of generating and adding (possibly out-of-order) branching sequences of blocks. This, in turn increases the chances of observing the event where the chain selection logic switches from a longer to a shorter (but heavier) chain containing a boosted block. Conflicts: ouroboros-consensus/test/storage-test/Test/Ouroboros/Storage/ChainDB/StateMachine.hs
This commit increases the generation frequencies of both the
'genAddBlock' and 'genAddPerasCert' constructions to help producing
denser chains of blocks. This way, some of the events that were
harder to trigger (especially TagSwitchedToShorterChain) are much
more common now:
* Before:
Tags (5784 in total):
39.83% TagGetIsValidJust
29.72% TagChainSelReprocessKeptSelection
27.92% TagGetIsValidNothing
2.42% TagChainSelReprocessChangedSelection
0.10% TagSwitchedToShorterChain
* After:
Tags (5202 in total):
38.66% TagGetIsValidJust
27.87% TagChainSelReprocessKeptSelection
26.43% TagGetIsValidNothing
5.71% TagChainSelReprocessChangedSelection
1.33% TagSwitchedToShorterChain
After analysing the effect of varying the security parameter (`k`) of the ChainDB state machine tests (currently hardcoded with 2), we have observed a tension between: 1) generating enough tests exercising the new Peras behavior where the chain selection mechanism switches to a shorter but heavier chain (cert boost is derived from k and must be large enough to overcome the weight of a longer chain), and 2) generating enough tests exercising the ImmutableDB logic (the chain must have at least k blocks) Here are some empirical results: k -> P(switch to shorter chain), P(generate a chain with >= k blocks) k=2 -> ~1.3%, ~40% k=3 -> ~1.9%, ~20% k=4 -> ~2.4%, ~9% k=5 -> ~2.5%, ~3% k=10 -> ~3%, ~0.05% We believe that the sweet spot between both desiderata appears to be around `k=2` and `k=4`. This commit introduces a random generator for `k` using a geometric distribution to bias the randomly generated `k`s to be relatively small, while still allowing larger ones to appear from time to time. Under the current parameters, roughly 75% of the tests use `k<=4`; ``` Security Parameter (k) (10000 in total): 50.82% 2 23.83% 3 12.62% 4 6.69% 5 3.08% 6 1.54% 7 0.74% 8 0.37% 9 0.16% 10 0.06% 11 0.05% 12 0.02% 13 0.01% 14 0.01% 17 ``` Yielding the following distributions for 1) and 2), respectively: ``` Tags (5161 in total): 39.35% TagGetIsValidJust 29.22% TagChainSelReprocessKeptSelection 25.91% TagGetIsValidNothing 3.88% TagChainSelReprocessChangedSelection 1.65% TagSwitchedToShorterChain <- HERE ``` ``` Chain length >= k (10000 in total): 73.25% False 26.75% True <- HERE ```
Brings in cardano-base and propagates a set of `CardanoFeatureFlag`s from the top-level `RunNodeArgs` down to the `NodeKernelArgs`. This is currently needed by an upcoming PR to the GSM to distinguish whether having an established PerasCertDiffusion connection with a given peer is necessary or not when trying to decide if such peer is idling. Conflicts: cabal.project flake.lock ouroboros-consensus-diffusion/src/ouroboros-consensus-diffusion/Ouroboros/Consensus/Node.hs
Bumps the external ouroboros-network source-repository-package to the updated peras-staging/pr-5202, which incorporates the changes from: IntersectMBO/ouroboros-network#5202 In addition, it tweak call sites of `nodeToNodeProtocols` to match its updated signature, passing down the enabled feature flags. Conflicts: cabal.project ouroboros-consensus-diffusion/src/ouroboros-consensus-diffusion/Ouroboros/Consensus/Network/NodeToNode.hs
Conflicts: ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/MiniProtocol/ObjectDiffusion/Inbound.hs
Conflicts: ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/MiniProtocol/ObjectDiffusion/ObjectPool/PerasCert.hs ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Storage/ChainDB/API.hs ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Storage/PerasCertDB/API.hs ouroboros-consensus/test/storage-test/Test/Ouroboros/Storage/PerasCertDB/Model.hs
This reverts commit 591445e.
Conflicts: ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/MiniProtocol/ObjectDiffusion/ObjectPool/PerasCert.hs ouroboros-consensus/test/consensus-test/Test/Consensus/MiniProtocol/ObjectDiffusion/PerasCert/Smoke.hs
With the currently selected generation sizes, we hit the different
voting rules in a somewhat decent proportion, even against randomly
generated functions as part of the PerasVotingView interface.
ouroboros-consensus
Peras
Peras voting rules
isPerasVotingAllowed: OK (1.55s)
+++ OK, passed 10000 tests.
Actual result (10000 in total):
60.29% NoVoteReason(VR-1A or VR-2A)
20.90% NoVoteReason(VR-1A or VR-2B)
9.60% VoteReason(VR-2A and VR-2B)
4.93% VoteReason(VR-1A and VR-1B)
2.67% NoVoteReason(VR-1B or VR-2A)
1.61% NoVoteReason(VR-1B or VR-2B)
Should vote according to model (10000 in total):
85.47% False
14.53% True
VR-(1A|1B|2A|2B) (10000 in total):
21.30% (False,True,False,False)
21.24% (False,False,False,False)
10.63% (False,False,True,False)
10.27% (False,True,True,False)
8.99% (False,False,False,True)
8.76% (False,True,False,True)
4.67% (False,True,True,True)
4.26% (False,False,True,True)
1.93% (True,False,False,False)
1.77% (True,True,False,False)
1.71% (True,True,True,False)
1.61% (True,False,True,False)
0.75% (True,True,True,True)
0.74% (True,False,False,True)
0.70% (True,True,False,True)
0.67% (True,False,True,True)
VR-1A (10000 in total):
90.12% False
9.88% True
VR-1B (10000 in total):
50.07% False
49.93% True
VR-2A (10000 in total):
65.43% False
34.57% True
VR-2B (10000 in total):
70.46% False
29.54% True
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR rebases the suffix of
peras-stagingcontaining code that's not yet merged ontomain. Whenever possible, I reworded the commits with conflicts to include the conflicting files.I made sure to keep the codebase buildable after solving every set of conflicts. I will later pin some non-trivial changes I had to make as part of this.