Block validation flow v2 + Batch (serial) sig verification #2250

mratsim · 2021-01-20T18:50:33Z

Overview

This reworks the block validation flow to introduce SigVerified blocks besides the existing blocks and trusted blocks.

This mention but does not introduce TransitionVerified blocks as without generics or static enum that would lead to lots of duplication.

The flow matrix is the following

	Signature unchecked	Signature verified
State transition unchecked	- UntrustedBeaconBlock	- SigVerifiedBeaconBlock
	- UntrustedIndexedAttestation	- TrustedIndexedAttestation
	- UntrustedAttestation	- TrustedAttestation
----------------------------	-------------------------------	-----------------------------
State transition verified	- TransitionVerifiedBeaconBlock	- TrustedSignedBeaconBlock
	- UntrustedIndexedAttestation	- TrustedIndexedAttestation
	- UntrustedAttestation	- TrustedAttestation

Notice that at the moment TrustedAttestation is imprecise and actually means SigVerifiedAttestation.

The current code checks cryptography then state meaning SigVerifiedAttestation is a state that only exists between those lines of code:

nimbus-eth2/beacon_chain/block_pools/clearance.nim

Lines 199 to 205 in ea8fe25

    
             if not batchVerify(sigs, quarantine.sigVerifCache): 
        
               return err((ValidationResult.Reject, Invalid)) 
        
           static: doAssert sizeof(SignedBeaconBlock) == sizeof(SigVerifiedSignedBeaconBlock) 
        
           let sigVerifBlock = cast[SigVerifiedSignedBeaconBlock](signedBlock) 
        
           let (valRes, blockErr) = addRawBlockCheckStateTransition(dag, quarantine, sigVerifBlock, cache)

Implementation

This reimplements the branch https://github.com/status-im/nimbus-eth2/tree/block-validation-flow which is stuck due to issues mentioned in #2219. In particular, besides workarounds needed to avoid compiler ICE, the branch is stuck trying to deal with generic types in macro nim-lang/RFCs#44

Switching to multithreading

Switching to multithreading requires changing this line

nimbus-eth2/beacon_chain/spec/signatures_batch.nim

Lines 285 to 286 in ea8fe25

    
           # TODO: For now only enable serial batch verification 
        
           return batchVerifySerial(cache, sigs, secureRandomBytes)

to batchVerify and compile with -d:openmp.

beacon_chain/spec/signatures_batch.nim

arnetheduck · 2021-01-21T12:00:05Z

docs/block_validation_flow.md

+
+### TransitionVerifiedBeaconBlocks
+
+A block that passes the state transition checks and can be successfully applied to the beacon chain is considered `TransitionVerified`.


Hm, the state transition checks all signatures (unless it's a sigverified), so how can this type of block happen? I think in all cases we want to run signature checks before or as part of state transition, but never after (as running state transition on blocks that haven't passed sigver is fishy)

It's up to us. In the current state it can't happen because I do crypto verif first hence why TransitionVerified isn't materialized in the code.

That said currently before this PR, within a state_transition epoch/block we did all the checks first and crypto verification last because it is much more expensive. This is an option that we can keep in mind, depending on which worst case scenario we want: either faulty or malicious parties use valid crypto and fake blocks and in that case we want to verify state transition first, or they use valid blocks and invalid crypto and in that case we want to verify crypto first.

beacon_chain/spec/signatures_batch.nim

arnetheduck · 2021-01-21T12:20:30Z

beacon_chain/block_pools/clearance.nim

+      return err((ValidationResult.Reject, Invalid))
+
+  static: doAssert sizeof(SignedBeaconBlock) == sizeof(SigVerifiedSignedBeaconBlock)
+  let sigVerifBlock = cast[SigVerifiedSignedBeaconBlock](signedBlock)


this certainly looks fishy... I'm surprised it even works with the List types and Trusted vs normal bls values

That's AFAIK the simplest way to convert. I changed TrustedSig so that it's a "distinct" type from normal Signatures (just an object with the exact same fields) and has the same byte representation.

if it's a distinct type, SigVerifiedBeaconBlock(signedBlock) should work - without the cast - I'm a bit afraid the cast might do bad things to the gc

ok, looked it up, it's not a distinct type, just a similar type.. so when assigning something to something else, the gc needs new refcounts for seqs - I'm not sure that will happen with cast - check the generated C code

so, looking a bit more at this, the codegen does some extremely inefficient things for this cast - it will generate a temporary union type with the two types as members, then genericAssign twice, once for each type, in and out of the union.. it's "correct:ish" but extremely wasteful.

here's something better:

var sigVerifBlock: SigVerifiedSignedBeaconBlock assign(sigVerifiedBlock, cast[ptr SigVerifiedSignedBeaconBlock](signedBlock)[])

using stew/assign - this "should" work as well.

In that case I think casting through pointer in the function call will produce no copy (because verified blocks are never used where they are produced, they are passed to another consumer).

I guess, yeah - the only thing I'm thinking about is strict pointer aliasing - the nim impl is inefficient but correct (with stew/assign it would be decently efficient as well but...)..

hm, not sure what the right thing here is tbh - if I look at vtune, copying blocks is one of the highest cpu consumers apart from crypto because of how extremely slow genericAssign is.. we could do the pointer thing which represents a small risk of strict aliasing issues, but we're not looping here or anything such, so perhaps it's fine.

New codegen

#line 235 "/home/beta/Programming/Status/nim-beacon-chain/beacon_chain/block_pools/clearance.nim" T29_ = addRawBlockCheckStateTransition__ln6s2fp4XrmWohuHu9bOV8A(dag, quarantine, (&(*((tyObject_SigVerifiedSignedBeaconBlock__FeS1XMPF6R63Ue3RhifLCg*) (signedBlock)))), (&cache)); valRes = T29_.Field0; blockErr = T29_.Field1; // [...] #line 242 "/home/beta/Programming/Status/nim-beacon-chain/beacon_chain/block_pools/clearance.nim" addResolvedBlock__kpy9cmxed3iYehH784xNCUA(dag, quarantine, (&(*(*dag)).clearanceState), (&(*((tyObject_TrustedSignedBeaconBlock__hdN6xn3RN9aAfpVMBC35UEw*) (signedBlock)))), parent, (&cache), onBlockAdded);

beacon_chain/block_pools/clearance.nim

beacon_chain/spec/datatypes.nim

…ent)

mratsim · 2021-01-21T18:35:39Z

#2255 was needed to unbreak unstable.

I think all concerns are addressed for this PR.

arnetheduck · 2021-01-25T12:10:20Z

beacon_chain/beacon_chain_db.nim

@@ -304,12 +304,16 @@ func toBeaconBlockSummary(v: SomeBeaconBlock): BeaconBlockSummary =
    parent_root: v.parent_root,
  )

+# TODO: we should only store TrustedSignedBeaconBlock in the DB.


with the new asTrusted where memory layout is the same for all block kinds, this should actually be a simple change

…rypto and state transition

… br_hmac_drbg_context in stdlib_assertions.nim.c)

…ent)

…e pubkey validations is now done first in signatures_batch)

…loaded

mratsim · 2021-01-25T17:21:22Z

The PR has been rebased and should be good to merge.

Following #2259 I modified signatures_batch.nim. Previously since signatures and pubkeys were verified at deserialization it could assume that the keys were correct and avoid if foo.isNone(): return false else: foo.unsafeGet() dance. Now signatures_batch.nim assumes that anything can go rogue.

* bump nim-blscurve * Outline the block validation flow * introduce the SigVerified types, pass the tests * Split clearance/quarantine to prepare for batch crypto verif * Add a batch signature collector * Make clearance use SigVerified block and split verification between crypto and state transition * Always use signedBeaconBlock for the onBlockAdded callback * RANDAO signing_root is the epoch instead of the full block * Support skipping BLS for testing * Fix compilation of the validator client * Try to fix strange errors MacOS and Jenkins (Clang, unknown type name br_hmac_drbg_context in stdlib_assertions.nim.c) * address #2250 (comment) * address #2250 (comment) * onBlockAdded callback should use TrustedSignedBeaconBlock #2250 (comment) * address #2250 (comment) * Use the application RNG: #2250 (comment) * Improve codegen of conversion zero-cost) * Quick fixes with loadWithCache after #2259 (TODO: graceful error since pubkey validations is now done first in signatures_batch) * Graceful handle rogue pubkeys and signatures now that those are lazy-loaded

arnetheduck reviewed Jan 21, 2021

View reviewed changes

zah reviewed Jan 21, 2021

View reviewed changes

beacon_chain/spec/datatypes.nim Show resolved Hide resolved

mratsim added a commit that referenced this pull request Jan 21, 2021

address #2250 (comment)

1271081

mratsim added a commit that referenced this pull request Jan 21, 2021

address #2250 (comment)

8d36785

mratsim added a commit that referenced this pull request Jan 21, 2021

onBlockAdded callback should use TrustedSignedBeaconBlock #2250 (comm…

37c7fbb

…ent)

mratsim added a commit that referenced this pull request Jan 21, 2021

address #2250 (comment)

c346d13

mratsim added a commit that referenced this pull request Jan 21, 2021

Use the application RNG: #2250 (comment)

8611e02

mratsim closed this Jan 21, 2021

mratsim reopened this Jan 21, 2021

mratsim added the merge ready label Jan 21, 2021

arnetheduck reviewed Jan 25, 2021

View reviewed changes

arnetheduck approved these changes Jan 25, 2021

View reviewed changes

mratsim added 17 commits January 25, 2021 16:14

bump nim-blscurve

12073cd

Outline the block validation flow

37d00aa

introduce the SigVerified types, pass the tests

4c7891b

Split clearance/quarantine to prepare for batch crypto verif

02e2d55

Add a batch signature collector

208ec77

Make clearance use SigVerified block and split verification between c…

25679b7

…rypto and state transition

Always use signedBeaconBlock for the onBlockAdded callback

ba3ccbc

RANDAO signing_root is the epoch instead of the full block

0af30dc

Support skipping BLS for testing

7b75044

Fix compilation of the validator client

f7f51d2

Try to fix strange errors MacOS and Jenkins (Clang, unknown type name…

dc6d857

… br_hmac_drbg_context in stdlib_assertions.nim.c)

address #2250 (comment)

97c3be6

address #2250 (comment)

b5a2835

onBlockAdded callback should use TrustedSignedBeaconBlock #2250 (comm…

5391baf

…ent)

address #2250 (comment)

66bd2f5

Use the application RNG: #2250 (comment)

2ed532e

Improve codegen of conversion zero-cost)

2ba3795

Quick fixes with loadWithCache after #2259 (TODO: graceful error sinc…

2378ccc

…e pubkey validations is now done first in signatures_batch)

mratsim force-pushed the block-validation-flow-v2 branch from 681a0a8 to 2378ccc Compare January 25, 2021 16:16

Graceful handle rogue pubkeys and signatures now that those are lazy-…

11fa3e0

…loaded

zah merged commit 70a0365 into unstable Jan 25, 2021

zah deleted the block-validation-flow-v2 branch January 25, 2021 18:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Block validation flow v2 + Batch (serial) sig verification #2250

Block validation flow v2 + Batch (serial) sig verification #2250

mratsim commented Jan 20, 2021

arnetheduck Jan 21, 2021

mratsim Jan 21, 2021

arnetheduck Jan 21, 2021

mratsim Jan 21, 2021

arnetheduck Jan 21, 2021

arnetheduck Jan 21, 2021

arnetheduck Jan 21, 2021

mratsim Jan 21, 2021

arnetheduck Jan 21, 2021

mratsim Jan 21, 2021

mratsim commented Jan 21, 2021

arnetheduck Jan 25, 2021

mratsim commented Jan 25, 2021

	if not batchVerify(sigs, quarantine.sigVerifCache):
	return err((ValidationResult.Reject, Invalid))

	static: doAssert sizeof(SignedBeaconBlock) == sizeof(SigVerifiedSignedBeaconBlock)
	let sigVerifBlock = cast[SigVerifiedSignedBeaconBlock](signedBlock)

	let (valRes, blockErr) = addRawBlockCheckStateTransition(dag, quarantine, sigVerifBlock, cache)

	# TODO: For now only enable serial batch verification
	return batchVerifySerial(cache, sigs, secureRandomBytes)


		### TransitionVerifiedBeaconBlocks

		A block that passes the state transition checks and can be successfully applied to the beacon chain is considered `TransitionVerified`.

Block validation flow v2 + Batch (serial) sig verification #2250

Block validation flow v2 + Batch (serial) sig verification #2250

Conversation

mratsim commented Jan 20, 2021

Overview

Implementation

Switching to multithreading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mratsim commented Jan 21, 2021

Choose a reason for hiding this comment

mratsim commented Jan 25, 2021