Batch state blocks signatures verification #956

SergiySW · 2018-07-07T13:54:39Z

No description provided.

clemahieu · 2018-07-09T22:47:43Z

Very nice! I'll take a look in a bit, if anyone else can too that'd be good.

PlasmaPower · 2018-07-11T18:36:08Z

rai/lib/numbers.cpp

+		messages_pointers[i] = messages[i].bytes.data ();
+		messages_lengths[i] = sizeof (messages[i].bytes);
+		public_keys_pointers[i] = public_keys[i].bytes.data ();
+		signatures_pointers[i] = signatures[i].bytes.data ();


I'd prefer to prevent an additional copy here. Could we change the function signature to require C style arrays of arrays of bytes?

Sure we can

While you're at it, where it's called from, instead of copying the e.g. signature bytes we could just make an array of pointers to them.

PlasmaPower · 2018-07-11T18:45:43Z

This PR is exciting! I'm worried about messing with signature verification, as that's a very critical piece of code, but this looks good to me.

Does this speed up bootstrapping?

SergiySW · 2018-07-11T18:59:24Z

While it's increasing verification speed up to 40-50% for large amounts of state blocks (I forget to add old test --debug_verify_profile_batch, thx for reminder), currently only 14.4% of blocks are state. So it won't increase current bootstrap speed significantly

slact · 2018-07-25T04:25:48Z

I can confirm a max 40-60% speedup from my batch verification experiments. Increasing the batch size beyond the default (64 i think) doesn't help much either.

clemahieu · 2018-07-27T22:01:39Z

What about this process:

In addition to block_processor::blocks there is block_processor::state_blocks
block_processor::add puts state blocks in to the state block queue
block_processor::process_blocks first runs signature checks on everything in the state_blocks deque. Anything that fails a signature check is dropped. Anything that succeeds is put in to the blocks deque. The blocks deque is then processed as normal
Signature checking in removed for state blocks in the ledger_processor
A test is added to prove that incorrectly signed state blocks are rejected.

SergiySW · 2018-08-01T11:09:36Z

Seems valid process

SergiySW · 2018-08-01T16:33:52Z

Still I think that signature verification shouldn't be removed from ledger processor by default because other functions can use it except block_processor::process_receive_many

argakiig · 2018-08-02T03:48:33Z

MSVC conforms to C90 standards and does not support variable length arrays (C99) which is what is causing the appveyor failures

Until variable length arrays support or better VLA emulation

State blocks with epoch links != epoch blocks Validate all blocks with epoch link by common rules

PlasmaPower · 2018-10-12T16:02:11Z

rai/node/node.cpp

 {
 	if (!rai::work_validate (block_a->root (), block_a->block_work ()))
 	{
 		std::lock_guard<std::mutex> lock (mutex);
 		if (blocks_hashes.find (block_a->hash ()) == blocks_hashes.end ())
 		{
-			blocks.push_back (std::make_pair (block_a, origination));
-			blocks_hashes.insert (block_a->hash ());
+			if (!unchecked && block_a->type () == rai::block_type::state && block_a->link () != node.ledger.epoch_link)


I believe the unchecked parameter is named incorrectly. Shouldn't it be called checked? If it's true, it bypasses signature validation.

Can we construct a test on that?

Well, probably require renaming. Parameter for state blocks from unchecked table. As all state blocks in unchecked already was validated

I see. Is the performance improvement worth the risk?

I guess yes. Rare risks are signature corruption on disk in unchecked table or bit flips. Same risk for state blocks table & usual signature check.
But this can be moved to separate PR

In terms of risk I was thinking we might accidentally put unvalidated blocks in the unchecked table.

Though I think I've brought this up before. Either way should be fine.

As far as I see unchecked_put used only with previous_gap or source_gap, that status goes after signature checks for state blocks

Let’s go with the safer option. Can we make a test that has a block with a bad signature in unchecked and make sure it isn’t processed?

Unchecked thing removed. I guess we can return to this after 17.0, now better polish what we have for new release

PlasmaPower · 2018-10-12T16:02:54Z

Other than that LGTM

for future different PR to reduce CPU load

… blocks

rkeene · 2018-10-12T17:58:37Z

rai/lib/blocks.cpp

@@ -1440,6 +1460,11 @@ rai::block_hash rai::receive_block::root () const
 	return hashables.previous;
 }

+rai::block_hash rai::receive_block::link () const
+{
+	return 0;


Should we assert(false) here (or just not implement this virtual function for non-state blocks) ? The receive block probably has the equivalent of a link

That's similar to source (), previous () & some other functions returning 0 for some block types. We can just not implement such functions when they return 0 I guess to save some lines in code

Is this a case of using release_assert since we shouldn’t be looking for the link of non-state blocks?

Either release_assert() or not declaring the virtual methods for these block types which should generate a fatal exception

rai/node/node.cpp

rkeene · 2018-10-12T20:44:27Z

rai/node/node.cpp

@@ -1051,7 +1102,7 @@ void rai::block_processor::process_receive_many (std::unique_lock<std::mutex> &
 					node.ledger.rollback (transaction, successor->hash ());
 				}
 			}
-			auto process_result (process_receive_one (transaction, block.first, block.second));
+			auto process_result (process_receive_one (transaction, block.first, block.second, !force));


Using !force here is not clear that it's directly related to state blocks... can we do something more clear ?

Forced state blocks are not validated in verify_state_blocks () function
Because of that we should set set validated_state_block as "false" for forced blocks (!force)

But !force will evaluate to true when force is false which can happen for things other than state blocks (or verified blocks ?)

Yes, but only state blocks vefirication can be skipped by flag in ledger.processor

rkeene · 2018-10-13T04:36:23Z

Awesome work !

rkeene

Thanks !

Missed in nanocurrency#956

Missed in #956

SergiySW added 10 commits July 7, 2018 00:56

Batch signatures validation

f2a09de

Batch state blocks verification

43a1eb3

Add valid_signature option to ledger_processor

3cf25a0

Add valid_signature to process_receive_one

1e3ee88

Simplify validate_blocks

8816f91

Test for batch block check

798a9cc

Batch block_processor for state blocks

d551bc5

Extra vector to prevent possible junk validation DDoS on node

ee9e144

Typo

fbf8194

Clang formatting

9448c3d

PlasmaPower reviewed Jul 11, 2018

View reviewed changes

SergiySW added 5 commits July 11, 2018 22:43

Merge remote-tracking branch 'upstream/master' into batch_signatures_2

d50d535

Test debug_verify_profile_batch

db7b505

Merge remote-tracking branch 'upstream/master' into batch_signatures_2

2da990c

Exception for epoch blocks

b37da3c

Fix mutex.try_lock () issue

9764412

SergiySW added 3 commits August 1, 2018 14:17

Merge remote-tracking branch 'upstream/master' into batch_signatures_2

952e6f5

Simplify batch signature verification processor

187a61f

Validate epoch blocks

7278510

Fix

fa56cae

SergiySW added 2 commits August 2, 2018 15:26

MSVC workaround

002665a

Until variable length arrays support or better VLA emulation

block_processor_signatures test

cb345e1

SergiySW added 10 commits September 8, 2018 20:43

Merge remote-tracking branch 'upstream/master' into batch_signatures_2

82b52d0

Merge remote-tracking branch 'upstream/master' into batch_signatures_2

b5fad43

Merge remote-tracking branch 'upstream/master' into batch_signatures_2

b1b6ade

Merge remote-tracking branch 'upstream/master' into batch_signatures_2

13a73c9

Remove epoch blocks from batch verification

6118b53

State blocks with epoch links != epoch blocks Validate all blocks with epoch link by common rules

Typo

62a7fde

Merge remote-tracking branch 'upstream/master' into batch_signatures_2

1fd2e35

Merge remote-tracking branch 'upstream/master' into batch_signatures_2

4785938

Merge remote-tracking branch 'upstream/master' into batch_signatures_2

7a7b194

Merge remote-tracking branch 'upstream/master' into batch_signatures_2

2e83193

rkeene added the major This item indicates the need for or supplies a major or notable change label Oct 12, 2018

rkeene requested review from clemahieu and rkeene October 12, 2018 15:27

PlasmaPower reviewed Oct 12, 2018

View reviewed changes

SergiySW added 4 commits October 12, 2018 19:32

Merge remote-tracking branch 'upstream/master' into batch_signatures_2

4bad6d3

Remove unchecked from block_processor::add

0550cc5

for future different PR to reduce CPU load

Extent block_processor_signatures test with forced blocks & unchecked…

c025a5e

… blocks

Move force check

797d560

rkeene reviewed Oct 12, 2018

View reviewed changes

rai/node/node.cpp Show resolved Hide resolved

rkeene reviewed Oct 12, 2018

View reviewed changes

SergiySW added 2 commits October 13, 2018 00:03

Extra info

7468154

More clear validated_state_block value

d86047c

rkeene approved these changes Oct 14, 2018

View reviewed changes

rkeene merged commit f438f46 into nanocurrency:master Oct 14, 2018

SergiySW added a commit to SergiySW/raiblocks that referenced this pull request Jan 8, 2019

Restore blocks_hashes.insert

ea35cc9

Missed in nanocurrency#956

SergiySW mentioned this pull request Jan 8, 2019

Restore blocks_hashes.insert #1569

Merged

zhyatt pushed a commit that referenced this pull request Jan 10, 2019

Restore blocks_hashes.insert (#1569)

eaa8155

Missed in #956

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Batch state blocks signatures verification #956

Batch state blocks signatures verification #956

SergiySW commented Jul 7, 2018

clemahieu commented Jul 9, 2018

PlasmaPower Jul 11, 2018

SergiySW Jul 11, 2018

PlasmaPower Jul 11, 2018

PlasmaPower commented Jul 11, 2018

SergiySW commented Jul 11, 2018

slact commented Jul 25, 2018

clemahieu commented Jul 27, 2018

SergiySW commented Aug 1, 2018

SergiySW commented Aug 1, 2018

argakiig commented Aug 2, 2018

PlasmaPower Oct 12, 2018

clemahieu Oct 12, 2018

SergiySW Oct 12, 2018

PlasmaPower Oct 12, 2018

SergiySW Oct 12, 2018

PlasmaPower Oct 12, 2018

PlasmaPower Oct 12, 2018

SergiySW Oct 12, 2018

clemahieu Oct 12, 2018

SergiySW Oct 12, 2018

PlasmaPower commented Oct 12, 2018

rkeene Oct 12, 2018

SergiySW Oct 12, 2018

clemahieu Oct 12, 2018

rkeene Oct 12, 2018

rkeene Oct 12, 2018

SergiySW Oct 12, 2018

rkeene Oct 12, 2018

SergiySW Oct 12, 2018

rkeene commented Oct 13, 2018

rkeene left a comment

Batch state blocks signatures verification #956

Batch state blocks signatures verification #956

Conversation

SergiySW commented Jul 7, 2018

clemahieu commented Jul 9, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PlasmaPower commented Jul 11, 2018

SergiySW commented Jul 11, 2018

slact commented Jul 25, 2018

clemahieu commented Jul 27, 2018

SergiySW commented Aug 1, 2018

SergiySW commented Aug 1, 2018

argakiig commented Aug 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PlasmaPower commented Oct 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rkeene commented Oct 13, 2018

rkeene left a comment

Choose a reason for hiding this comment