Guarantee net_plugin processes votes after it has has processed the block the vote was for

[arhag]

Depends on AntelopeIO/leap#2102.

The net_plugin will be responsible for respecting this order per connection. It can leverage the existing cache of block IDs received and sent per connection to know when it is the appropriate time to process the votes per each connection.

We also need to make sure we process the block (add to fork database) before we process the votes for that block, otherwise they will be dropped.

[heifner]

• 3000 vote_message (size 32+1+96+192) is less than 1MB.
  • Limit to 3000 votes per connection, drop incoming votes if > 3000
  • Maintain count per connection for quick check
• Purge votes on LIB (index by block_num)
• Active thread pool for processing
  • We know we don't want to use net threads
  • Chain threads would compete with trx signature verification. Don't want trx sig verification slowing down votes.
  • New thread pool (vote processing)
    • --vote-threads default to 2 4
  • vote_processor in chain library

   atomic<uint32_t> latest_block_num;
   atomic<uint32_t> lib;
   set_lib(lib) { lib = lib; notify_one(); }
   add_vote( v ) { append(v); latest_block_num = std::max(latest_block_num, v.block_num); notify_one(); }

   each thread:
      while (true) {
         _cond.wait([]() {
            if (stopped || !empty) return true;
         }
         if (stopped) break;
         remove_votes_before_lib();
         uint32_t block_num = latest_block_num;
         block_state_ptr bsp = forkdb.get(block_num);
         for each connection by latency {
            if( block_num != latest_block_num) {
               block_num = latest_block_num;
               bsp = forkdb.get(block_num);
            }
            process_votes_for_block(conn_id, bsp);
            block_num = idx.get_latest_block_num();
         }
      }

   process_votes_for_block(conn_id, bsp) {
      for bsp->finalizer_size():
         process_vote(conn_id, bsp);
   }

[heifner]

AntelopeIO/leap#2102 (comment)