`execute` Benchmark Mode

[marioevz]

After the execute live session we came to the conclusion that there needs to be a separation of modes depending on whether we are running execute to test consensus or to benchmark.

Current issues:

• Execute sends every transaction to the chain regardless of whether the transaction is part of the benchmark test's setup or workload, so it's very difficult/impossible to clearly separate blocks that contain setup and workloads of these tests.
• There's no reporting whatsoever of when (which block) these transactions landed in the blockchain, and therefore it's difficult to construct a dashboard that highlights the execution of a single block when we don't really know its contents (test-wise).
• Currently we use state_test or blockchain_test specs to also construct benchmark tests, but this leads up to having to infer what each transaction of the test is trying to do, we should have benchmark_test and benchmark_state_test that contains extra fields to clearly separate and label each transaction (accounts can also be setup-only or workload-only labeled, e.g. of a setup-only account could be an account whose only purpose is to send transactions that deploy setup contracts).

Potential path forward:

• Have benchmark_test and benchmark_state_test have fields that allow the test writer to clearly state which transactions are part of the setup and which transactions are part of the workload.
• Have a new execute remote --benchmark or execute benchmark that only can run tests that are explicitly using the correct benchmark specs (benchmark_test and benchmark_state_test), and disallow execute remote or execute hive from running these tests.
• Allow execute benchmark to explicitly setup the remote blockchain to deploy contracts and accounts that the workload is going to need, along with checking that any stubbed contracts are already in place and correct.
  • We could use a two-phase approach (similar to how we fill engine-x tests) where the first phase runs all benchmark tests but in setup mode so it only deploys the contracts, and then the second phase actually runs the tests.
  • The setup phase could even be deterministic so we check all whether the contracts are already there before deploying any of them. This could be achieved by having a setup-seed so we can run the workload phase many times over and over with a single setup and just passing a --setup-seed flag or similar.

cc @jochem-brouwer @LouisTsai-Csie @kamilchodola @marcindsobczak

[kamilchodola]

Execute sends every transaction to the chain regardless of whether the transaction is part of the benchmark test's setup or workload, so it's very difficult/impossible to clearly separate blocks that contain setup and workloads of these tests.

But each transaction have complex id with phase - so you can tell which one is which?

[kamilchodola]

NethermindEth/gas-benchmarks#57

Here using exactly this field to distinguish and wait for more txs to convert them into block.

If i dont get any more transaction with "setup" phase for more than a 0,5 second i build a block.
If by any reason i'd keep on getting more setup ones - I'd construct other one (did not notced such case yet but it would meam a complex scenario and should only haplen at setup, cant be like this on testing)

[marioevz]

Execute sends every transaction to the chain regardless of whether the transaction is part of the benchmark test's setup or workload, so it's very difficult/impossible to clearly separate blocks that contain setup and workloads of these tests.

But each transaction have complex id with phase - so you can tell which one is which?

Yes but there's some tests which extend the setup phase further without execute knowing really what the purpose of each transaction is.

E.g. this in this test

execution-spec-tests/tests/benchmark/test_worst_stateful_opcodes.py

Lines 50 to 116 in b3624b5

	def test_worst_address_state_cold(
	blockchain_test: BlockchainTestFiller,
	pre: Alloc,
	fork: Fork,
	opcode: Op,
	absent_accounts: bool,
	env: Environment,
	gas_benchmark_value: int,
	):
	"""Test running a block with as many stateful opcodes accessing cold accounts."""
	attack_gas_limit = gas_benchmark_value
	gas_costs = fork.gas_costs()
	intrinsic_gas_cost_calc = fork.transaction_intrinsic_cost_calculator()
	# For calculation robustness, the calculation below ignores "glue" opcodes like PUSH and POP.
	# It should be considered a worst-case number of accounts, and a few of them might not be
	# targeted before the attacking transaction runs out of gas.
	num_target_accounts = (
	attack_gas_limit - intrinsic_gas_cost_calc()
	) // gas_costs.G_COLD_ACCOUNT_ACCESS
	blocks = []
	post = {}
	# Setup
	# The target addresses are going to be constructed (in the case of absent=False) and called
	# as addr_offset + i, where i is the index of the account. This is to avoid
	# collisions with the addresses indirectly created by the testing framework.
	addr_offset = int.from_bytes(pre.fund_eoa(amount=0))
	if not absent_accounts:
	factory_code = Op.PUSH4(num_target_accounts) + While(
	body=Op.POP(Op.CALL(address=Op.ADD(addr_offset, Op.DUP6), value=10)),
	condition=Op.PUSH1(1) + Op.SWAP1 + Op.SUB + Op.DUP1 + Op.ISZERO + Op.ISZERO,
	)
	factory_address = pre.deploy_contract(code=factory_code, balance=10**18)
	setup_tx = Transaction(
	to=factory_address,
	gas_limit=env.gas_limit,
	sender=pre.fund_eoa(),
	)
	blocks.append(Block(txs=[setup_tx]))
	for i in range(num_target_accounts):
	addr = Address(i + addr_offset + 1)
	post[addr] = Account(balance=10)
	# Execution
	op_code = Op.PUSH4(num_target_accounts) + While(
	body=Op.POP(opcode(Op.ADD(addr_offset, Op.DUP1))),
	condition=Op.PUSH1(1) + Op.SWAP1 + Op.SUB + Op.DUP1 + Op.ISZERO + Op.ISZERO,
	)
	op_address = pre.deploy_contract(code=op_code)
	op_tx = Transaction(
	to=op_address,
	gas_limit=attack_gas_limit,
	sender=pre.fund_eoa(),
	)
	blocks.append(Block(txs=[op_tx]))
	blockchain_test(
	pre=pre,
	post=post,
	blocks=blocks,
	exclude_full_post_state_in_output=True,
	)

where we have setup_tx but execute will see this as a normal transaction that is part of the test.