Optionally bypass state root verification in reference test worldstate

[garyschulte]

PR description

if the passed block header has a stateroot of Hash.ZERO, bypass bonsai state root verification in bonsai reference test worldstate only.

Fixed Issue(s)

fixes #5934

[github-actions]

• I thought about documentation and added the doc-change-required label to this PR if updates are required.
• I thought about the changelog and included a changelog update if required.
• If my PR includes database changes (e.g. KeyValueSegmentIdentifier) I have thought about compatibility and performed forwards and backwards compatibility tests

[siladu]

The is beyond my current understanding of Bonsai, but please can you explain why doing this doesn't undermine the integrity of the tests? IOW, how do we know this won't result in missing some genuine mismatch errors?

[garyschulte]

The is beyond my current understanding of Bonsai, but please can you explain why doing this doesn't undermine the integrity of the tests? IOW, how do we know this won't result in missing some genuine mismatch errors?

Bonsai defers calculating a state root until we 'persist' the worldstate. When we persist, it is always in the context of a blockheader. As an additional check to ensure consistency, we validate that the stateroot in the blockheader is the same as the one calculated when persisting the worldstate.

There is a bit of a chicken and egg issue with the reference tests however, because the tests are designed with stateroot as an output rather than an input. See Dimitry's comment - when writing tests something has to provide the stateroot as an output as some point.

As far as not undermining the integrity of the tests, in the original pr 5686 we:

• intercept the state test inputs and rewrite the generated block header with the correct state root. This handles tests not written specifically for besu that do not expect state root to be an input.
• add currentStateRoot to ReferenceTestEnv and all of our t8n reference-test-style test cases. This gives us the capability to write reference-test style test cases that do provide state root as an input.

So at least when we run the reference test suites via besu CI we will be validating the state root, and processes that use t8n to run reference tests directly (like retesteth or hive) we will skip state root verification.

With currentStateRoot, we have the ability to validate stateRoot for any of our own tests like the ones for t8n tool. All of our current tests except for the one I added in this PR have a non zero stateroot, but we will have to be mindful in the future when developing reference-test style tests to include currentStateRoot.

[siladu]

Thanks for the clear explanation!

but we will have to be mindful in the future when developing reference-test style tests to include currentStateRoot.

This concerns me a bit. How do you think we can mitigate this? I find this area of besu mysterious because it relates to external tools and not well documented so feel like this could easily slip under the radar in the future.

[siladu]

nit: would be nice to say why we bypass here

[garyschulte]

added a verbose javadoc 👍

[garyschulte]

Thanks for the clear explanation!

but we will have to be mindful in the future when developing reference-test style tests to include currentStateRoot.

This concerns me a bit. How do you think we can mitigate this? I find this area of besu mysterious because it relates to external tools and not well documented so feel like this could easily slip under the radar in the future.

I suspect that the balance of example will be enough when we develop more of our own reference-style tests. More often than not, tests cases borrow a lot from existing examples, and all of our examples (save this one) have a non-zero stateroot.

Willing to be convinced otherwise though.