Update ethereum-tests to v15.0 by jochem-brouwer · Pull Request #3871 · ethereumjs/ethereumjs-monorepo

jochem-brouwer · 2025-02-14T16:42:36Z

Updates ethereum-tests to https://github.com/ethereum/tests/releases/tag/v15.0

codecov · 2025-02-14T16:44:30Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 75.87%. Comparing base (58ac923) to head (73cd8dc).
Report is 1 commits behind head on master.

Additional details and impacted files

Flag	Coverage Δ
block	`76.87% <ø> (ø)`
blockchain	`85.69% <ø> (ø)`
client	`66.28% <ø> (ø)`
common	`90.72% <ø> (ø)`
devp2p	`76.27% <ø> (ø)`
ethash	`81.04% <ø> (ø)`
evm	`71.04% <ø> (+1.38%)`	⬆️
genesis	`99.84% <ø> (ø)`
mpt	`59.72% <ø> (-0.24%)`	⬇️
rlp	`69.70% <ø> (ø)`
statemanager	`70.47% <ø> (ø)`
tx	`80.96% <ø> (ø)`
util	`85.54% <ø> (ø)`
vm	`57.81% <ø> (ø)`
wallet	`83.78% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

ScottyPoi · 2025-02-17T21:15:51Z

There is no longer a test file at .../EOFTests/EIP4200/
However there is a test file that we currently don't access at .../EOFTests/ori/

For whatever reason, I can't find where in the commit history EIP4200 was deleted. Wondering if it was replaced by this ori test? Or if that is something else entirely?

jochem-brouwer · 2025-02-17T21:29:44Z

Hi, thanks for taking over :)

To find the deletion:

cd ./packages/ethereum-tests (ensure you are at v15.0)

git log --diff-filter=D --summary -- ./EOFTests/EIP4200/validInvalid.json

commit ae4791077e8fcf716136e70fe8392f1a1f1495fb
Author: Paweł Bylica <pawel@ethereum.org>
Date:   Wed Dec 18 21:51:27 2024 +0100

    remove migrated EOF RJUMP validation tests

 delete mode 100644 EOFTests/EIP4200/validInvalid.json

ethereum/tests@ae47910

jochem-brouwer · 2025-02-17T21:33:03Z

I'm fairly sure the idea is that all ethereum-tests migrate to execution-spec-tests. For the deletion PR see ethereum/tests#1421

jochem-brouwer · 2025-02-17T21:37:05Z

The tests for specific EIP-4200 should be removed, but the test runner should run all header validation tests in the ./EOFTests folder.

ScottyPoi · 2025-02-17T21:39:38Z

To find the deletion:

cd ./packages/ethereum-tests (ensure you are at v15.0)

git log --diff-filter=D --summary -- ./EOFTests/EIP4200/validInvalid.json

Ah, sweet thanks!

jochem-brouwer · 2025-02-17T23:42:34Z

EVM now runs all container tests in ethereum-tests/EOFTests on CI now. Ready for review & merge 😄 👍

Thanks @ScottyPoi 😄

jochem-brouwer · 2025-02-17T23:43:09Z

  Code0Outputs = 'first code section should have 0x80 (terminating section) outputs',
  MaxStackHeight = `expected maxStackHeight`,
-  MaxStackHeightLimit = `stack height limit of 1024 exceeded: `,
+  MaxStackHeightLimit = `EOF_MaxStackHeightExceeded`,


Wait, is this necessary?

This the exception as it's written in the test. Do we have another way of validating that the correct error was thrown?

Ah right I see. We don't actually validate these errors (that these match the expected errors). For these specific tests we could maybe do so though, but in general the specification does not specify in which order certain errors have to be checked. So for instance a spec could say:

The EVM should check A and B.

For implementation purposes (like readability or optimizations) we could decide to check B before A, and this could thus yield a different error.

Also note that these errors will be thrown by the EVM. If we want to test those errors we should setup an error map which maps the expected error (so for instance EOF_StackUnderflow) to the error string we throw (for instance: stack underflow).

I just saw you added examples, we should likely investigate these (although I think that we throw the correct error on the correct place, but maybe the error string we throw is a bit misleading/faulty)?

Would be great to add :) Will leave this PR open.

Ok, we could do the mapping of EOF_ErrorName to our error messages and leave our errors as they are

This is not so simple actually -- see more examples below

ScottyPoi

In the EOF validation tests, we don't actually compare the _exception value from the test files to the error message, we're just confirming that an error is thrown.

If we do compare the _exception value, we get a bunch of errors. Mostly this is just semantic, but a few of them seem suspicious, like maybe we are throwing the wrong error.

ScottyPoi · 2025-02-18T00:23:59Z

For example -- we give the same error on these two tests, but the expected errors are different according to the test files

jochem-brouwer · 2025-02-18T00:35:28Z

 // Rename this test dir to the location of EOF header tests
 // To test, use `npx vitest run ./scripts/eof-header-validation.spec.ts
-const testDir = path.resolve('../ethereum-tests/EOFStuff/fixtures/eof_tests')
+const testDir = path.resolve('../ethereum-tests/EOFTests')


Ah right this file should be disabled for browser tests

ScottyPoi · 2025-02-18T00:42:18Z

Here's another example -- where the EOF_Error is more specific than the error that we throw:

ScottyPoi · 2025-02-18T00:49:41Z

Then, here, the other way around. We give two different errors for the same expected exception:

ScottyPoi

LGTM. Will address Error message testing in future PR

jochem-brouwer added PR state: needs review type: tests package: monorepo type: test all hardforks This special label enables VM state and blockchain tests for all hardforks on the respective PR. labels Feb 14, 2025

jochem-brouwer marked this pull request as draft February 17, 2025 23:35

jochem-brouwer marked this pull request as ready for review February 17, 2025 23:41

jochem-brouwer commented Feb 17, 2025

View reviewed changes

ScottyPoi reviewed Feb 18, 2025

View reviewed changes

jochem-brouwer commented Feb 18, 2025

View reviewed changes

jochem-brouwer and others added 8 commits February 19, 2025 09:59

Update eth-tests to v15.0

ba9b874

Use Osaka testdata in test

e706027

Update error messages to expected

36712ff

remove eip-4200 test

792ae4a

evm: ensure ethereum-tests/EOFTests container tests are being ran

8ec0a7c

evm: exclude eof header validation from browser tests

4d37543

revert change error text

6df27dd

revert direct exception compare

ca2190f

ScottyPoi force-pushed the tests-v15 branch from a0b5d4e to ca2190f Compare February 19, 2025 16:59

Merge branch 'master' into tests-v15

73cd8dc

ScottyPoi approved these changes Feb 21, 2025

View reviewed changes

jochem-brouwer merged commit cfc5309 into master Feb 21, 2025

acolytec3 deleted the tests-v15 branch February 21, 2025 00:55

ScottyPoi mentioned this pull request Feb 21, 2025

ethereum-tests error messages #3874

Open

2 tasks

Uh oh!

Conversation

jochem-brouwer commented Feb 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Feb 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ScottyPoi commented Feb 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jochem-brouwer commented Feb 17, 2025

Uh oh!

jochem-brouwer commented Feb 17, 2025

Uh oh!

jochem-brouwer commented Feb 17, 2025

Uh oh!

ScottyPoi commented Feb 17, 2025

Uh oh!

jochem-brouwer commented Feb 17, 2025

Uh oh!

jochem-brouwer Feb 17, 2025

Choose a reason for hiding this comment

Uh oh!

ScottyPoi Feb 18, 2025

Choose a reason for hiding this comment

Uh oh!

jochem-brouwer Feb 18, 2025

Choose a reason for hiding this comment

Uh oh!

ScottyPoi Feb 18, 2025

Choose a reason for hiding this comment

Uh oh!

ScottyPoi Feb 18, 2025

Choose a reason for hiding this comment

Uh oh!

ScottyPoi left a comment

Choose a reason for hiding this comment

Uh oh!

ScottyPoi commented Feb 18, 2025

Uh oh!

jochem-brouwer Feb 18, 2025

Choose a reason for hiding this comment

Uh oh!

ScottyPoi commented Feb 18, 2025

Uh oh!

ScottyPoi commented Feb 18, 2025

Uh oh!

ScottyPoi left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jochem-brouwer commented Feb 14, 2025 •

edited

Loading

codecov Bot commented Feb 14, 2025 •

edited

Loading

ScottyPoi commented Feb 17, 2025 •

edited

Loading