engine: return -38003 for FCUv2 payloadAttributes mismatch

[Muzry]

This PR updates engine_forkchoiceUpdatedV2 to return -38003: Invalid payload attributes when the wrong payloadAttributes version is used.

In particular, FCUv2 payload-attribute version mismatches such as:

• missing withdrawals at or after Shanghai
• unexpected withdrawals before Shanghai

should be treated as Invalid payload attributes, not Invalid params.

Why

This change aligns the client with the latest Engine API spec update in:

• engine: use -38003 for wrong payloadAttributes version in forkchoiceUpdatedV2 ethereum/execution-apis#761

It also follows the implementation discussion and prior client-side change in:

• eth/catalyst: return -32602 for FCU V2 wrong payloadAttributes version ethereum/go-ethereum#33918

The spec was clarified so that FCUv2 now behaves consistently with newer forkchoiceUpdated versions for payloadAttributes structure/version
mismatches.

What changed

• Updated FCUv2 payload attributes validation to return -38003 for payloadAttributes version mismatches.
• Added/updated regression coverage for the affected FCUv2 cases.

Hive impact

This fixes the Hive engine-withdrawals failure caused by returning the wrong error code for FCUv2 payloadAttributes mismatches.

Relevant Hive failure:

• https://hive.ethpandaops.io/#/test/generic/1773131670-9be21c1a6d01e061c51d963e43511cc1

After this change, the client returns the expected error code for the affected FCUv2 cases.

If my understanding or interpretation of the spec change is incorrect, please let me know and I can adjust the implementation accordingly.

[macfarla]

@Muzry you'll need to add signoff to your commits to satisfy the DCO check. I need to check what's going on with hive tests as I'm not seeing any engine-withdrawal failures when running on main locally. what specific hive test is your PR addressing?

[Muzry]

@macfarla Thanks, I’ve added signoff to my commit and re-ran the full engine-withdrawals suite locally against hive main using the Besu client from hive.

Hive commit used for the run: 1fc72ff41ba0903b46c4e210ead80f55a63e1718

commands: ./hive --sim ethereum/engine --sim.limit 'engine-withdrawals/' --client besu --sim.parallelism 50 --docker.output --docker.pull

The run produced 34 test cases: 20 passed and 14 failed.

The main failure cluster is exactly what this PR addresses: for engine_forkchoiceUpdatedV2, hive expects -38003, while the Besu version currently used by hive still returns -32602.

So this PR is addressing the engine-withdrawals FCUv2 error-code failures, rather than a withdrawal execution failure in engine_newPayload.

You can also check: https://hive.ethpandaops.io/#/test/generic/1773131670-9be21c1a6d01e061c51d963e43511cc1. The recent test runs there are also failing.

[macfarla]

running hive tests locally,

• 10 engine-withdrawals failures running on main branch
• failures look like: FAIL (Withdrawals Fork On Genesis): Expected error code on EngineForkchoiceUpdatedV2: want=-38003, got=-32602
• 0 engine-withdrawals failures running on this PR

[macfarla]

@Muzry this does fix a bunch of hive tests which is great. There are some unit tests (maybe only 1) that need updating with this change as well - if you can run the unit tests locally that will make the feedback loop faster - eg

> Task :ethereum:api:test

EngineForkchoiceUpdatedV1Test > shouldReturnInvalidIfPayloadTimestampNotGreaterThanHead() FAILED
    org.opentest4j.AssertionFailedError at AbstractEngineForkchoiceUpdatedTest.java:788

[Muzry]

@macfarla Thanks, sorry for the delay. I've updated the unit tests for this change. Could you take another look?

[macfarla]

@macfarla Thanks, sorry for the delay. I've updated the unit tests for this change. Could you take another look?

@Muzry can you amend your latest commits to include signoff - to fix the DCO check

[Muzry]

@macfarla I’ve updated the commit to include signoff. Could you please take another look?

[macfarla]

LGTM

[macfarla]

@Muzry there's an acceptance test failing as well https://github.com/besu-eth/besu/actions/runs/23820570721/job/69433410658?pr=10014 - you can run these tests locally, it will massively speed up the feedback loop for getting your PR in.

[Muzry]

@macfarla I updated the Paris acceptance test fixture for engine_forkchoiceUpdatedV1 to match the current V1 behavior (-32602 / Invalid withdrawals) and re-ran the acceptance test locally. It is now passing. Could you please take another look?

[macfarla]

@macfarla I updated the Paris acceptance test fixture for engine_forkchoiceUpdatedV1 to match the current V1 behavior (-32602 / Invalid withdrawals) and re-ran the acceptance test locally. It is now passing. Could you please take another look?

@macfarla I updated the Paris acceptance test fixture for engine_forkchoiceUpdatedV1 to match the current V1 behavior (-32602 / Invalid withdrawals) and re-ran the acceptance test locally. It is now passing. Could you please take another look?

ExecutionEngineShanghaiAcceptanceTest is failing

[macfarla]

@Muzry run this locally, it takes ~ 20 min, then you can be sure all the tests are passing
./gradlew build

[Muzry]

@macfarla I updated the related unit tests and acceptance fixtures for this change, and re-ran the relevant tests locally. The FCU V1/V2 unit tests are passing, and the Paris / Shanghai acceptance cases related to this change are also now passing locally. Could you please take another look?