refactor(tests),feat(test-forks): add filter_combinations pytest marker to avoid runtime skips#2543
Merged
marioevz merged 6 commits intoMar 24, 2026
Conversation
danceratopz
added
C-refactor
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## forks/amsterdam #2543 +/- ##
================================================
Coverage 86.35% 86.35%
================================================
Files 599 599
Lines 36904 36904
Branches 3771 3771
================================================
Hits 31868 31868
Misses 4485 4485
Partials 551 551
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
marioevz
reviewed
Mar 23, 2026
Member
marioevz
left a comment
marioevz
left a comment
There was a problem hiding this comment.
Very nice new feature. Just a couple of comments that would be nice to have. Thanks!
| if not marker.args: | ||
| continue | ||
| predicate = marker.args[0] | ||
| if not predicate(**params): |
Member
There was a problem hiding this comment.
Add an assertion that predicate is callable, otherwise raise the appropriate error with the test information.
Member
Author
There was a problem hiding this comment.
Good idea, thanks! Done!
| items_to_remove = [] | ||
| deselected: List[pytest.Item] = [] | ||
| # function name -> [reason, total, deselected_count] | ||
| filter_stats: Dict[str, List[Any]] = {} |
Member
There was a problem hiding this comment.
Suggested change
| filter_stats: Dict[str, List[Any]] = {} | |
| filter_stats: Dict[str, Tuple[str, int, int]] = {} |
It's a bit nicer than a list.
Add a collection-time marker that deselects parametrized test cases based on cross-parameter predicates, replacing runtime `pytest.skip()` calls for invalid parameter combinations. - Run predicates in `pytest_collection_modifyitems` after all parametrize expansion is complete, so both covariant and regular parameters are visible uniformly. - Report deselected items via `pytest_deselected` for accurate terminal counts. - Print a per-function summary with reasons at default verbosity; expand to individual node IDs at `-vv`. - Fail with `USAGE_ERROR` if a predicate eliminates every parametrization of a test function.
- Replace runtime `pytest.skip()` with a `filter_combinations` marker that deselects non-CREATE opcodes paired with `transaction_create=True` at collection time.
- Replace runtime `pytest.skip()` with `filter_combinations` markers that deselect call opcodes without a `value` kwarg when paired with nonzero value, in both `test_set_code_to_self_caller` and `test_set_code_call_set_code`.
- Verify cross-parameter filtering with covariant and regular parametrize axes. - Verify filtering with only regular parametrize axes. - Verify no-op predicate keeps all combinations. - Verify stacked markers apply AND logic. - Verify predicate that empties a function exits with USAGE_ERROR.
- Assert that the first argument to `filter_combinations` is callable, exiting with `USAGE_ERROR` and the test node ID if not.
- Change filter_stats from Dict[str, List[Any]] to Dict[str, Tuple[str, int, int]] for stronger typing.
danceratopz
force-pushed
the
delete-skip-in-covariant-decorator
branch
from
10fe818 to
a30ca5f
Compare
5 tasks
4 tasks
danceratopz
added a commit
to danceratopz/execution-specs
that referenced
this pull request
May 13, 2026
…cell in `test_varying_calldata_costs` `STORAGE_CLEAR` refund is zero on revert, so `gas_used_pre_refund == gas_used_post_refund` and the `(post, pre)` interval that `DATA_FLOOR_BETWEEN_TX_GAS_BEFORE_AND_AFTER` requires is empty. Replace the runtime `pytest.skip()` with the `filter_combinations` marker introduced in ethereum#2543, which runs in `pytest_collection_modifyitems` and can express predicates across covariant-marker-injected and inline parametrize axes uniformly. Keeps `with_all_refund_types()` so new refund types added to the fork in future automatically participate, preserves the original three-decorator parametrize stack, and produces byte-identical fixture IDs and content vs the pre-refactor branch (verified via `diff` on `varying_calldata_costs.json` keys and bodies). The two infeasible items per fork are now deselected at collection time rather than skipped at runtime.
marioevz
pushed a commit
that referenced
this pull request
May 13, 2026
…cell in `test_varying_calldata_costs` (#2852) `STORAGE_CLEAR` refund is zero on revert, so `gas_used_pre_refund == gas_used_post_refund` and the `(post, pre)` interval that `DATA_FLOOR_BETWEEN_TX_GAS_BEFORE_AND_AFTER` requires is empty. Replace the runtime `pytest.skip()` with the `filter_combinations` marker introduced in #2543, which runs in `pytest_collection_modifyitems` and can express predicates across covariant-marker-injected and inline parametrize axes uniformly. Keeps `with_all_refund_types()` so new refund types added to the fork in future automatically participate, preserves the original three-decorator parametrize stack, and produces byte-identical fixture IDs and content vs the pre-refactor branch (verified via `diff` on `varying_calldata_costs.json` keys and bodies). The two infeasible items per fork are now deselected at collection time rather than skipped at runtime.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
What?
This PR removes runtime skips that were required in tests that used the
CovariantDecoratorsystem provided by theforksplugin due to invalid generated combinations.Why?
The aim is to reduce unnecessary noise in fill/pytest results.
Details
The approach is to add a
filter_combinationspytest marker that deselects parametrized test cases at collection time based on cross-parameter predicates. This replaces runtimepytest.skip()calls that exist because the covariant decorator system can only filter individual parameter values viaselector, not combinations across covariant and regular parametrize axes.reasonkwarg shown in collection output so operators can see why combinations were dropped.pytest_collection_modifyitemsafter all parametrize expansion is complete, so it sees both covariant and regular parameters uniformly. Deselected items are reported viapytest_deselectedfor accurate terminal counts.-q.USAGE_ERRORif a predicate eliminates every parametrization of a test function, guarding against overly broad predicates silently dropping all test cases.Refactored Test Modules
tests/frontier/create/test_create_suicide_during_init.pytest_create_suicide_during_transaction_createtests/prague/eip7702_set_code_tx/test_set_code_txs.pytest_set_code_to_self_callertests/prague/eip7702_set_code_tx/test_set_code_txs.pytest_set_code_call_set_codeThe 12 remaining skips in the frontier create test are unrelated (pre-existing validity marker skips, not from
pytest.skip()).Verification
I validated fixtures before and after this refactor using
uv run hasher compare. There was no change in the fixture set.Before
After
✅ Checklist
toxchecks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:uvx tox -e statictype(scope):.Cute Animal Picture