Add ability to run framework tests separately by leolara · Pull Request #4538 · ethereum/consensus-specs

leolara · 2025-08-27T05:49:11Z

These are a set of small improvements suggested by @danceratopz

Change a test to pytest that was done in unittest by mistake
Change make file so spec tests are not run with infrastructure tests

jtraglia · 2025-08-27T12:53:17Z

+# Run test framework tests.
+#
+# To run a specific test, append k=<test>, eg:
+#   make test k=test_verify_kzg_proof
+# To run tests with a specific bls library, append bls=<bls>, eg:
+#   make test bls=arkworks
+test_infra: MAYBE_TEST := $(if $(k),-k=$(k))
+# Disable parallelism which running a specific test.
+# Parallelism makes debugging difficult (print doesn't work).
+test_infra: MAYBE_PARALLEL := $(if $(k),,-n auto)
+test_infra: pyspec
+	@mkdir -p $(TEST_REPORT_DIR)
+	@$(PYTHON_VENV) -m pytest \
+		$(MAYBE_PARALLEL) \
+		--capture=no \
+		$(MAYBE_TEST) \
+		$(CURDIR)/tests/infra


I want to keep these tests bundled together. make test should run all tests.

Also, after deleting this, can we fix the typo on line 101:

Disable parallelism which running a specific test.

Should be:

Disable parallelism when running a specific test.

I want to keep these tests bundled together. make test should run all tests.

I think going forward, as more framework tests get added, there would be value in allowing separate execution of these tests:

It creates a clearer separation of intent that makes it immediately clear to a new developer that these "framework tests" have an entirely different purpose.

It allows a developer to iterate faster on framework changes that are covered by fw tests (typically fw tests will run faster than the actual tests).

It allows better reporting, locally, particularly in CI; consensus-specs could have two workflows that would run make tests and make test_infra.

For now, until we improve the CI as in 2., we could add this new test_infra target as a dependency to the test target. This already enables 1. and 2. We can improve the CI in a follow-up PR for 3. and potentially remove the test_infra dep from test.

I understand, but there are better ways of doing this. This solution introduces unnecessary complexity for everyone: (1) we'll need to update documentation telling contributors that they need to run both commands; (2) we need to update the CI checks to run both commands; (3) we'll need to update the release action to run these; (4) we need to update the Makefile help section.

There are two other solutions which I would prefer:

Ensure infrastructure tests start with def test_infra_* and use make test k=test_infra.

Similar to the current style, make the list of files dynamic and use make test infra-only=true.

I added a TODO list.

Additionally to what @danceratopz said I don't think the normal spec test developer will not need to run the infra tests, so another advantage of having them separated.

No that's not what I'm saying. Seriously, let's not add a separate make command for infrastructure tests. Please.

I pushed ec3edc0 which does it how I think it should be done. Just run make test fw=true.

After thinking about it some more, I've come to this solution:

make test (tests everything)

make test component=fw (runs only framework tests)

make test component=pyspec (runs only pyspec tests)

Are we okay with this? Does it do what you want it to?

jtraglia · 2025-08-27T12:57:08Z

Hey Leo, can we give this PR a better title? Something less ambiguous but still <= 50 chars.

danceratopz

Couple of comments below, mainly to explain the motivation for this small change.

danceratopz · 2025-08-28T11:18:54Z

+# Run test framework tests.
+#
+# To run a specific test, append k=<test>, eg:
+#   make test k=test_verify_kzg_proof
+# To run tests with a specific bls library, append bls=<bls>, eg:
+#   make test bls=arkworks
+test_infra: MAYBE_TEST := $(if $(k),-k=$(k))
+# Disable parallelism which running a specific test.
+# Parallelism makes debugging difficult (print doesn't work).
+test_infra: MAYBE_PARALLEL := $(if $(k),,-n auto)
+test_infra: pyspec
+	@mkdir -p $(TEST_REPORT_DIR)
+	@$(PYTHON_VENV) -m pytest \
+		$(MAYBE_PARALLEL) \
+		--capture=no \
+		$(MAYBE_TEST) \
+		$(CURDIR)/tests/infra


I want to keep these tests bundled together. make test should run all tests.

I think going forward, as more framework tests get added, there would be value in allowing separate execution of these tests:

It creates a clearer separation of intent that makes it immediately clear to a new developer that these "framework tests" have an entirely different purpose.

It allows a developer to iterate faster on framework changes that are covered by fw tests (typically fw tests will run faster than the actual tests).

It allows better reporting, locally, particularly in CI; consensus-specs could have two workflows that would run make tests and make test_infra.

For now, until we improve the CI as in 2., we could add this new test_infra target as a dependency to the test target. This already enables 1. and 2. We can improve the CI in a follow-up PR for 3. and potentially remove the test_infra dep from test.

Co-authored-by: danceratopz <danceratopz@gmail.com>

jtraglia · 2025-08-29T03:42:47Z

If you're okay with my change, we can merge this after the broken test is fixed.

Also, please update the title & be sure that it's not crazy long.

Hey Leo, can we give this PR a better title? Something less ambiguous but still <= 50 chars.

Co-authored-by: Leo Lara <leolara@users.noreply.github.com>

leolara · 2025-09-11T09:13:31Z

@jtraglia continue-on-error doesn't work?

jtraglia · 2025-09-11T13:48:25Z

@jtraglia continue-on-error doesn't work?

It does work but I don't want it to continue. I will not merge anything with failing CI tests & there's no point in running 30+ mins of extra tests. I didn't see this change in your suggestion; I thought it just updated the task name.

jtraglia · 2025-09-12T13:32:34Z

@leolara could you fix the failing CI checks?

leolara · 2025-09-15T10:02:08Z

@jtraglia please check again

Improvements to the tests

e8f6f3d

jtraglia reviewed Aug 27, 2025

View reviewed changes

jtraglia added the testing CI, actions, tests, testing infra label Aug 27, 2025

danceratopz reviewed Aug 28, 2025

View reviewed changes

leolara changed the title ~~Improvements to the tests~~ Separate tests between spec tests and infra tests AND change framework for test_template_test Aug 28, 2025

leolara and others added 6 commits August 28, 2025 23:48

Update Makefile

3321ef2

Co-authored-by: danceratopz <danceratopz@gmail.com>

Apply suggestion from @danceratopz

5dcd5a9

Co-authored-by: danceratopz <danceratopz@gmail.com>

Add fw=true option to make test

ec3edc0

Add framework tests step

d05890b

Merge branch 'master' into leolara/improv-tests-framework

32b0482

Fix merge mistakes

ab76fc9

jtraglia added 2 commits August 28, 2025 23:07

Rethink solution

b599bd1

Update release & nightly workflows

4daf576

leolara commented Aug 29, 2025

View reviewed changes

Comment thread .github/workflows/release.yml Outdated

jtraglia and others added 2 commits August 29, 2025 06:49

Fix workflow step name

38bb4b6

Co-authored-by: Leo Lara <leolara@users.noreply.github.com>

Fix mistake & remove continue-on-error line

1874b0c

Merge branch 'master' into leolara/improv-tests-framework

96f9656

Merge branch 'master' into leolara/improv-tests-framework

e75a3cd

leolara and others added 2 commits September 15, 2025 07:51

Fix test

5ba45c7

Merge branch 'master' into leolara/improv-tests-framework

666a1ef

jtraglia approved these changes Sep 16, 2025

View reviewed changes

jtraglia changed the title ~~Separate tests between spec tests and infra tests AND change framework for test_template_test~~ Add ability to run framework tests separately Sep 16, 2025

jtraglia merged commit 41a8470 into ethereum:master Sep 16, 2025
15 checks passed

danceratopz mentioned this pull request Jan 15, 2026

Add Leo from STEEL, ethereum/consensus-specs protocolguild/documentation#471

Merged

Conversation

leolara commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jtraglia commented Aug 27, 2025

Uh oh!

danceratopz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jtraglia commented Aug 29, 2025

Uh oh!

Uh oh!

leolara commented Sep 11, 2025

Uh oh!

jtraglia commented Sep 11, 2025

Uh oh!

jtraglia commented Sep 12, 2025

Uh oh!

leolara commented Sep 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

leolara commented Aug 27, 2025 •

edited

Loading