[RFC] test: add/use a pytest wrapper for the build/boot tests #2045
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
I like it! Tests don't pass though :( Also I think there is a typo in the 2nd commit where you say it splits it into build and boot or boot, I think you mean build and boot or build. I'd also split the gen-manifest change into a commit by itself instead of mixing it with the pytest change but that's a minor 'complaint'. |
mvo5
force-pushed
the
images-pytest
branch
3 times, most recently
from
02afaed to
37a5bb0
Compare
We need a `-print-only` option so that we can see what gen-manifests would generate. This will be input for a pytest based framework (but seems generally be useful). This also tweaks the outputs so that any status message now goes to stderr so that one can parse stdout for the list of image manifests that would be generated.
mvo5
force-pushed
the
images-pytest
branch
2 times, most recently
from
a066047 to
8b482e6
Compare
Member
|
From the title I was kinda against this but after seeing this in action I feel it's nice. There's a little bug though: |
This is a tiny wrapper around the build/boot tests scripts to run them via pytest. Why another layer of indirection you ask? 1. it makes it easy to enumerate all tests one can run via `pytest --collect-only` 2. it makes it easy to run selected tests: `pytest -k rhel-10.2` 3. it provides a uniform way to that is the same for all pytest projects 4. with a sufficiently small number of tests (via -k) we could do --parallel 5. it abstracts away from the underlying scripts, we can (potentially) change how we run this without changing the higher layers Now that tests are easy to run locally it seems like a really nice feature.
This skips the build and boot tests for non-root users, they are currently required to run as root. A nice side effect is that the normal workflow of running pytest to run the "normal" tests is not interrupted. We could of course also use pytest and a custom "marker" like "@pytest.mark.integration" if this commit is considered too loose.
This makes it nicer to use `pytest -k build_and_boot` to find only our build and boot tests.
We need to excude the integration tests in the python-test workflow. So add a custom pytest marker. We could also move the integration tests into their own dir, I have no strong opinion.
lzap
approved these changes
Dec 3, 2025
Contributor
lzap
left a comment
lzap
left a comment
There was a problem hiding this comment.
I like that I can still use scripts directly. Nice.
thozza
approved these changes
Dec 3, 2025
Member
thozza
left a comment
thozza
left a comment
There was a problem hiding this comment.
This seems like a nice idea. Thanks.
mvo5
added a commit
to mvo5/images
that referenced
this pull request
Dec 8, 2025
Tiny followup for osbuild#2045 that adds some documentation under test/README.md with examples how the pytest wrapping can be useful.
mvo5
added a commit
to achilleas-k/images
that referenced
this pull request
Dec 9, 2025
In osbuild#2045 the script was converted to use pytest. However this does not work for ostree configs. A ostree config must be rewritten so that it points to an actual ostree commit and server. This is done in the `setup_dependencies()` script that then creates a different config/config_name. As this is different from the configs we get via `gen-manifests --dry-run` pytest cannot be used here. So for now be pragmatic and just revert https://github.com/osbuild/images/pull/2045/files#diff-8a0342654218957707e9769f498c86df3f070ee601f5b059f77b30d4d1f50d1eL18 We probably want get back to this eventually and make this setup part of pytest.
github-merge-queue bot
pushed a commit
that referenced
this pull request
Dec 10, 2025
In #2045 the script was converted to use pytest. However this does not work for ostree configs. A ostree config must be rewritten so that it points to an actual ostree commit and server. This is done in the `setup_dependencies()` script that then creates a different config/config_name. As this is different from the configs we get via `gen-manifests --dry-run` pytest cannot be used here. So for now be pragmatic and just revert https://github.com/osbuild/images/pull/2045/files#diff-8a0342654218957707e9769f498c86df3f070ee601f5b059f77b30d4d1f50d1eL18 We probably want get back to this eventually and make this setup part of pytest.
mvo5
added a commit
to mvo5/images
that referenced
this pull request
Dec 10, 2025
Tiny followup for osbuild#2045 that adds some documentation under test/README.md with examples how the pytest wrapping can be useful.
github-merge-queue bot
pushed a commit
that referenced
this pull request
Dec 10, 2025
Tiny followup for #2045 that adds some documentation under test/README.md with examples how the pytest wrapping can be useful.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[draft as this is potentially controversial]
This is a tiny wrapper around the build/boot tests scripts to run them via pytest. Why another layer of indirection you ask?
pytest --collect-only(708!)pytest -k rhel-10.2build_onlyandbuild_and_boottestcases now, this makes it trivial to usepytest --collect-only -k build_only(or `-k build_and_boot) and see that we do 443 build tests and 265 boot tests.Now that tests are easy to run locally it seems like a really nice feature.
[edit: Another thing I would like to do is make the tests that are skipped currently more visible, either via pytest.skip() when ensure_can_run_qemu_test() returns can exception or by having a declaratve skip-test list so that we can easily see what is currently skipped for what reason, right now the set of boot tests that we claim to do is a bit bigger than what we actually do (as we do not e.g. minimal-raw as it has no cloud-init]