Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TST: migrate image comparison tests from nose to pytest-mpl (⏰ wait on #4122) #4118

Closed

Conversation

neutrinoceros
Copy link
Member

@neutrinoceros neutrinoceros commented Sep 10, 2022

PR Summary

Partially addresses #4102

This is far from done, but I need early iterations with CI to see what problems I encounter.
Right now I'm only aiming at image tests being skipped on nose

approach:
the main difficulty is that our yield base image tests are built via loops, while pytest-mpl relies on return values of test functions. When pytest.mark.parametrize is appropriate, this can be a one-to-one translation, but otherwise I rely on generated code to reproduce existing test (by which I mean that I write many alike plain test functions, just not purely by hand).

TODO:

  • stop using image answer tests (replace them with pytest-mpl)
    • GenericImageTest
    • VRImageComparisonTest
    • PhasePlotAttributeTest
    • PixelizedParticleProjectionValuesTest
    • PixelizedProjectionValuesTest
    • PlotWindowAttributeTest
  • create image baseline folder (should probably be a submodule, not sure if a new one is warranted, or if we should just reuse answer-store)
  • add image image comparison-only test job
  • cleanup tests/report_failed_answers.py
  • cleanup uses of filename=... in mpl_image_compar markers
  • consider refactoring again with pytest.mark.parametrize

opened questions:

  • How can we handle "big data" image tests, and more generally image tests that use real (large) data ? Currently these tests are run on Jenkins + nose

@neutrinoceros neutrinoceros added enhancement Making something better tests: running tests Issues with the test setup pytest refactor improve readability, maintainability, modularity viz: 2D and removed enhancement Making something better labels Sep 10, 2022
@neutrinoceros neutrinoceros force-pushed the pytest_mpl_image_tests branch 2 times, most recently from 2a2fd8f to f96bd77 Compare September 11, 2022 18:23
@neutrinoceros
Copy link
Member Author

neutrinoceros commented Sep 12, 2022

I was not originally planning to fully replace the "answer tests" job on GHA but I see that after removing all image tests from it, it now runs only a very small set of tests and takes about 20s to run (and 10min to build), so I think I should just clear that out completely and replace the job with a new one 100% focused on image-comparison tests.

@neutrinoceros
Copy link
Member Author

I think I should split this in two parts and separate tests that require real data (running on Jenkins) from the rest (running on GH)

@neutrinoceros neutrinoceros changed the title TST: migrate image comparison tests from nose to pytest-mpl TST: migrate image comparison tests from nose to pytest-mpl (⏰ wait on #4122) Sep 27, 2022
@neutrinoceros
Copy link
Member Author

This is probably beyond saving at this point. In any case, #4122 has most of this work in a mergeable state, so I think I should just close this one.

@neutrinoceros neutrinoceros deleted the pytest_mpl_image_tests branch March 15, 2023 15:22
@matthewturk
Copy link
Member

Can you describe the difference with #4122?

@neutrinoceros
Copy link
Member Author

Mostly, I was initially going for a wider target, and wanted to migrate all image tests at once, including the ones that require large datasets that are only run on Jenkins. In #4122 I just focused on GHA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
refactor improve readability, maintainability, modularity tests: running tests Issues with the test setup viz: 2D
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants