diff --git a/docs/src/developers_guide/contributing_pytest_conversions.rst b/docs/src/developers_guide/contributing_pytest_conversions.rst new file mode 100644 index 0000000000..c6bb35c2cd --- /dev/null +++ b/docs/src/developers_guide/contributing_pytest_conversions.rst @@ -0,0 +1,56 @@ +.. include:: ../common_links.inc + +.. _contributing_pytest_conversions: + +******************************************* +Converting From ``unittest`` to ``pytest`` +******************************************* + +Conversion Checklist +-------------------- +.. note:: + Please bear in mind the following checklist is for general use; there may be + some cases which require extra context or thought before implementing these changes. + +#. Before making any manual changes, run https://github.com/dannysepler/pytestify + on the file. This does a lot of the brunt work for you! +#. Check for references to :class:`iris.tests.IrisTest`. If a class inherits + from this, remove the inheritance. Inheritance is unnecessary for + pytest tests, so :class:`iris.tests.IrisTest` has been deprecated + and its convenience methods have been moved to the + :mod:`iris.tests._shared_utils` module. +#. Check for references to ``unittest``. Many of the functions within unittest + are also in pytest, so often you can just change where the function is imported + from. +#. Check for references to ``self.assert``. Pytest has a lighter-weight syntax for + assertions, e.g. ``assert x == 2`` instead of ``assertEqual(x, 2)``. In the + case of custom :class:`~iris.tests.IrisTest` assertions, the majority of these + have been replicated in + :mod:`iris.tests._shared_utils`, but with snake_case instead of camelCase. + Some :class:`iris.tests.IrisTest` assertions have not been converted into + :mod:`iris.tests._shared_utils`, as these were deemed easy to achieve via + simple ``assert ...`` statements. +#. Check for references to ``setUp()``. Replace this with ``_setup()`` instead. + Ensure that this is decorated with ``@pytest.fixture(autouse=True)``. + + .. code-block:: python + + @pytest.fixture(autouse=True) + def _setup(self): + ... + +#. Check for references to ``@tests``. These should be changed to ``@_shared_utils``. +#. Check for references to ``with mock.patch("...")``. These should be replaced with + ``mocker.patch("...")``. Note, ``mocker.patch("...")`` is NOT a context manager. +#. Check for ``np.testing.assert...``. This can usually be swapped for + ``_shared_utils.assert...``. +#. Check for references to ``super()``. Most test classes used to inherit from + :class:`iris.tests.IrisTest`, so references to this should be removed. +#. Check for references to ``self.tmp_dir``. In pytest, ``tmp_path`` is used instead, + and can be passed into functions as a fixture. +#. Check for ``if __name__ == 'main'``. This is no longer needed with pytest. +#. Check for ``mock.patch("warnings.warn")``. This can be replaced with + ``pytest.warns(match=message)``. +#. Check the file against https://github.com/astral-sh/ruff , using ``pip install ruff`` -> + ``ruff check --select PT ``. + diff --git a/docs/src/developers_guide/contributing_running_tests.rst b/docs/src/developers_guide/contributing_running_tests.rst index f60cedba05..a72caa5881 100644 --- a/docs/src/developers_guide/contributing_running_tests.rst +++ b/docs/src/developers_guide/contributing_running_tests.rst @@ -87,7 +87,7 @@ experimental dependency not being present. SKIPPED [1] lib/iris/tests/unit/util/test_demote_dim_coord_to_aux_coord.py:29: Test(s) require external data. All Python decorators that skip tests will be defined in - ``lib/iris/tests/__init__.py`` with a function name with a prefix of + ``lib/iris/tests/_shared_utils.py`` with a function name with a prefix of ``skip_``. You can also run a specific test module. The example below runs the tests for diff --git a/docs/src/developers_guide/contributing_testing.rst b/docs/src/developers_guide/contributing_testing.rst deleted file mode 100644 index a65bcebd55..0000000000 --- a/docs/src/developers_guide/contributing_testing.rst +++ /dev/null @@ -1,147 +0,0 @@ -.. include:: ../common_links.inc - -.. _developer_test_categories: - - -Test Categories -*************** - -There are two main categories of tests within Iris: - -- :ref:`testing.unit_test` -- :ref:`testing.integration` - -Ideally, all code changes should be accompanied by one or more unit -tests, and by zero or more integration tests. - -But if in any doubt about what tests to add or how to write them please -feel free to submit a pull-request in any state and ask for assistance. - - -.. _testing.unit_test: - -Unit Tests -========== - -Code changes should be accompanied by enough unit tests to give a -high degree of confidence that the change works as expected. In -addition, the unit tests can help describe the intent behind a change. - -The docstring for each test module must state the unit under test. -For example: - - :literal:`"""Unit tests for the \`iris.experimental.raster.export_geotiff\` function."""` - -All unit tests must be placed and named according to the following -structure: - - -.. _testing.classes: - -Classes -------- - -When testing a class all the tests must reside in the module: - - :literal:`lib/iris/tests/unit//test_.py` - -Within this test module each tested method must have one or more -corresponding test classes, for example: - -* ``Test_`` -* ``Test___`` - -And within those test classes, the test methods must be named according -to the aspect of the tested method which they address. - -**Examples**: - -All unit tests for :py:class:`iris.cube.Cube` must reside in: - - :literal:`lib/iris/tests/unit/cube/test_Cube.py` - -Within that file the tests might look something like: - -.. code-block:: python - - # Tests for the Cube.xml() method. - class Test_xml(tests.IrisTest): - def test_some_general_stuff(self): - ... - - - # Tests for the Cube.xml() method, focussing on the behaviour of - # the checksums. - class Test_xml__checksum(tests.IrisTest): - def test_checksum_ignores_masked_values(self): - ... - - - # Tests for the Cube.add_dim_coord() method. - class Test_add_dim_coord(tests.IrisTest): - def test_normal_usage(self): - ... - - def test_coord_already_present(self): - ... - - -.. _testing.functions: - -Functions ---------- - -When testing a function all the tests must reside in the module: - - :literal:`lib/iris/tests/unit//test_.py` - -Within this test module there must be one or more test classes, for example: - -* ``Test`` -* ``TestAspectOfFunction`` - -And within those test classes, the test methods must be named according -to the aspect of the tested function which they address. - -**Examples**: - -All unit tests for :py:func:`iris.experimental.raster.export_geotiff` -must reside in: - - :literal:`lib/iris/tests/unit/experimental/raster/test_export_geotiff.py` - -Within that file the tests might look something like: - -.. code-block:: python - - # Tests focussing on the handling of different data types. - class TestDtypeAndValues(tests.IrisTest): - def test_int16(self): - ... - - def test_int16_big_endian(self): - ... - - - # Tests focussing on the handling of different projections. - class TestProjection(tests.IrisTest): - def test_no_ellipsoid(self): - ... - - -.. _testing.integration: - -Integration Tests -================= - -Some code changes may require tests which exercise several units in -order to demonstrate an important consequence of their interaction which -may not be apparent when considering the units in isolation. - -These tests must be placed in the ``lib/iris/tests/integration`` folder. -Unlike unit tests, there is no fixed naming scheme for integration -tests. But folders and files must be created as required to help -developers locate relevant tests. It is recommended they are named -according to the capabilities under test, e.g. -``metadata/test_pp_preservation.py``, and not named according to the -module(s) under test. diff --git a/docs/src/developers_guide/contributing_testing_index.rst b/docs/src/developers_guide/contributing_testing_index.rst index 2f5ae411e8..2d57da3d93 100644 --- a/docs/src/developers_guide/contributing_testing_index.rst +++ b/docs/src/developers_guide/contributing_testing_index.rst @@ -6,9 +6,9 @@ Testing .. toctree:: :maxdepth: 3 - contributing_testing - testing_tools + contributing_tests contributing_graphics_tests contributing_running_tests contributing_ci_tests contributing_benchmarks + contributing_pytest_conversions diff --git a/docs/src/developers_guide/contributing_tests.rst b/docs/src/developers_guide/contributing_tests.rst new file mode 100644 index 0000000000..e18a6987d2 --- /dev/null +++ b/docs/src/developers_guide/contributing_tests.rst @@ -0,0 +1,264 @@ +.. include:: ../common_links.inc + +.. _contributing_tests: + +************* +Writing Tests +************* + +.. note:: + If you're converting UnitTest tests to PyTest, check out + :ref:`contributing_pytest_conversions`. + +.. _developer_pytest_categories: + +Test Categories +=============== + +There are two main categories of tests within Iris: + +- `unit tests` +- `integration tests` + +Ideally, all code changes should be accompanied by one or more unit +tests, and by zero or more integration tests. + +Code changes should be accompanied by enough unit tests to give a +high degree of confidence that the change works as expected. In +addition, the unit tests can help describe the intent behind a change. + +The docstring for each test module must state the unit under test. +For example: + + :literal:`"""Unit tests for the \`iris.experimental.raster.export_geotiff\` function."""` + +When testing a class, all the tests must reside in the module: + + :literal:`lib/iris/tests/unit//test_.py` + +When testing a function, all the tests must reside in the module: + + :literal:`lib/iris/tests/unit//test_.py` + +Some code changes may require tests which exercise several units in +order to demonstrate an important consequence of their interaction which +may not be apparent when considering the units in isolation. These tests must +be placed in the ``lib/iris/tests/integration`` folder. + +With integration tests, folders and files must be created as required to help +developers locate relevant tests. It is recommended they are named +according to the capabilities under test, e.g. +``metadata/test_pp_preservation.py``, and not named according to the +module(s) under test. + +If in any doubt about what tests to add or how to write them please +feel free to submit a pull-request in any state and ask for assistance. + +.. _testing_style_guide: + +PyTest Style Guide +================== + +.. note:: + If you're converting UnitTest tests to PyTest, check out + :ref:`contributing_pytest_conversions`. + +This style guide should be approached pragmatically. Many of the guidelines laid out +below will not be practical in every scenario, and as such should not be considered +firm rules. + +At time of writing, some existing tests have already been written in PyTest, +so might not be abiding by these guidelines. + +`conftest.py `_ +----------------------------------------------------------------------------------------------------------------------------- + +There should be a ``conftest.py`` file in the ``root/unit`` and ``root/integration`` +folders. Additional lower level ``conftest``\s can be added if it is agreed there +is a need. + +`Fixtures `_ +------------------------------------------------------------------------------------ + +As far as is possible, the actual test function should do little else but the +actual assertion. Separating off preparation into fixtures may make the code +harder to follow, so compromises are acceptable. For example, setting up a test +``Cube`` should be a fixture, whereas creating a simple string +(``expected = "foo"``), or a single use setup, should *not* be a fixture. + + +New fixtures should always be considered for ``conftest`` when added. If it is +decided that they are not suitably reusable, they can be placed within the +local test file. + +`Parameterisation `_ +-------------------------------------------------------------------------------- + +Though it is a useful tool, we should not be complicating tests to work around +parameters; they should only be used when it is simple and apparent to implement. + +Where you are parameterising multiple tests with the same parameters, it is +usually prudent to use the `parameterisation within fixtures +`_. +When doing this, ensure within the tests that it is apparent that they are being +parameterised, either within the fixture name or with comments. + +All parameterisation benefits from +`ids `_, +and so should be used where possible. + +`Mocks `_ +-------------------------------------------------------------------- + +Any mocking should be done with ``pytest.mock``, and monkeypatching where suitable. + +.. note:: + If you think we're missing anything important here, please consider creating an + issue or discussion and share your ideas with the team! + +`Classes `_ +--------------------------------------------------------------------------------------------------- + +How and when to group tests within classes can be based on personal opinion, +we do not deem consistency on this a vital concern. + +Naming Test Classes and Functions +--------------------------------- + +When testing classes and their methods, each tested method within a test module +may have corresponding test classes, for example: + +* ``Test_`` +* ``Test___`` + +Within these test classes, the test methods must be named according +to the aspect of the tested method which they address. + +**Examples**: + +All unit tests for :py:class:`iris.cube.Cube` reside in: + + :literal:`lib/iris/tests/unit/cube/test_Cube.py` + +Within that file the tests might look something like: + +.. code-block:: python + + # A single test for the Cube.xml() method. + def test_xml_some_general_stuff(self): + ... + + + # A single test for the Cube.xml() method, focussing on the behaviour of + # the checksums. + def test_xml_checksum_ignores_masked_values(self): + ... + + + # Tests for the Cube.add_dim_coord() method. + class Test_add_dim_coord: + def test_normal_usage(self): + ... + + def test_coord_already_present(self): + ... + +When testing functions, within the test module there may be test classes, for +example: + +* ``Test`` +* ``TestAspectOfFunction`` + +Within those test classes, the test methods must be named according +to the aspect of the tested function which they address. + +**Examples**: + +All unit tests for :py:func:`iris.experimental.raster.export_geotiff` +must reside in: + + :literal:`lib/iris/tests/unit/experimental/raster/test_export_geotiff.py` + +Within that file the tests might look something like: + +.. code-block:: python + + # Tests focussing on the handling of different data types. + class TestDtypeAndValues: + def test_int16(self): + ... + + def test_int16_big_endian(self): + ... + + + # Tests focussing on the handling of different projections. + def test_no_ellipsoid(self): + ... + +There is no fixed naming scheme for integration tests. + +.. _testing_tools: + +Testing tools +============= + +.. note:: + :class:`iris.tests.IrisTest` has been deprecated, and replaced with + the :mod:`iris.tests._shared_utils` module. + +Iris has various internal convenience functions and utilities available to +support writing tests. Using these makes tests quicker and easier to write, and +also consistent with the rest of Iris (which makes it easier to work with the +code). Most of these conveniences are accessed through the +:mod:`iris.tests._shared_utils` module. + +.. tip:: + + All functions listed on this page are defined within + :mod:`iris.tests._shared_utils`. They can be accessed within a test using + ``_shared_utils.example_function``. + +Custom assertions +----------------- + +:mod:`iris.tests._shared_utils` supports a variety of custom pytest-style +assertions, such as :func:`~iris.tests._shared_utils.assert_array_equal`, and +:func:`~iris.tests._shared_utils.assert_array_almost_equal`. + +.. _create-missing: + +Saving results +-------------- + +Some tests compare the generated output to the expected result contained in a +file. Custom assertions for this include +:func:`~iris.tests._shared_utils.assert_CML_approx_data` +:func:`~iris.tests._shared_utils.assert_CDL` +:func:`~iris.tests._shared_utils.assert_CML` and +:func:`~iris.tests._shared_utils.assert_text_file`. See docstrings for more +information. + +.. note:: + + Sometimes code changes alter the results expected from a test containing the + above methods. These can be updated by removing the existing result files + and then running the file containing the test with a ``--create-missing`` + command line argument, or setting the ``IRIS_TEST_CREATE_MISSING`` + environment variable to anything non-zero. This will create the files rather + than erroring, allowing you to commit the updated results. + +Capturing exceptions and logging +-------------------------------- + +:mod:`~iris.tests._shared_utils` includes several context managers that can be used +to make test code tidier and easier to read. These include +:meth:`~iris.tests._shared_utils.assert_no_warnings_regexp` and +:meth:`~iris.tests._shared_utils.assert_logs`. + +Graphic tests +------------- + +As a package capable of generating graphical outputs, Iris has utilities for +creating and updating graphical tests - see :ref:`testing.graphics` for more +information. \ No newline at end of file diff --git a/docs/src/developers_guide/testing_tools.rst b/docs/src/developers_guide/testing_tools.rst deleted file mode 100755 index dd628d37fc..0000000000 --- a/docs/src/developers_guide/testing_tools.rst +++ /dev/null @@ -1,80 +0,0 @@ -.. include:: ../common_links.inc - -.. _testing_tools: - -Testing tools -************* - -Iris has various internal convenience functions and utilities available to -support writing tests. Using these makes tests quicker and easier to write, and -also consistent with the rest of Iris (which makes it easier to work with the -code). Most of these conveniences are accessed through the -:class:`iris.tests.IrisTest` class, from -which Iris' test classes then inherit. - -.. tip:: - - All functions listed on this page are defined within - :mod:`iris.tests.__init__.py` as methods of - :class:`iris.tests.IrisTest_nometa` (which :class:`iris.tests.IrisTest` - inherits from). They can be accessed within a test using - ``self.exampleFunction``. - -Custom assertions -================= - -:class:`iris.tests.IrisTest` supports a variety of custom unittest-style -assertions, such as :meth:`~iris.tests.IrisTest_nometa.assertArrayEqual`, -:meth:`~iris.tests.IrisTest_nometa.assertArrayAlmostEqual`. - -.. _create-missing: - -Saving results --------------- - -Some tests compare the generated output to the expected result contained in a -file. Custom assertions for this include -:meth:`~iris.tests.IrisTest_nometa.assertCMLApproxData` -:meth:`~iris.tests.IrisTest_nometa.assertCDL` -:meth:`~iris.tests.IrisTest_nometa.assertCML` and -:meth:`~iris.tests.IrisTest_nometa.assertTextFile`. See docstrings for more -information. - -.. note:: - - Sometimes code changes alter the results expected from a test containing the - above methods. These can be updated by removing the existing result files - and then running the file containing the test with a ``--create-missing`` - command line argument, or setting the ``IRIS_TEST_CREATE_MISSING`` - environment variable to anything non-zero. This will create the files rather - than erroring, allowing you to commit the updated results. - -Context managers -================ - -Capturing exceptions and logging --------------------------------- - -:class:`iris.tests.IrisTest` includes several context managers that can be used -to make test code tidier and easier to read. These include -:meth:`~iris.tests.IrisTest_nometa.assertWarnsRegexp` and -:meth:`~iris.tests.IrisTest_nometa.assertLogs`. - -Temporary files ---------------- - -It's also possible to generate temporary files in a concise fashion with -:meth:`~iris.tests.IrisTest_nometa.temp_filename`. - -Patching -======== - -:meth:`~iris.tests.IrisTest_nometa.patch` is a wrapper around ``unittest.patch`` -that will be automatically cleaned up at the end of the test. - -Graphic tests -============= - -As a package capable of generating graphical outputs, Iris has utilities for -creating and updating graphical tests - see :ref:`testing.graphics` for more -information. \ No newline at end of file