Added a style guide for iris pytest

docs/src/developers_guide/contributing_running_tests.rst

@@ -87,7 +87,7 @@ experimental dependency not being present.
       SKIPPED [1] lib/iris/tests/unit/util/test_demote_dim_coord_to_aux_coord.py:29: Test(s) require external data.
 
    All Python decorators that skip tests will be defined in
-   ``lib/iris/tests/__init__.py`` with a function name with a prefix of
+   ``lib/iris/tests/_shared_utils.py`` with a function name with a prefix of
    ``skip_``.
 
 You can also run a specific test module.  The example below runs the tests for

docs/src/developers_guide/contributing_testing.rst

@@ -45,18 +45,19 @@ When testing a class all the tests must reside in the module:
 
     :literal:`lib/iris/tests/unit/<fully/qualified/module>/test_<ClassName>.py`
 
-Within this test module each tested method must have one or more
-corresponding test classes, for example:
+
+Within this test module each tested method may corresponding test classes,
+for example:
 
 * ``Test_<name of public method>``
 * ``Test_<name of public method>__<aspect of method>``
 
-And within those test classes, the test methods must be named according
+Within test classes, the test methods must be named according
 to the aspect of the tested method which they address.
 
 **Examples**:
 
-All unit tests for :py:class:`iris.cube.Cube` must reside in:
+All unit tests for :py:class:`iris.cube.Cube` reside in:
 
     :literal:`lib/iris/tests/unit/cube/test_Cube.py`
 
@@ -95,12 +96,12 @@ When testing a function all the tests must reside in the module:
 
     :literal:`lib/iris/tests/unit/<fully/qualified/module>/test_<function_name>.py`
 
-Within this test module there must be one or more test classes, for example:
+Within this test module there may be test classes, for example:
 
 * ``Test``
 * ``TestAspectOfFunction``
 
-And within those test classes, the test methods must be named according
+Within those test classes, the test methods must be named according
 to the aspect of the tested function which they address.
 
 **Examples**:

docs/src/developers_guide/contributing_testing_index.rst

@@ -12,3 +12,5 @@ Testing
    contributing_running_tests
    contributing_ci_tests
    contributing_benchmarks
+   testing_style_guide
+   contributing_tests

docs/src/developers_guide/contributing_tests.rst

@@ -0,0 +1,301 @@
+.. include:: ../common_links.inc
+
+.. _contributing_tests:
+
+***********************
+Contributing Iris Tests
+***********************
+
+.. _developer_pytest_categories:
+
+Test Categories
+===============
+
+There are two main categories of tests within Iris:
+
+- :ref:`testing.unit_test`
+- :ref:`testing.integration`
+
+Ideally, all code changes should be accompanied by one or more unit
+tests, and by zero or more integration tests.
+
+But if in any doubt about what tests to add or how to write them please
+feel free to submit a pull-request in any state and ask for assistance.
+
+
+.. _pytesting.unit_test:
+
+Unit Tests
+----------
+
+Code changes should be accompanied by enough unit tests to give a
+high degree of confidence that the change works as expected. In
+addition, the unit tests can help describe the intent behind a change.
+
+The docstring for each test module must state the unit under test.
+For example:
+
+    :literal:`"""Unit tests for the \`iris.experimental.raster.export_geotiff\` function."""`
+
+All unit tests must be placed and named according to the following
+structure:
+
+
+.. _pytesting.classes:
+
+Classes
+-------
+
+When testing a class all the tests must reside in the module:
+
+    :literal:`lib/iris/tests/unit/<fully/qualified/module>/test_<ClassName>.py`
+
+
+Within this test module each tested method may corresponding test classes,
+for example:
+
+* ``Test_<name of public method>``
+* ``Test_<name of public method>__<aspect of method>``
+
+Within test classes, the test methods must be named according
+to the aspect of the tested method which they address.
+
+**Examples**:
+
+All unit tests for :py:class:`iris.cube.Cube` reside in:
+
+    :literal:`lib/iris/tests/unit/cube/test_Cube.py`
+
+Within that file the tests might look something like:
+
+.. code-block:: python
+
+    # Tests for the Cube.xml() method.
+    class Test_xml(tests.IrisTest):
+        def test_some_general_stuff(self):
+            ...
+
+
+    # Tests for the Cube.xml() method, focussing on the behaviour of
+    # the checksums.
+    class Test_xml__checksum(tests.IrisTest):
+        def test_checksum_ignores_masked_values(self):
+            ...
+
+
+    # Tests for the Cube.add_dim_coord() method.
+    class Test_add_dim_coord(tests.IrisTest):
+        def test_normal_usage(self):
+            ...
+
+        def test_coord_already_present(self):
+            ...
+
+
+.. _pytesting.functions:
+
+Functions
+---------
+
+When testing a function all the tests must reside in the module:
+
+    :literal:`lib/iris/tests/unit/<fully/qualified/module>/test_<function_name>.py`
+
+Within this test module there may be test classes, for example:
+
+* ``Test``
+* ``TestAspectOfFunction``
+
+Within those test classes, the test methods must be named according
+to the aspect of the tested function which they address.
+
+**Examples**:
+
+All unit tests for :py:func:`iris.experimental.raster.export_geotiff`
+must reside in:
+
+    :literal:`lib/iris/tests/unit/experimental/raster/test_export_geotiff.py`
+
+Within that file the tests might look something like:
+
+.. code-block:: python
+
+    # Tests focussing on the handling of different data types.
+    class TestDtypeAndValues(tests.IrisTest):
+        def test_int16(self):
+            ...
+
+        def test_int16_big_endian(self):
+            ...
+
+
+    # Tests focussing on the handling of different projections.
+    class TestProjection(tests.IrisTest):
+        def test_no_ellipsoid(self):
+            ...
+
+
+.. _pytesting.integration:
+
+Integration Tests
+-----------------
+
+Some code changes may require tests which exercise several units in
+order to demonstrate an important consequence of their interaction which
+may not be apparent when considering the units in isolation.
+
+These tests must be placed in the ``lib/iris/tests/integration`` folder.
+Unlike unit tests, there is no fixed naming scheme for integration
+tests. But folders and files must be created as required to help
+developers locate relevant tests. It is recommended they are named
+according to the capabilities under test, e.g.
+``metadata/test_pp_preservation.py``, and not named according to the
+module(s) under test.
+
+.. _pytesting_style_guide:
+
+PyTest Style Guide
+==================
+
+This style guide should be approached pragmatically. Most of the guidelines laid out
+below will not be practical in every scenario, and as such should not be considered
+firm rules.
+
+At time of writing, some existing tests have already been written in PyTest,
+so might not be abiding by these guidelines.
+
+Directory
+---------
+
+Where suitable, tests should be located within the relevant directories.
+In most circumstance, that means new tests should not be placed in the
+root test directory, but in the relevant sub-folders.
+
+`conftest.py <https://docs.pytest.org/en/7.1.x/reference/fixtures.html#conftest-py-sharing-fixtures-across-multiple-files>`_
+----------------------------------------------------------------------------------------------------------------------------
+
+There should be a ``conftest.py`` file in the ``root/unit`` and ``root/integration``
+folders. Additional lower level conftests can be added if it is agreed there
+is a need.
+
+`Fixtures <https://docs.pytest.org/en/stable/how-to/fixtures.html#how-to-fixtures>`_
+------------------------------------------------------------------------------------
+
+As far as is possible, the actual test function should do little else but the
+actual assertion. Separating off preparation into fixtures may make the code
+harder to follow, so compromises are acceptable. For example, setting up a test
+``Cube`` should be a fixture, whereas creating a simple string
+(``expected = "foo"``), or a single use setup, should *not* be a fixture.
+
+
+New fixtures should always be considered for conftest when added. If it is
+decided that they are not suitably reusable, they can be placed within the
+local test file.
+
+`Parameterisation <https://docs.pytest.org/en/stable/example/parametrize.html>`_
+--------------------------------------------------------------------------------
+
+Though it is a useful tool, we should not be complicating tests to work around
+parameters; they should only be used when it is simple and apparent to implement.
+
+Where you are parameterising multiple tests with the same parameters, it is
+usually prudent to use the `parameterisation within fixtures
+<https://docs.pytest.org/en/stable/how-to/fixtures.html#parametrizing-fixtures>`_.
+When doing this, ensure within the tests that it is apparent that they are being
+parameterised, either within the fixture name or with comments.
+
+All parameterisation benefits from
+`ids <https://docs.pytest.org/en/stable/example/parametrize.html#different-options-for-test-ids>`_,
+and so should be used where possible.
+
+`Classes <https://docs.pytest.org/en/stable/getting-started.html#group-multiple-tests-in-a-class>`_
+---------------------------------------------------------------------------------------------------
+
+How and when to group tests within classes can be based on personal opinion,
+we do not deem consistency on this a vital concern.
+
+`Mocks <https://docs.pytest.org/en/stable/how-to/monkeypatch.html>`_
+--------------------------------------------------------------------
+
+Any mocking should be done with ``pytest.mock``, and monkeypatching where suitable.
+
+.. note::
+    If you think we're missing anything important here, please consider creating an
+    issue or discussion and share your ideas with the team!
+
+.. _pytesting_tools:
+
+=============
+Testing tools
+=============
+
+.. note::
+    :class:`iris.tests.IrisTest` has been deprecated, and replaced with
+    the :mod:`iris.tests._shared_utils` module.
+
+Iris has various internal convenience functions and utilities available to
+support writing tests. Using these makes tests quicker and easier to write, and
+also consistent with the rest of Iris (which makes it easier to work with the
+code). Most of these conveniences are accessed through the
+:mod:`iris.tests._shared_utils` module.
+
+.. tip::
+
+    All functions listed on this page are defined within
+    :mod:`iris.tests._shared_utils`. They can be accessed within a test using
+    ``_shared_utils.exampleFunction``.
+
+Custom assertions
+-----------------
+
+:mod:`iris.tests._shared_utils` supports a variety of custom pytest-style
+assertions, such as :meth:`~iris.tests._shared_utils.assert_array_equal`, and
+:meth:`~iris.tests._shared_utils.assert_array_almost_equal`.
+
+.. _pycreate-missing:
+
+Saving results
+--------------
+
+Some tests compare the generated output to the expected result contained in a
+file. Custom assertions for this include
+:meth:`~iris.tests._shared_utils.assert_CML_approx_data`
+:meth:`~iris.tests._shared_utils.assert_CDL`
+:meth:`~iris.tests._shared_utils.assert_CML` and
+:meth:`~iris.tests._shared_utils.assert_text_file`. See docstrings for more
+information.
+
+.. note::
+
+    Sometimes code changes alter the results expected from a test containing the
+    above methods. These can be updated by removing the existing result files
+    and then running the file containing the test with a ``--create-missing``
+    command line argument, or setting the ``IRIS_TEST_CREATE_MISSING``
+    environment variable to anything non-zero. This will create the files rather
+    than erroring, allowing you to commit the updated results.
+
+Context managers
+----------------
+
+Capturing exceptions and logging
+--------------------------------
+
+:mod:`~iris.tests._shared_utils` includes several context managers that can be used
+to make test code tidier and easier to read. These include
+:meth:`~iris.tests.IrisTest_nometa.assert_no_warnings_regexp` and
+:meth:`~iris.tests.IrisTest_nometa.assert_logs`.
+
+Patching
+--------
+
+After the change from ``unittest`` to ``pytest``, ``IrisTest.patch`` has been
+converted into :meth:`~iris.tests._shared_utils.patch`.
+
+This is currently not implemented, and will raise an error if called.
+
+Graphic tests
+-------------
+
+As a package capable of generating graphical outputs, Iris has utilities for
+creating and updating graphical tests - see :ref:`testing.graphics` for more
+information.