pytestCheckHook: support inclusion and exclusion of path globs, test items, keywords, and markers

doc/languages-frameworks/python.section.md

@@ -1233,73 +1233,131 @@ test run would be:
 However, many repositories' test suites do not translate well to nix's build
 sandbox, and will generally need many tests to be disabled.
 
-To filter tests using pytest, one can do the following:
+This is achievable by
+- Including paths or test items (`path/to/file.py::MyClass` or `path/to/file.py::MyClass::test_method`) with positional arguments.
+- Excluding paths with `--ignore` or globbed paths with `--ignore-glob`.
+- Excluding test items using the `--deselect` flag.
+- Including or excluding classes or test methods by their name using the `-k` flag.
+- Including or excluding test by their marks using the `-m` flag.
 
-```nix
-{
-  nativeCheckInputs = [ pytest ];
-  # avoid tests which need additional data or touch network
-  checkPhase = ''
-    runHook preCheck
+We highly recommend `pytestCheckHook` for an easier and more structural setup.
 
-    pytest tests/ --ignore=tests/integration -k 'not download and not update' --ignore=tests/test_failing.py
+#### Using pytestCheckHook {#using-pytestcheckhook}
 
-    runHook postCheck
-  '';
+`pytestCheckHook` is a convenient hook which will set up (or configure)
+a [`checkPhase`](#ssec-check-phase) to run `pytest`. This is also beneficial
+when a package may need many items disabled to run the test suite.
+Most packages use `pytest` or `unittest`, which is compatible with `pytest`,
+so you will most likely use `pytestCheckHook`.
+
+To use `pytestCheckHook`, add it to `nativeCheckInputs`.
+Adding `pytest` is not required, since it is included with `pytestCheckHook`.
+
+```nix
+{
+  nativeCheckInputs = [
+    pytestCheckHook
+  ];
 }
 ```
 
-`--ignore` will tell pytest to ignore that file or directory from being
-collected as part of a test run. This is useful is a file uses a package
-which is not available in nixpkgs, thus skipping that test file is much
-easier than having to create a new package.
+`pytestCheckHook` recognizes the following attributes:
 
-`-k` is used to define a predicate for test names. In this example, we are
-filtering out tests which contain `download` or `update` in their test case name.
-Only one `-k` argument is allowed, and thus a long predicate should be concatenated
-with “\\” and wrapped to the next line.
+`enabledTestPaths` and `disabledTestPaths`
 
-::: {.note}
-In pytest==6.0.1, the use of “\\” to continue a line (e.g. `-k 'not download \'`) has
-been removed, in this case, it's recommended to use `pytestCheckHook`.
-:::
+:   To specify path globs (files or directories) or test items.
 
-#### Using pytestCheckHook {#using-pytestcheckhook}
+`enabledTests` and `disabledTests`
 
-`pytestCheckHook` is a convenient hook which will set up (or configure)
-a [`checkPhase`](#ssec-check-phase) to run `pytest`. This is also beneficial
-when a package may need many items disabled to run the test suite.
-Most packages use `pytest` or `unittest`, which is compatible with `pytest`,
-so you will most likely use `pytestCheckHook`.
+:   To specify keywords for class names or test method names.
+
+`enabledTestMarks` and `disabledTestMarks`
+
+:   To specify test marks.
 
-Using the example above, the analogous `pytestCheckHook` usage would be:
+`pytestFlags`
+
+:   To append additional command-line arguments to `pytest`.
+
+By default, `pytest` automatically discovers which tests to run.
+If tests are explicitly enabled, only those tests will run.
+A test, that is both enabled and disabled, will not run.
+
+The following example demonstrates usage of various `pytestCheckHook` attributes:
 
 ```nix
 {
   nativeCheckInputs = [
     pytestCheckHook
   ];
 
-  # requires additional data
-  pytestFlags = [
+  # Allow running the following test paths and test objects.
+  enabledTestPaths = [
+    # Find tests under the tests directory.
+    # The trailing slash is not necessary.
     "tests/"
-    "--ignore=tests/integration"
+
+    # Additionally run test_foo
+    "other-tests/test_foo.py::Foo::test_foo"
   ];
 
+  # Override the above-enabled test paths and test objects.
+  disabledTestPaths = [
+    # Tests under tests/integration requires additional data.
+    "tests/integration"
+  ];
+
+  # Allow tests by keywords matching their class names or method names.
+  enabledTests = [
+    # pytest by default only runs test methods begin with "test_" or end with "_test".
+    # This includes all functions whose name contains "test".
+    "test"
+  ];
+
+  # Override the above-enabled tests by keywords matching their class names or method names.
   disabledTests = [
-    # touches network
+    # Tests touching networks.
+    "upload"
     "download"
-    "update"
   ];
 
-  disabledTestPaths = [
-    "tests/test_failing.py"
+  # Additional pytest flags
+  pytestFlags = [
+    # Disable benchmarks and run benchmarking tests only once.
+    "--benchmark-disable"
+  ];
+}
+```
+
+These attributes are all passed into the derivation directly
+and added to the `pytest` command without additional Bash expansion.
+It requires `__structuredAttrs = true` to pass list elements containing spaces.
+
+The `<enabled/disabled>TestsPaths` attributes expand Unix-style globs.
+If a test path contains characters like `*`, `?`, `[`, or `]`, you can
+quote them with square brackets (`[*]`, `[?]`, `[[]`, and `[]]`) to match literally.
+
+The `<enabled/disabled>Tests` and `<enabled/disabled>TestMarks` attribute pairs
+form a logical expression `((included_element1) or (included_element2)) and not (excluded_element1) and not (excluded_element2)`
+which will be passed to pytest's `-k` and `-m` flags respectively.
+With `__structuredAttrs = true` enabled, they additionally support sub-expressions.
+
+For example, you could disable test items like `TestFoo::test_bar_functionality`
+by disabling tests that match both `"Foo"` **and** `"bar"`:
+
+```nix
+{
+  __structuredAttrs = true;
+
+  disabledTests = [
+    "Foo and bar"
   ];
 }
 ```
 
-This is especially useful when tests need to be conditionally disabled,
-for example:
+The main benefits of using `pytestCheckHook` to construct `pytest` commands
+is structuralization and eval-time accessibility.
+This is especially helpful to select tests or specify flags conditionally:
 
 ```nix
 {
@@ -1317,10 +1375,6 @@ for example:
 }
 ```
 
-Trying to concatenate the related strings to disable tests in a regular
-[`checkPhase`](#ssec-check-phase) would be much harder to read. This also enables us to comment on
-why specific tests are disabled.
-
 #### Using pythonImportsCheck {#using-pythonimportscheck}
 
 Although unit tests are highly preferred to validate correctness of a package, not
@@ -2008,36 +2062,6 @@ Occasionally packages don't make use of a common test framework, which may then
 
 #### Common issues {#common-issues}
 
-* Non-working tests can often be deselected. Most Python modules
-  do follow the standard test protocol where the pytest runner can be used.
-  `pytest` supports the `-k` and `--ignore-glob` parameters to ignore test
-  methods or classes as well as whole files. For `pytestCheckHook` these are
-  conveniently exposed as `disabledTests` and `disabledTestPaths` respectively.
-
-  ```nix
-  buildPythonPackage {
-    # ...
-    nativeCheckInputs = [
-      pytestCheckHook
-    ];
-
-    disabledTests = [
-      "function_name"
-      "other_function"
-    ];
-
-    disabledTestPaths = [
-      "path/to/performance.py"
-      "path/to/connect-*.py"
-    ];
-  }
-  ```
-
-  ::: {.note}
-  If the test path to disable contains characters like `*`, `?`, `[`, and `]`,
-  quote them with square brackets (`[*]`, `[?]`, `[[]`, and `[]]`) to match literally.
-  :::
-
 * Tests that attempt to access `$HOME` can be fixed by using the following
   work-around before running tests (e.g. `preCheck`): `export HOME=$(mktemp -d)`
 * Compiling with Cython causes tests to fail with a `ModuleNotLoadedError`.

pkgs/development/interpreters/python/hooks/default.nix

@@ -170,6 +170,14 @@ in
               "test_print"
             ] ++ previousPythonAttrs.disabledTests or [ ];
           });
+          disabledTests-expression = objprint.overridePythonAttrs (previousPythonAttrs: {
+            __structuredAttrs = true;
+            pname = "test-pytestCheckHook-disabledTests-expression-${previousPythonAttrs.pname}";
+            disabledTests = [
+              "TestBasic and test_print"
+              "test_str"
+            ] ++ previousPythonAttrs.disabledTests or [ ];
+          });
           disabledTestPaths = objprint.overridePythonAttrs (previousPythonAttrs: {
             pname = "test-pytestCheckHook-disabledTestPaths-${previousPythonAttrs.pname}";
             disabledTestPaths = [
@@ -184,6 +192,12 @@ in
               ] ++ previousPythonAttrs.disabledTestPaths or [ ];
             })
           );
+          disabledTestPaths-item = objprint.overridePythonAttrs (previousPythonAttrs: {
+            pname = "test-pytestCheckHook-disabledTestPaths-item-${previousPythonAttrs.pname}";
+            disabledTestPaths = [
+              "tests/test_basic.py::TestBasic"
+            ] ++ previousPythonAttrs.disabledTestPaths or [ ];
+          });
           disabledTestPaths-glob = objprint.overridePythonAttrs (previousPythonAttrs: {
             pname = "test-pytestCheckHook-disabledTestPaths-glob-${previousPythonAttrs.pname}";
             disabledTestPaths = [
@@ -198,6 +212,78 @@ in
               ] ++ previousPythonAttrs.disabledTestPaths or [ ];
             })
           );
+          enabledTests = objprint.overridePythonAttrs (previousPythonAttrs: {
+            pname = "test-pytestCheckHook-enabledTests-${previousPythonAttrs.pname}";
+            enabledTests = [
+              "TestBasic"
+            ] ++ previousPythonAttrs.disabledTests or [ ];
+          });
+          enabledTests-expression = objprint.overridePythonAttrs (previousPythonAttrs: {
+            __structuredAttrs = true;
+            pname = "test-pytestCheckHook-enabledTests-expression-${previousPythonAttrs.pname}";
+            enabledTests = [
+              "TestBasic and test_print"
+              "test_str"
+            ] ++ previousPythonAttrs.disabledTests or [ ];
+          });
+          enabledTests-disabledTests = objprint.overridePythonAttrs (previousPythonAttrs: {
+            pname = "test-pytestCheckHook-enabledTests-disabledTests-${previousPythonAttrs.pname}";
+            enabledTests = [
+              "TestBasic"
+            ] ++ previousPythonAttrs.disabledTests or [ ];
+            disabledTests = [
+              "test_print"
+            ] ++ previousPythonAttrs.disabledTests or [ ];
+          });
+          enabledTestPaths = objprint.overridePythonAttrs (previousPythonAttrs: {
+            pname = "test-pytestCheckHook-enabledTestPaths-${previousPythonAttrs.pname}";
+            enabledTestPaths = [
+              "tests/test_basic.py"
+            ] ++ previousPythonAttrs.enabledTestPaths or [ ];
+          });
+          enabledTestPaths-nonexistent = testers.testBuildFailure (
+            objprint.overridePythonAttrs (previousPythonAttrs: {
+              pname = "test-pytestCheckHook-enabledTestPaths-nonexistent-${previousPythonAttrs.pname}";
+              enabledTestPaths = [
+                "tests/test_foo.py"
+              ] ++ previousPythonAttrs.enabledTestPaths or [ ];
+            })
+          );
+          enabledTestPaths-dir = objprint.overridePythonAttrs (previousPythonAttrs: {
+            pname = "test-pytestCheckHook-enabledTestPaths-dir-${previousPythonAttrs.pname}";
+            enabledTestPaths = [
+              "tests"
+            ] ++ previousPythonAttrs.enabledTestPaths or [ ];
+          });
+          enabledTestPaths-dir-disabledTestPaths = objprint.overridePythonAttrs (previousPythonAttrs: {
+            pname = "test-pytestCheckHook-enabledTestPaths-dir-disabledTestPaths-${previousPythonAttrs.pname}";
+            enabledTestPaths = [
+              "tests"
+            ] ++ previousPythonAttrs.enabledTestPaths or [ ];
+            disabledTestPaths = [
+              "tests/test_basic.py"
+            ] ++ previousPythonAttrs.disabledTestPaths or [ ];
+          });
+          enabledTestPaths-glob = objprint.overridePythonAttrs (previousPythonAttrs: {
+            pname = "test-pytestCheckHook-enabledTestPaths-glob-${previousPythonAttrs.pname}";
+            enabledTestPaths = [
+              "tests/test_obj*.py"
+            ] ++ previousPythonAttrs.enabledTestPaths or [ ];
+          });
+          enabledTestPaths-glob-nonexistent = testers.testBuildFailure (
+            objprint.overridePythonAttrs (previousPythonAttrs: {
+              pname = "test-pytestCheckHook-enabledTestPaths-glob-nonexistent-${previousPythonAttrs.pname}";
+              enabledTestPaths = [
+                "tests/test_foo*.py"
+              ] ++ previousPythonAttrs.enabledTestPaths or [ ];
+            })
+          );
+          enabledTestPaths-item = objprint.overridePythonAttrs (previousPythonAttrs: {
+            pname = "test-pytestCheckHook-enabledTestPaths-item-${previousPythonAttrs.pname}";
+            enabledTestPaths = [
+              "tests/test_basic.py::TestBasic"
+            ] ++ previousPythonAttrs.enabledTestPaths or [ ];
+          });
         };
       };
     } ./pytest-check-hook.sh