diff --git a/doc/languages-frameworks/python.section.md b/doc/languages-frameworks/python.section.md index bbc5da116a6a8..96dec589dadc9 100644 --- a/doc/languages-frameworks/python.section.md +++ b/doc/languages-frameworks/python.section.md @@ -1233,46 +1233,57 @@ test run would be: However, many repositories' test suites do not translate well to nix's build sandbox, and will generally need many tests to be disabled. -To filter tests using pytest, one can do the following: +This is achievable by +- Including paths or test items (`path/to/file.py::MyClass` or `path/to/file.py::MyClass::test_method`) with positional arguments. +- Excluding paths with `--ignore` or globbed paths with `--ignore-glob`. +- Excluding test items using the `--deselect` flag. +- Including or excluding classes or test methods by their name using the `-k` flag. +- Including or excluding test by their marks using the `-m` flag. -```nix -{ - nativeCheckInputs = [ pytest ]; - # avoid tests which need additional data or touch network - checkPhase = '' - runHook preCheck +We highly recommend `pytestCheckHook` for an easier and more structural setup. - pytest tests/ --ignore=tests/integration -k 'not download and not update' --ignore=tests/test_failing.py +#### Using pytestCheckHook {#using-pytestcheckhook} - runHook postCheck - ''; +`pytestCheckHook` is a convenient hook which will set up (or configure) +a [`checkPhase`](#ssec-check-phase) to run `pytest`. This is also beneficial +when a package may need many items disabled to run the test suite. +Most packages use `pytest` or `unittest`, which is compatible with `pytest`, +so you will most likely use `pytestCheckHook`. + +To use `pytestCheckHook`, add it to `nativeCheckInputs`. +Adding `pytest` is not required, since it is included with `pytestCheckHook`. + +```nix +{ + nativeCheckInputs = [ + pytestCheckHook + ]; } ``` -`--ignore` will tell pytest to ignore that file or directory from being -collected as part of a test run. This is useful is a file uses a package -which is not available in nixpkgs, thus skipping that test file is much -easier than having to create a new package. +`pytestCheckHook` recognizes the following attributes: -`-k` is used to define a predicate for test names. In this example, we are -filtering out tests which contain `download` or `update` in their test case name. -Only one `-k` argument is allowed, and thus a long predicate should be concatenated -with “\\” and wrapped to the next line. +`enabledTestPaths` and `disabledTestPaths` -::: {.note} -In pytest==6.0.1, the use of “\\” to continue a line (e.g. `-k 'not download \'`) has -been removed, in this case, it's recommended to use `pytestCheckHook`. -::: +: To specify path globs (files or directories) or test items. -#### Using pytestCheckHook {#using-pytestcheckhook} +`enabledTests` and `disabledTests` -`pytestCheckHook` is a convenient hook which will set up (or configure) -a [`checkPhase`](#ssec-check-phase) to run `pytest`. This is also beneficial -when a package may need many items disabled to run the test suite. -Most packages use `pytest` or `unittest`, which is compatible with `pytest`, -so you will most likely use `pytestCheckHook`. +: To specify keywords for class names or test method names. + +`enabledTestMarks` and `disabledTestMarks` + +: To specify test marks. -Using the example above, the analogous `pytestCheckHook` usage would be: +`pytestFlags` + +: To append additional command-line arguments to `pytest`. + +By default, `pytest` automatically discovers which tests to run. +If tests are explicitly enabled, only those tests will run. +A test, that is both enabled and disabled, will not run. + +The following example demonstrates usage of various `pytestCheckHook` attributes: ```nix { @@ -1280,26 +1291,73 @@ Using the example above, the analogous `pytestCheckHook` usage would be: pytestCheckHook ]; - # requires additional data - pytestFlags = [ + # Allow running the following test paths and test objects. + enabledTestPaths = [ + # Find tests under the tests directory. + # The trailing slash is not necessary. "tests/" - "--ignore=tests/integration" + + # Additionally run test_foo + "other-tests/test_foo.py::Foo::test_foo" ]; + # Override the above-enabled test paths and test objects. + disabledTestPaths = [ + # Tests under tests/integration requires additional data. + "tests/integration" + ]; + + # Allow tests by keywords matching their class names or method names. + enabledTests = [ + # pytest by default only runs test methods begin with "test_" or end with "_test". + # This includes all functions whose name contains "test". + "test" + ]; + + # Override the above-enabled tests by keywords matching their class names or method names. disabledTests = [ - # touches network + # Tests touching networks. + "upload" "download" - "update" ]; - disabledTestPaths = [ - "tests/test_failing.py" + # Additional pytest flags + pytestFlags = [ + # Disable benchmarks and run benchmarking tests only once. + "--benchmark-disable" + ]; +} +``` + +These attributes are all passed into the derivation directly +and added to the `pytest` command without additional Bash expansion. +It requires `__structuredAttrs = true` to pass list elements containing spaces. + +The `TestsPaths` attributes expand Unix-style globs. +If a test path contains characters like `*`, `?`, `[`, or `]`, you can +quote them with square brackets (`[*]`, `[?]`, `[[]`, and `[]]`) to match literally. + +The `Tests` and `TestMarks` attribute pairs +form a logical expression `((included_element1) or (included_element2)) and not (excluded_element1) and not (excluded_element2)` +which will be passed to pytest's `-k` and `-m` flags respectively. +With `__structuredAttrs = true` enabled, they additionally support sub-expressions. + +For example, you could disable test items like `TestFoo::test_bar_functionality` +by disabling tests that match both `"Foo"` **and** `"bar"`: + +```nix +{ + __structuredAttrs = true; + + disabledTests = [ + "Foo and bar" ]; } ``` -This is especially useful when tests need to be conditionally disabled, -for example: +The main benefits of using `pytestCheckHook` to construct `pytest` commands +is structuralization and eval-time accessibility. +This is especially helpful to select tests or specify flags conditionally: ```nix { @@ -1317,10 +1375,6 @@ for example: } ``` -Trying to concatenate the related strings to disable tests in a regular -[`checkPhase`](#ssec-check-phase) would be much harder to read. This also enables us to comment on -why specific tests are disabled. - #### Using pythonImportsCheck {#using-pythonimportscheck} Although unit tests are highly preferred to validate correctness of a package, not @@ -2008,36 +2062,6 @@ Occasionally packages don't make use of a common test framework, which may then #### Common issues {#common-issues} -* Non-working tests can often be deselected. Most Python modules - do follow the standard test protocol where the pytest runner can be used. - `pytest` supports the `-k` and `--ignore-glob` parameters to ignore test - methods or classes as well as whole files. For `pytestCheckHook` these are - conveniently exposed as `disabledTests` and `disabledTestPaths` respectively. - - ```nix - buildPythonPackage { - # ... - nativeCheckInputs = [ - pytestCheckHook - ]; - - disabledTests = [ - "function_name" - "other_function" - ]; - - disabledTestPaths = [ - "path/to/performance.py" - "path/to/connect-*.py" - ]; - } - ``` - - ::: {.note} - If the test path to disable contains characters like `*`, `?`, `[`, and `]`, - quote them with square brackets (`[*]`, `[?]`, `[[]`, and `[]]`) to match literally. - ::: - * Tests that attempt to access `$HOME` can be fixed by using the following work-around before running tests (e.g. `preCheck`): `export HOME=$(mktemp -d)` * Compiling with Cython causes tests to fail with a `ModuleNotLoadedError`. diff --git a/pkgs/development/interpreters/python/hooks/default.nix b/pkgs/development/interpreters/python/hooks/default.nix index d8b8f661ee2ac..d80013fa212da 100644 --- a/pkgs/development/interpreters/python/hooks/default.nix +++ b/pkgs/development/interpreters/python/hooks/default.nix @@ -170,6 +170,14 @@ in "test_print" ] ++ previousPythonAttrs.disabledTests or [ ]; }); + disabledTests-expression = objprint.overridePythonAttrs (previousPythonAttrs: { + __structuredAttrs = true; + pname = "test-pytestCheckHook-disabledTests-expression-${previousPythonAttrs.pname}"; + disabledTests = [ + "TestBasic and test_print" + "test_str" + ] ++ previousPythonAttrs.disabledTests or [ ]; + }); disabledTestPaths = objprint.overridePythonAttrs (previousPythonAttrs: { pname = "test-pytestCheckHook-disabledTestPaths-${previousPythonAttrs.pname}"; disabledTestPaths = [ @@ -184,6 +192,12 @@ in ] ++ previousPythonAttrs.disabledTestPaths or [ ]; }) ); + disabledTestPaths-item = objprint.overridePythonAttrs (previousPythonAttrs: { + pname = "test-pytestCheckHook-disabledTestPaths-item-${previousPythonAttrs.pname}"; + disabledTestPaths = [ + "tests/test_basic.py::TestBasic" + ] ++ previousPythonAttrs.disabledTestPaths or [ ]; + }); disabledTestPaths-glob = objprint.overridePythonAttrs (previousPythonAttrs: { pname = "test-pytestCheckHook-disabledTestPaths-glob-${previousPythonAttrs.pname}"; disabledTestPaths = [ @@ -198,6 +212,78 @@ in ] ++ previousPythonAttrs.disabledTestPaths or [ ]; }) ); + enabledTests = objprint.overridePythonAttrs (previousPythonAttrs: { + pname = "test-pytestCheckHook-enabledTests-${previousPythonAttrs.pname}"; + enabledTests = [ + "TestBasic" + ] ++ previousPythonAttrs.disabledTests or [ ]; + }); + enabledTests-expression = objprint.overridePythonAttrs (previousPythonAttrs: { + __structuredAttrs = true; + pname = "test-pytestCheckHook-enabledTests-expression-${previousPythonAttrs.pname}"; + enabledTests = [ + "TestBasic and test_print" + "test_str" + ] ++ previousPythonAttrs.disabledTests or [ ]; + }); + enabledTests-disabledTests = objprint.overridePythonAttrs (previousPythonAttrs: { + pname = "test-pytestCheckHook-enabledTests-disabledTests-${previousPythonAttrs.pname}"; + enabledTests = [ + "TestBasic" + ] ++ previousPythonAttrs.disabledTests or [ ]; + disabledTests = [ + "test_print" + ] ++ previousPythonAttrs.disabledTests or [ ]; + }); + enabledTestPaths = objprint.overridePythonAttrs (previousPythonAttrs: { + pname = "test-pytestCheckHook-enabledTestPaths-${previousPythonAttrs.pname}"; + enabledTestPaths = [ + "tests/test_basic.py" + ] ++ previousPythonAttrs.enabledTestPaths or [ ]; + }); + enabledTestPaths-nonexistent = testers.testBuildFailure ( + objprint.overridePythonAttrs (previousPythonAttrs: { + pname = "test-pytestCheckHook-enabledTestPaths-nonexistent-${previousPythonAttrs.pname}"; + enabledTestPaths = [ + "tests/test_foo.py" + ] ++ previousPythonAttrs.enabledTestPaths or [ ]; + }) + ); + enabledTestPaths-dir = objprint.overridePythonAttrs (previousPythonAttrs: { + pname = "test-pytestCheckHook-enabledTestPaths-dir-${previousPythonAttrs.pname}"; + enabledTestPaths = [ + "tests" + ] ++ previousPythonAttrs.enabledTestPaths or [ ]; + }); + enabledTestPaths-dir-disabledTestPaths = objprint.overridePythonAttrs (previousPythonAttrs: { + pname = "test-pytestCheckHook-enabledTestPaths-dir-disabledTestPaths-${previousPythonAttrs.pname}"; + enabledTestPaths = [ + "tests" + ] ++ previousPythonAttrs.enabledTestPaths or [ ]; + disabledTestPaths = [ + "tests/test_basic.py" + ] ++ previousPythonAttrs.disabledTestPaths or [ ]; + }); + enabledTestPaths-glob = objprint.overridePythonAttrs (previousPythonAttrs: { + pname = "test-pytestCheckHook-enabledTestPaths-glob-${previousPythonAttrs.pname}"; + enabledTestPaths = [ + "tests/test_obj*.py" + ] ++ previousPythonAttrs.enabledTestPaths or [ ]; + }); + enabledTestPaths-glob-nonexistent = testers.testBuildFailure ( + objprint.overridePythonAttrs (previousPythonAttrs: { + pname = "test-pytestCheckHook-enabledTestPaths-glob-nonexistent-${previousPythonAttrs.pname}"; + enabledTestPaths = [ + "tests/test_foo*.py" + ] ++ previousPythonAttrs.enabledTestPaths or [ ]; + }) + ); + enabledTestPaths-item = objprint.overridePythonAttrs (previousPythonAttrs: { + pname = "test-pytestCheckHook-enabledTestPaths-item-${previousPythonAttrs.pname}"; + enabledTestPaths = [ + "tests/test_basic.py::TestBasic" + ] ++ previousPythonAttrs.enabledTestPaths or [ ]; + }); }; }; } ./pytest-check-hook.sh diff --git a/pkgs/development/interpreters/python/hooks/pytest-check-hook.sh b/pkgs/development/interpreters/python/hooks/pytest-check-hook.sh index ff95f73ba20c8..bbdc066d48b24 100644 --- a/pkgs/development/interpreters/python/hooks/pytest-check-hook.sh +++ b/pkgs/development/interpreters/python/hooks/pytest-check-hook.sh @@ -3,22 +3,64 @@ echo "Sourcing pytest-check-hook" +function _pytestIncludeExcludeExpr() { + local includeListName="$1" + local -n includeListRef="$includeListName" + local excludeListName="$2" + local -n excludeListRef="$excludeListName" + local includeString excludeString + if [[ -n "${includeListRef[*]-}" ]]; then + # ((element1) or (element2)) + includeString="(($(concatStringsSep ") or (" "$includeListName")))" + fi + if [[ -n "${excludeListRef[*]-}" ]]; then + # and not (element1) and not (element2) + excludeString="${includeString:+ and }not ($(concatStringsSep ") and not (" "$excludeListName"))" + fi + echo "$includeString$excludeString" +} + function pytestCheckPhase() { echo "Executing pytestCheckPhase" runHook preCheck # Compose arguments local -a flagsArray=(-m pytest) - if [ -n "${disabledTests[*]-}" ]; then - disabledTestsString="not $(concatStringsSep " and not " disabledTests)" - flagsArray+=(-k "$disabledTestsString") - fi - local -a _pathsArray=() + local -a _pathsArray + local path + + _pathsArray=() + concatTo _pathsArray enabledTestPaths + for path in "${_pathsArray[@]}"; do + if [[ "$path" =~ "::" ]]; then + flagsArray+=("$path") + else + # The `|| kill "$$"` trick propagates the errors from the process substitutiton subshell, + # which is suggested by a StackOverflow answer: https://unix.stackexchange.com/a/217643 + readarray -t -O"${#flagsArray[@]}" flagsArray < <(@pythonCheckInterpreter@ - "$path" <