Please remove the dependency on 'py'

[mgorny]

Since 'py' isn't really receiving any attention, and distributions end up carrying growing pile of patches just to keep it working, I think it'd be reasonable for pytest to stop requiring it. Just to be clear, right now we need to patch tests to work with pytest-4+, we hackaround breakages in python3.8 and python3.9 is around the corner with yet another set of problems.

[RonnyPfannschmidt]

We are working on it, it's still going to take years

[bluetech]

In master, we are down to depending on only two parts of py: py.path.local and py.iniconfig (and one instance of py.error.Error which I can remove). Migrating from py.path.local would require a big effort, but py.iniconfig can be imported quite easily. WDYT @RonnyPfannschmidt?

[RonnyPfannschmidt]

Ini config is already externalized, no need to copy into pytest

[kerizane]

Is the plan documented anyway? Any preferences or guidelines?

[RonnyPfannschmidt]

the plan is not documented anywhere, in detail yet, we created https://github.com/pytest-dev/pytest/projects/4 to de-tangle and track it

[Zac-HD]

Personally I'd be happy to have a hard-cutover to pathlib in pytest 6.0

It's a significant change and may be a breaking one for some users and plugins, but I think likely to cause less churn in dependencies than trying to invent a compatibility layer. I estimate that almost no end-users will be directly affected, and only a small fraction of plugins - and importantly that it will be easier on pytest maintainers to help plugins migrate than to have an upstream compat layer.

[nicoddemus]

It's a significant change and may be a breaking one for some users and plugins

Oh no it would break the world: tmpdir (a very used fixture) is a py.path.local object.

[Zac-HD]

Ah. Yes, that would do it 🤦‍♂️

[RonnyPfannschmidt]

Personally i will reject a compatibility layer beyond casting some objects to local paths and leaving behind the legacy properties which do

Support for py.path.local will then be constrained to the python versions where pylib works and pylib will turn into a optional dependency that's declared dead

[bluetech]

In master, we are down to depending on only two parts of py: py.path.local and py.iniconfig

For some reason, one slipped my mind: py.xml used in the junitxml plugin.

[nicoddemus]

Oh right. That will be a simple, but arduous refactoring.

[kerizane]

junitxml already has a ticket:
#4970

[gnikonorov]

Question. What is the reason for py’s existence? Were the features it added just not available in standard python at the time or was it optimized for pytest?

[nicoddemus]

The first one: it dates before/around Python 3.0 was released, so it contained a bunch of utilities that worked in both Python 2 and Python 3.0 at the time, plus general utilities such as xml manipulation, file first-class objects, I/O, and including py.test, which was the early stages of pytest. pytest was in fact born by extracting py.test from py.

Since then, there are better alternatives for all of which is implemented there, that's why we have no interest in adding new features to it, or even maintaining it if possible.

[gnikonorov]

Thank you for the explanation @nicoddemus

[bluetech]

As mentioned in this comment, I have looked more closely at what remains to be done before we can deprecate py.path. Of course, before we can deprecate, there must be replacements for everything which takes or returns a py.path. After all of our recent PRs, I created a branch which removes all py.path functionality for which there exist replacements (i.e. can be readily deprecated), here: https://github.com/bluetech/pytest/commits/rm-py. That branch passes all tests and lints which ensures I don't miss anything.

There are only two remaining py.path usages which cannot be easily handled:

pathlist

The pathlist ini type (as in parser.addini("mypaths", "my ini option", type="pathlist")). This type takes a line list and returns the values as a list of paths relative to the ini file. This type is not documented anywhere that I can see.

It is not possible to fix a return type, so this would either need to be removed, replaced or broken without deprecation.

Plugins which use this

pytest-pythonpath/pytest_pythonpath.py
9-    # py.test has an issue where the cwd is not in the PYTHONPATH. Fix it here.
10-    if os.getcwd() not in sys.path:
11-        sys.path.insert(0, os.getcwd())
12:    parser.addini("python_paths", type="pathlist", help="space seperated directory paths to add to PYTHONPATH via sys.path.insert(0, path)",
13-                  default=[])
14:    parser.addini("site_dirs", type="pathlist", help="space seperated directory paths to add to PYTHONPATH via site.addsitedir(path)",
15-                  default=[])
16-
17-

pytest-cloud/pytest_cloud/plugin.py
174-    parser.addini(
175-        "cloud_develop_eggs",
176-        "list of python package paths to install in develop mode on the remote side",
177:        type="pathlist",
178-    )
179-
180-

pytest-srcpaths/pytest_srcpaths.py
7-
8-def pytest_addoption(parser):
9-    parser.addini("srcpaths",
10:                  type="pathlist",
11-                  help="Add paths to sys.path")
12-
13-

pytest-xdist/src/xdist/plugin.py
142-    parser.addini(
143-        "rsyncdirs",
144-        "list of (relative) paths to be rsynced for remote distributed testing.",
145:        type="pathlist",
146-    )
147-    parser.addini(
148-        "rsyncignore",
149-        "list of (relative) glob-style paths to be ignored for rsyncing.",
150:        type="pathlist",
151-    )
152-    parser.addini(
153-        "looponfailroots",
154:        type="pathlist",
155-        help="directories to check for changes",
156-        default=[py.path.local()],
157-    )

pytest-postgresql/src/pytest_postgresql/plugin.py
100-
101-    parser.addini(
102-        name='postgresql_load',
103:        type='pathlist',
104-        help=_help_load,
105-        default=None
106-    )

reportinfo

This is a method that needs to be implemented by custom Items. One of its return values is a py.path.

Most plugins implement the method, which is normally called by pytest, but some plugins do call it. So the py.path is in both a covariant and contravariant position.

If we only care about the plugin-implements case, then we can change the contract to "returns path-like object" instead of "returns py.path". If we also care about the plugin-calls case, then we need to come up with another solution.

Plugins which use this

pytest-isort/pytest_isort.py
184:    def reportinfo(self):

pytest-pycodestyle/src/pytest_pycodestyle.py
85:    def reportinfo(self):

pytest-pycodestyle/tests/test_pytest_pycodestyle.py
118:    def test_reportinfo(self):

pytest-ghostinspector/pytest_gi/plugin.py
241:    def reportinfo(self):

pytest-qml/src/pytestqml/nodes.py
130:    def reportinfo(self):

pytest-alembic/src/pytest_alembic/plugin/plugin.py
105:    def reportinfo(self):

pytest-codestyle/tests/test_pytest_pycodestyle.py
118:    def test_reportinfo(self):

pytest-codestyle/src/pytest_pycodestyle.py
85:    def reportinfo(self):

pytest-check-links/pytest_check_links/plugin.py
223:    def reportinfo(self):

pytest-gherkin/pytest_gherkin/gherkin.py
174:    def reportinfo(self):

pytest_oot/pytest_oot/oot_plugin.py
223:    def reportinfo(self):

pytest-flake8/pytest_flake8.py
141:    def reportinfo(self):

pytest-eradicate/pytest_eradicate.py
84:    def reportinfo(self):

pytest-asptest/asptest/classes.py
171:    def reportinfo(self):

pytest-freeze-reqs/freeze_reqs/pytest_freeze_reqs.py
99:    def reportinfo(self):

pytest-mypy-plugins/pytest_mypy_plugins/item.py
352:    def reportinfo(self) -> Tuple[Union[py.path.local, str], Optional[int], str]:

pytest-jasmine/pytest_jasmine.py
157:    def reportinfo(self):

pytest-notebook/pytest_notebook/plugin.py
306:    def reportinfo(self):

pytest-checkipdb/pytest_checkipdb.py
47:    def reportinfo(self):

pytest-pydocstyle/tests/test_pytest_pydocstyle.py
131:    def test_reportinfo(self):

pytest-pydocstyle/src/pytest_pydocstyle.py
104:    def reportinfo(self):

python-pytest-cases/pytest_cases/case_parametrizer_new.py
827:#             fspath, lineno, _ = item.reportinfo()

pytest-bandit/pytest_bandit/controller.py
95:    def reportinfo(self):

pytest-mypy/src/pytest_mypy.py
176:    def reportinfo(self):

pytest-translations/pytest_translations/po_spelling.py
123:    def reportinfo(self):

pytest-translations/pytest_translations/mo_files.py
85:    def reportinfo(self):

pytest-translations/pytest_translations/po_files.py
83:    def reportinfo(self):
101:    def reportinfo(self):
119:    def reportinfo(self):

pytest-black/pytest_black.py
89:    def reportinfo(self):

agent-python-pytest/pytest_reportportal/service.py
717:            return test_item.reportinfo()[2]

pytest-mccabe/pytest_mccabe.py
92:    def reportinfo(self):

pytest-requests/pytest_python_requests.py
418:    def reportinfo(self):

pytest-reqs/pytest_reqs.py
160:    def reportinfo(self):

pytest-dash/pytest_dash/behaviors.py
85:    def reportinfo(self):

pytest-docstyle/src/pytest_pydocstyle.py
104:    def reportinfo(self):

pytest-docstyle/tests/test_pytest_pydocstyle.py
131:    def test_reportinfo(self):

pytest-yapf3/src/pytest_yapf3/__init__.py
147:    def reportinfo(self):

pytest-pylint/pytest_pylint/plugin.py
391:    def reportinfo(self):

pytest-molecule/pytest_molecule/__init__.py
233:    def reportinfo(self):

pytest-contexts/pytest_contexts.py
31:    def reportinfo(self):
64:    def reportinfo(self):

dbt-adapter-tests/pytest_dbt_adapter/spec_file.py
689:    def reportinfo(self):

pytest-mypy-testing/src/pytest_mypy_testing/plugin.py
74:    def reportinfo(self) -> Tuple[str, Optional[int], str]:

allure-pytest/allure/pytest_plugin.py
115:            description = item.reportinfo()[2]

pytest-cpp/src/pytest_cpp/plugin.py
155:    def reportinfo(self):

pytest-flakes/pytest_flakes.py
102:    def reportinfo(self):

pytest-cram/pytest_cram/__init__.py
83:    def reportinfo(self):

[RonnyPfannschmidt]

i wonder if we can revise the concept of the reportinfo while ridding us of py.path

i'd also like to rid pytest of the copy of pylibs exceptioninfo to bring in better features for tracebacks on async, twisted and frameworks in general

[bluetech]

As for pathlist (see comment #7259 (comment) above), I think we can add a paths INI type (parser.addini("mypaths", "my ini option", type="paths")) to replace pathlist. This would of course return a list of Paths instead of py.paths. Then we could deprecate pathlist.

Another alternative is to just deprecate pathlist without replacement, it's not used that much and not really hard to implement oneself.

I think I will submit a PR to add paths unless someone has other preferences.

i wonder if we can revise the concept of the reportinfo while ridding us of py.path

Right, for reportinfo there doesn't seem to be simple solutions. This ^ would need some design work.

[bluetech]

OK, here is my suggestion:

For 6.3.0 (next minor release)

• Add a single PendingDeprecationWarning type for py.path. Go over all py.path usages and add it. Message for each case would be different and indicate the suggested replacement. It would be possible to ignore the entire class based on the type, or specific cases based on the message.
• Add a section to the deprecation docs with a rationale, the deprecation timeline and listing all of the deprecated py.path usages and their replacements (single central place to link to), along with the version from which the replacement is available.
• Add a note to the release notes indicating the deprecation has began and linking to the doc.

One hitch here is reportinfo which has no replacement but I suggest we punt it for now -- better than nothing. Of course it won't be marked deprecated until there is a replacement.

For 7.0.0 (next major release)

Change the PendingDeprecationWarning to a DeprecationWarning.

For 8.0.0

Remove. I estimate this would be 2-3 years from now.

---

If this sounds OK, I can do the work for the 6.3.0 release.

The idea here is just to give ample time to transition.

Does anyone have alternative ideas on how we could reduce the pain here? The audiences are:

• Users: presumably we want them to just pin or require a new pytest version and convert all tmpdir to tmp_path etc. and be done with it. Can we ease these migrations?

• Plugins 1: they often would want to support older versions which don't have full pathlib support. Should we just rely on the long deprecation period + dropping support for older versions, or can we offer some way to do it without dropping compat?

• Plugins 2: some probably have their own APIs which expose py.path. Can we aid them in migrating to pathlib?

I know @RonnyPfannschmidt has ideas around legacy_path etc. but I don't think I've seen the exact idea/plan.

[RonnyPfannschmidt]

@bluetech the basic idea of legacy_path is that we can turn the pylib dependency optional at some point while leaving the deprecated calls in - a legacy_path would eventually act "like" a py.path.local but only pylib is installed

so in sense we would pass around "py.path.local" tokens that only turn into a py.path.local when actually used

exposing legacy_path would also help them to migrate

beleive the basic idea should be to help everyone migrate off py.path.local as soon as possible and investigate how other plugins can have that easy (maybe by providing some extra collection classes in a compat package)

[The-Compiler]

FWIW I'm strongly opposed to adding more to the 6.3.0 (or 7.0.0, if we decide to go for that) backlog unless absolutely required. We haven't had a feature release in more than half a year, and adding more stuff rather than finishing what we've started (mostly #8435 at this point?) is only going to make things worse.

[RonnyPfannschmidt]

Thanks for the check, I'll aim at finishing that by the end of the week and let's push the rest to 6.4/7

[nicoddemus]

Thanks @bluetech for the write up and bringing this up to attention again.

• Add a single PendingDeprecationWarning type for py.path. Go over all py.path usages and add it. Message for each case would be different and indicate the suggested replacement. It would be possible to ignore the entire class based on the type, or specific cases based on the message.

👍

• Add a section to the deprecation docs with a rationale, the deprecation timeline and listing all of the deprecated py.path usages and their replacements (single central place to link to), along with the version from which the replacement is available.
• Add a note to the release notes indicating the deprecation has began and linking to the doc.

👍

Just to mention that IMHO deprecation docs (both rationale and how to port the code) are absolutely the most important thing we should do regarding deprecations. Normal users are just annoyed by deprecations ("my code was working"), so having a good, concise explanations and even better a simple copy/paste suggestionst to migrate does wonders here.

Users: presumably we want them to just pin or require a new pytest version and convert all tmpdir to tmp_path etc. and be done with it. Can we ease these migrations?

Unfortunately this will be a huge huge pain. There are very large suites out there that make extensive use of tmpdir, and porting to tmp_path is a trivial but manually intensive labor. Suggesting to pin to old versions is not a solution, as those are "live" code bases that want to use the latest pytest.

We have discussed this before, but IIRC the best proposal to ease that pain is to create a pytest-tmpdir or pytest-pypath-legacy plugin, which provides the tmpdir fixture back. That plugin could provide a shim to the py.path API, using pathlib.Path as the implementation. It should be maintained by the core team, IMHO. The advantage of that being a plugin (instead of keeping tmpdir in the core using a shim) is that we don't need to keep the old API around pytest itself, plus a plugin can make less guarantees about the shim compatibility: by that I mean "behaves the same" but some things might be slightly different, say some operation might leave a "/" at the end which doesn't happen in the original py.path, but should not affect normal usage otherwise.

the basic idea of legacy_path is that we can turn the pylib dependency optional at some point while leaving the deprecated calls in - a legacy_path would eventually act "like" a py.path.local but only pylib is installed

Hmm but then we now have to support that mechanism long term as well... until now I thought of legacy_path more as a central place to keep py.path stuff for easy removal later, rather than being actually a feature.

Thanks for the check, I'll aim at finishing that by the end of the week and let's push the rest to 6.4/7

Agree, we should avoid pilling up more into 6.3.0. 👍

[The-Compiler]

Unfortunately this will be a huge huge pain. There are very large suites out there that make extensive use of tmpdir, and porting to tmp_path is a trivial but manually intensive labor.

We could try to write a tool in the spirit of unittest2pytest or hypothesis codemod to automate this to the extent that's possible. LibCST seems like an excellent base for something like this, but I have no real experience with it.

For reference, here's hypothesis' implementation: codemods.py

[nicoddemus]

From #7259 (comment):

Rollout of plugin

It would be great if we could issue warnings for the functions provided by the plugin, indicating to install the standalone version. This would mean keeping both internal and external plugins "around" for a bit, which would be a bit more work but provide a more smooth transition (and chance to catch early problems).

[RonnyPfannschmidt]

closing as completed as we removed the py dependency when the CVE did hit

[The-Compiler]

Didn't we have some further plans around this? I thought at some point we wanted to get rid of our py shim, possibly deprecating and then removing things in your legacypath built-in plugin, no?

[RonnyPfannschmidt]

Indeed, but we completed the removal of the actual dependency

[bluetech]

The proposed deprecation plan was here: #7259 (comment)

It should be mostly straightforward code-wise to deprecate and move to external plugin for those who still want it. At least after 8.1 removes various related already-deprecated functionality like the path parameters in hooks.

But for me at least the main reason for doing so was to remove the py dependency, and since we already did this then it's starting to look less appealing to put people through the deprecation pain (as @nicoddemus said, the py.path stuff is used a lot so this will be painful).

[RonnyPfannschmidt]

We have to put local path to a rest, it's a White elephant

[nicoddemus]

The pain in this case is for users that still use tmpdir in their code bases to add pytest-tmpdir to their dependencies... I think it is worthy tradeoff, as it has the big benefit of plain pytest installations no longer having tmpdir, which might point inexperienced users in the wrong direction otherwise.

Another point to consider is the absolute ton of content out there that expects tmpdir to be builtin, and will be out-dated.

[RonnyPfannschmidt]

It's important to put into perspective that pyllib was a good idea in 2005, we are now close to 2025, its not been aging well