Switch from appdirs to platformdirs #10202

domdfcoding · 2021-07-25T10:53:50Z

Changes of note:

`tests/unit/test_appdirs.py`

In appdirs, the platform was determined at import time but only checked when the functions were called. This allowed the global system attribute to be monkeypatched post-import to fake the current platform.

In platformdirs (post tox-dev/platformdirs#1) there are separate sets functions for each platform, and which ones get used is determined at import time. Therefore, although the system attribute exists, monkeypatching it no longer has the desired effect.

To work around this I have created three pytest fixtures (one each for Windows, macOS and Linux) which monkeypatch sys.platform and then reload both pip._vendor.platformdirs and pip._internal.utils.appdirs, so the import logic will run again. After the tests, the modules are re-reloaded to reset the state. If anyone can think of a better way I'm happy to hear it.

`pkg_resources`

Now patched as part of the vendoring process to use platformdirs.

macOS `site_config_dir`

Appdirs used to use /Library/Application Support/<app>, but it switched to /Library/Preferences/<app> in ActiveState/appdirs#100

Pip will now look in both locations for configuration

domdfcoding · 2021-07-26T16:25:06Z

Outstanding Issues

On Linux etc., pip patches appdirs to ignore an empty value for $XDG_CONFIG_DIRS and fallback to /etc/xdg/
Per https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html this looks to be the right way of doing it, so I have added a patch for platformdirs.

See also Linux: Empty environment variables should use defaults ActiveState/appdirs#130
Corrected *_config_dir() for OSX ActiveState/appdirs#100 changed appdirs.site_config_dir on macOS to use /Library/Preferences/ rather than /Library/Application Support/. There is a single test which checks the returned path and would need changing, but there's also the need to provide a migration path for users using the old path.

~~Shall I patch platformdirs to maintain the current behaviour for now and come back to this later?~~

uranusjr · 2021-07-26T21:59:41Z

For the site_config_dir issue, IMO we should do something similar to Linux and add both /Library/Application Support/ and /Library/Preferences/. The order is probably not significant (we can always fix it if someone complains), just pick the one that’s easier to implement.

pradyunsg · 2021-09-18T09:05:51Z

@domdfcoding Would it be possible for you to update this PR?

Notably, looks like we don't need to patch platformdirs anymore.

pradyunsg · 2021-09-22T04:32:34Z

A gentle nudge.

domdfcoding · 2021-09-22T20:48:11Z

Sorry, working on this now.

src/pip/_internal/utils/appdirs.py

pyproject.toml

tools/vendoring/patches/pkg_resources.patch

src/pip/_vendor/platformdirs/LICENSE.txt

tests/unit/test_appdirs.py

src/pip/_vendor/README.rst

pradyunsg · 2021-09-25T07:05:10Z

This PR needs a bunch of news/{name}.vendor.rst files, using all the package names, for the vendored packages which have been updated or otherwise modified.

Co-authored-by: Pradyun Gedam <[email protected]>

…t.mark.usefixtures`

Bumps [pip](https://github.com/pypa/pip) from 21.2.4 to 21.3. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p> <blockquote> <h1>21.3 (2021-10-11)</h1> <h2>Deprecations and Removals</h2> <ul> <li>Improve deprecation warning regarding the copying of source trees when installing from a local directory. (<code>[#10128](pypa/pip#10128) <https://github.com/pypa/pip/issues/10128></code>_)</li> <li>Suppress location mismatch warnings when pip is invoked from a Python source tree, so <code>ensurepip</code> does not emit warnings on CPython <code>make install</code>. (<code>[#10270](pypa/pip#10270) <https://github.com/pypa/pip/issues/10270></code>_)</li> <li>On Python 3.10 or later, the installation scheme backend has been changed to use <code>sysconfig</code>. This is to anticipate the deprecation of <code>distutils</code> in Python 3.10, and its scheduled removal in 3.12. For compatibility considerations, pip installations running on Python 3.9 or lower will continue to use <code>distutils</code>. (<code>[#10358](pypa/pip#10358) <https://github.com/pypa/pip/issues/10358></code>_)</li> <li>Remove the <code>--build-dir</code> option and aliases, one last time. (<code>[#10485](pypa/pip#10485) <https://github.com/pypa/pip/issues/10485></code>_)</li> <li>In-tree builds are now the default. <code>--use-feature=in-tree-build</code> is now ignored. <code>--use-deprecated=out-of-tree-build</code> may be used temporarily to ease the transition. (<code>[#10495](pypa/pip#10495) <https://github.com/pypa/pip/issues/10495></code>_)</li> <li>Un-deprecate source distribution re-installation behaviour. (<code>[#8711](pypa/pip#8711) <https://github.com/pypa/pip/issues/8711></code>_)</li> </ul> <h2>Features</h2> <ul> <li>Replace vendored appdirs with platformdirs. (<code>[#10202](pypa/pip#10202) <https://github.com/pypa/pip/issues/10202></code>_)</li> <li>Support <code>PEP 610 <https://www.python.org/dev/peps/pep-0610/></code>_ to detect editable installs in <code>pip freeze</code> and <code>pip list</code>. The <code>pip list</code> column output has a new <code>Editable project location</code> column, and the JSON output has a new <code>editable_project_location</code> field. (<code>[#10249](pypa/pip#10249) <https://github.com/pypa/pip/issues/10249></code>_)</li> <li><code>pip freeze</code> will now always fallback to reporting the editable project location when it encounters a VCS error while analyzing an editable requirement. Before, it sometimes reported the requirement as non-editable. (<code>[#10410](pypa/pip#10410) <https://github.com/pypa/pip/issues/10410></code>_)</li> <li><code>pip show</code> now sorts <code>Requires</code> and <code>Required-By</code> alphabetically. (<code>[#10422](pypa/pip#10422) <https://github.com/pypa/pip/issues/10422></code>_)</li> <li>Do not raise error when there are no files to remove with <code>pip cache purge/remove</code>. Instead log a warning and continue (to log that we removed 0 files). (<code>[#10459](pypa/pip#10459) <https://github.com/pypa/pip/issues/10459></code>_)</li> <li>When backtracking during dependency resolution, prefer the dependencies which are involved in the most recent conflict. This can significantly reduce the amount of backtracking required. (<code>[#10479](pypa/pip#10479) <https://github.com/pypa/pip/issues/10479></code>_)</li> <li>Cache requirement objects, to improve performance reducing reparses of requirement strings. (<code>[#10550](pypa/pip#10550) <https://github.com/pypa/pip/issues/10550></code>_)</li> <li>Support editable installs for projects that have a <code>pyproject.toml</code> and use a build backend that supports :pep:<code>660</code>. (<code>[#8212](pypa/pip#8212) <https://github.com/pypa/pip/issues/8212></code>_)</li> <li>When a revision is specified in a Git URL, use git's partial clone feature to speed up source retrieval. (<code>[#9086](pypa/pip#9086) <https://github.com/pypa/pip/issues/9086></code>_)</li> <li>Add a <code>--debug</code> flag, to enable a mode that doesn't log errors and propagates them to the top level instead. This is primarily to aid with debugging pip's crashes. (<code>[#9349](pypa/pip#9349) <https://github.com/pypa/pip/issues/9349></code>_)</li> <li>If a host is explicitly specified as trusted by the user (via the --trusted-host option), cache HTTP responses from it in addition to HTTPS ones. (<code>[#9498](pypa/pip#9498) <https://github.com/pypa/pip/issues/9498></code>_)</li> </ul> <h2>Bug Fixes</h2> <ul> <li>Present a better error message, when a <code>file:</code> URL is not found. (<code>[#10263](pypa/pip#10263) <https://github.com/pypa/pip/issues/10263></code>_)</li> <li>Fix the auth credential cache to allow for the case in which the index url contains the username, but the password comes from an external source, such as keyring. (<code>[#10269](pypa/pip#10269) <https://github.com/pypa/pip/issues/10269></code>_)</li> <li>Fix double unescape of HTML <code>data-requires-python</code> and <code>data-yanked</code> attributes. (<code>[#10378](pypa/pip#10378) <https://github.com/pypa/pip/issues/10378></code>_)</li> <li>New resolver: Fixes depth ordering of packages during resolution, e.g. a dependency 2 levels deep will be ordered before a dependecy 3 levels deep. (<code>[#10482](pypa/pip#10482) <https://github.com/pypa/pip/issues/10482></code>_)</li> </ul>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pypa/pip/commit/abec8a701bfa66aa15fedf4c898011aa2d95f29e"><code>abec8a7</code></a> Bump for release</li> <li><a href="https://github.com/pypa/pip/commit/68a70486c9224f9d25be3cbf56c73d8a33c6a713"><code>68a7048</code></a> Update AUTHORS.txt</li> <li><a href="https://github.com/pypa/pip/commit/9f18a403ca41f4e42fbb89d286b6571a099cb54b"><code>9f18a40</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/10481">#10481</a> from notatallshaw/prefer_failures</li> <li><a href="https://github.com/pypa/pip/commit/db496cbce518fa159476695db0cd4f1c1a8ab6f5"><code>db496cb</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/10563">#10563</a> from pradyunsg/shorter-timeout</li> <li><a href="https://github.com/pypa/pip/commit/4fac2b90a5d200b46e7b576013bb25f4ebb3f937"><code>4fac2b9</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/10550">#10550</a> from jbylund/joe/cache_requirement_creation</li> <li><a href="https://github.com/pypa/pip/commit/786957cf85a641d49b4cfcceef717ef229ac8238"><code>786957c</code></a> Use a shorter timeout, to ensure that this fails more often</li> <li><a href="https://github.com/pypa/pip/commit/1e3c127d4a938643aca1bbc25e6581493e316476"><code>1e3c127</code></a> Avoid passing <code>.</code> to vendoring</li> <li><a href="https://github.com/pypa/pip/commit/610424f9f8ad1f99d0a48bf9a53e7a9df4242304"><code>610424f</code></a> Quote "PreferenceInformation" to avoid runtime NameError</li> <li><a href="https://github.com/pypa/pip/commit/c01b5c6d8a4858cf733408b4b020933f902dda9e"><code>c01b5c6</code></a> Update a test for resolvelib 0.8.0</li> <li><a href="https://github.com/pypa/pip/commit/394a24eb1a5f9af5da7d4d2452ed5fe952de5db2"><code>394a24e</code></a> Upgrade resolvelib to 0.8.0</li> <li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/21.2.4...21.3">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=21.2.4&new-version=21.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details>

domdfcoding mentioned this pull request Jul 26, 2021

Linux: Empty environment variables should use defaults tox-dev/platformdirs#29

Closed

domdfcoding changed the title ~~[WIP] Switch from appdirs to platformdirs~~ Switch from appdirs to platformdirs Jul 27, 2021

domdfcoding marked this pull request as ready for review July 27, 2021 19:20

uranusjr added this to the 21.3 milestone Jul 28, 2021

github-actions bot added the needs rebase or merge PR has conflicts with current master label Aug 15, 2021

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Sep 22, 2021

domdfcoding force-pushed the platformdirs branch from d8ed3c1 to 60e294d Compare September 22, 2021 21:27

uranusjr reviewed Sep 23, 2021

View reviewed changes

src/pip/_internal/utils/appdirs.py Show resolved Hide resolved

pradyunsg reviewed Sep 24, 2021

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

tools/vendoring/patches/pkg_resources.patch Show resolved Hide resolved

pradyunsg mentioned this pull request Sep 24, 2021

Jython support tox-dev/platformdirs#43

Closed

Julian reviewed Sep 24, 2021

View reviewed changes

src/pip/_vendor/platformdirs/LICENSE.txt Show resolved Hide resolved

Julian reviewed Sep 24, 2021

View reviewed changes

tests/unit/test_appdirs.py Show resolved Hide resolved

pradyunsg reviewed Sep 24, 2021

View reviewed changes

src/pip/_vendor/README.rst Outdated Show resolved Hide resolved

pradyunsg added project: vendored dependency Related to a vendored dependency type: enhancement Improvements to functionality labels Sep 25, 2021

github-actions bot added the needs rebase or merge PR has conflicts with current master label Sep 26, 2021

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Sep 30, 2021

domdfcoding and others added 7 commits September 30, 2021 15:34

Switch from appdirs to platformdirs

ae6cf66

Update pkg_resources patch.

f2d29df

Lint test_appdirs.py

f2ff6ef

Bump platformdirs version.

d031c64

Downgrade platformdirs to 2.2.0

e498cee

Remove duplicate monkeypatches

147e8a8

Add note to _vendor/README.rst about pkg_resources patch.

8de9e0a

domdfcoding and others added 4 commits September 30, 2021 15:37

Remove "platformdirs" from tool.vendoring.typing-stubs

a36a2c5

Update src/pip/_vendor/README.rst

ea8aad7

Co-authored-by: Pradyun Gedam <[email protected]>

Add news entries.

188266b

Reformat test_appdirs.py, add type annotations, and switch to `@pytes…

78b8b8b

…t.mark.usefixtures`

domdfcoding force-pushed the platformdirs branch from 5de06b4 to 78b8b8b Compare September 30, 2021 15:57

pradyunsg approved these changes Oct 2, 2021

View reviewed changes

pradyunsg merged commit 7616583 into pypa:main Oct 2, 2021

github-actions bot locked as resolved and limited conversation to collaborators Oct 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch from appdirs to platformdirs #10202

Switch from appdirs to platformdirs #10202

domdfcoding commented Jul 25, 2021 •

edited

Loading

domdfcoding commented Jul 26, 2021 •

edited

Loading

uranusjr commented Jul 26, 2021

pradyunsg commented Sep 18, 2021 •

edited

Loading

pradyunsg commented Sep 22, 2021

domdfcoding commented Sep 22, 2021

pradyunsg commented Sep 25, 2021

Switch from appdirs to platformdirs #10202

Switch from appdirs to platformdirs #10202

Conversation

domdfcoding commented Jul 25, 2021 • edited Loading

tests/unit/test_appdirs.py

pkg_resources

macOS site_config_dir

domdfcoding commented Jul 26, 2021 • edited Loading

Outstanding Issues

uranusjr commented Jul 26, 2021

pradyunsg commented Sep 18, 2021 • edited Loading

pradyunsg commented Sep 22, 2021

domdfcoding commented Sep 22, 2021

pradyunsg commented Sep 25, 2021

domdfcoding commented Jul 25, 2021 •

edited

Loading

`tests/unit/test_appdirs.py`

`pkg_resources`

macOS `site_config_dir`

domdfcoding commented Jul 26, 2021 •

edited

Loading

pradyunsg commented Sep 18, 2021 •

edited

Loading