Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parse dependencies from Python setup.cfg files #2281

Closed
wants to merge 1 commit into from
Closed

Parse dependencies from Python setup.cfg files #2281

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions python/helpers/lib/parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
from distutils.core import run_setup
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Best to avoid using distutils, we're trying to actively deprecate it.

Copy link
Author

@akaihola akaihola Jul 20, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, so I take it that I should close this PR?

Is there an alternative best practice for getting the dependencies of a Python package which uses setuptools, no matter whether the dependencies are listed in setup.py or setup.cfg?

If there is, I'd like to add it as an answer to this Stack Overflow question which is one of the top search results when looking for solutions to read a Python package's dependencies.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This question is actually a bit difficult to answer. Assuming you actually just want to get the dependencies, for setuptools projects, the biggest issue you'll have is that many projects unfortunately don't know about (or haven't updated to use) environment markers, so it's distressingly common to see stuff like this:

requires = ["some_dependency"]
if sys.version_info[0] < 2:
     requires.append("some_py2_only_dependency")

E.g. from s3transfer. This means that depending on the machine where you run setup.py, you will get a different answer for the values of install_requires. The correct way to do this is to declare a fixed set of dependencies like so:

some_dependency
some_py2_only_dependency ; python_version < '2.7'

In which case even the conditional dependencies will be included in the wheel metadata.

Assuming that you are comfortable with accepting "whatever will be included in the wheel if I were to build it in whatever worker environment" (seems reasonable), then I would say that the best thing to do is to implement enough of a PEP 517/518 backend to execute prepare_metadata_for_build_wheel, and parse the dependency metadata from the .dist-info/METADATA file. The pep517 library should make this easy enough.

If the project doesn't have a pyproject.toml file, I recommend defaulting to use build-backend=setuptools.build_meta:__legacy__ and requires = ["setuptools >= 40.8.0", "wheel"], as we're doing in the python-build frontend. I think the majority of setuptools projects will work properly with these defaults. You can also fall back to using pip wheel to generate an actual wheel file and then extract the metadata from there, though that will consume more resources than necessary.

Using PEP 517 also has the advantage that it works automatically with any PEP 517 backend, not just setuptools. (We in the PyPA would also probably not be unhappy if you said, "We only support packages that work well with PEP 517", as a further spur to get people to actively adopt the new build system).

That said, I'm not exactly sure why you want this sort of generic information about a package's depdencies — my understanding of dependabot is that it is supposed to update your dependencies, no? So don't you need to know not only what they are, but also exactly where they are defined, so that you can make a PR to automatically update them?

Copy link
Author

@akaihola akaihola Jul 20, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my understanding of dependabot is that it is supposed to update your dependencies,

GitHub's Dependency Graph is implemented using dependabot. The way I arrived at this issue is that I wanted to simply see my project's dependencies in the GitHub UI and was surprised to see this message instead:

setup.py has no dependencies or is too large to display

So GitHub's Dependency Graph simply doesn't work for any Python packages which declare setup_requires in setup.cfg instead of setup.py.

I then found the tip to use distutils.core.run_setup(..., stop_after="init").install_requires and figured that dependabot could do that, and also as a side effect gain a possibly more robust way to extract dependencies from setup.py files than the current parse_setup() method which modifies setup.py and exec()s it in a patched Python environment. parse_setup() could still be used as a fallback for those cases where run_setup() fails.

It sounds like a great idea to make GitHub's Dependency Graph only work on modern well-formed Python packages. It should indeed encourage authors to adopt good practices in packaging. The current situation actually does the reverse – it discourages adopting e.g. a declarative setup.cfg and pyproject.toml and favors old-style packages with a traditional executable setup.py

Update: @jurre noted below that GitHub's Dependency Graph actually isn't based on dependabot at all. I must have misunderstood some information I've read about the Dependency Graph.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still eager to help make GitHub's Dependency Graph work for modern Python repositories. @pganssle could you advise what would be the most efficient and acceptable way for me to contribute?

Copy link

@graingert graingert Apr 11, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

import build.util
import importlib.metadata
import pathlib

def parse_from_setup_py(v: pathlib.Path) -> importlib.metadata.PackageMetadata:
    return build.util.project_wheel_metadata(v.parent)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, but I'm a complete Python noob, and I'm running into a bunch of missing import errors that I cannot seem to solve (on Python 3.6). So what I'd need it a complete example / standalone .py file that takes a directory / file as an argument and works out the box 🙄

Copy link

@graingert graingert Apr 11, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sschuberth what errors are you getting? Python 3.6 is EOL: you should try this on a recent version of python eg 3.9 or 3.10

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm stuck with Python 3.6, and I'm getting NameError for pathlib and importlib not being defined, AttributeError: module 'importlib' has no attribute 'metadata' after adding imports etc. I'll give https://stackoverflow.com/a/71276197/1127485 a try now.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For older versions, you have to install and use the backport importlib_metadata (notice the underscore)

from itertools import chain
import glob
import io
Expand Down Expand Up @@ -90,6 +91,21 @@ def setup(*args, **kwargs):
for key in extras_require_dict:
for req in extras_require_dict[key]:
parse_requirement(req, 'extras_require:{}'.format(key))

try:
# Use distutils.core.run_setup() - this will also parse setup.cfg
setup_result = run_setup(directory + "/setup.py", stop_after="init")
except Exception:
# Try with the original exec() method below
pass
else:
setup(
setup_requires=getattr(setup_result, "setup_requires", None),
install_requires=getattr(setup_result, "install_requires", None),
tests_requires=getattr(setup_result, "tests_requires", None),
extras_require=getattr(setup_result, "extras_require", None),
)

setuptools.setup = setup

def noop(*args, **kwargs):
Expand Down Expand Up @@ -132,7 +148,8 @@ def fake_open(*args, **kwargs):
# Run as main (since setup.py is a script)
__name__ = '__main__'

# Exec the setup.py
exec(content) in globals(), locals()
# Exec the setup.py if distutils.core.run_setup() above didn't succeed
if not setup_packages:
exec(content) in globals(), locals()

return json.dumps({ "result": setup_packages })