Testing Python packages which rely on entry points defined in the same package #11386
Labels
Comments
|
@michaelboulton : Hey! Sorry for the delayed response. When you say that "it relies on some entry points", do you mean that you are not able to call those entry points from within a test? Or are you expecting the entry point code to have run by the time the test does? If your If there is code that you want to run every time If I'm way off the mark here, please feel free to drop into Slack to discuss it! |
|
I'm guessing @michaelboulton means this: https://setuptools.readthedocs.io/en/latest/userguide/entry_point.html#console-scripts @michaelboulton this isn't supported at the moment. Pants can produce a distribution as outlined here: https://www.pantsbuild.org/docs/python-distributions. That distribution can include console scripts via This feature probably makes sense to support but will require some design and code work if I've understood your use case correctly. * Pants does install third party packages when they are dependencies (requirements) of your first party Python code; but, even then, the current method of installation does not properly expose third party console scripts**. ** Pants uses Pex to resolve third party requirements and recent changes in Pex will allow Pants to properly expose third party console scripts on the PATH of Python processes it executes. |
It does not, but you can include the built package in your chroot as a PEX file! See https://www.pantsbuild.org/docs/python-test-goal#depending-on-packages |
That's a nice feature and may serve as a workaround here. It is bloated though. For a python package with many console scripts - say 3, and a test that tests them all, this would require building 3 pexes to include in the test chroot with identical contents save for the default entrypoint. |
|
From reading the discussion I'm guessing what I want isn't supported yet - for reference I'm looking for support for the |
|
Aha - thanks for the clarification @michaelboulton. Indeed - we have no way of supporting that today. For that, the |
|
@Eric-Arellano this should be doable now with |
|
As a workaround, but I think the feature request is different, per John's reply:
|
|
I think this is doable - suboptimally again, but less so - with a dependency on a Unfortunately, I think this is all still clunky to get setup correctly. A naive attempt at a demo in the Pants repo: Nets: This is because local dists are not 1st class and we don't follow their dependencies: |
|
I've workarounded this issue by writing a custom entrypoint scanner based on BUILD files.: |
|
I posted a possible solution on #17308 In that if the entry_points.txt is wrapped in a resources() target and the test depends on it, then it should be on the sys.path at test time. |
|
Exposing entry points is actually fairly easy even without installing the package or using I wrote a rule in #18132 that generates an |
cognifloyd
added a commit
that referenced
this issue
Apr 10, 2023
## About Editable Installs Editable installs were traditionally done via pip with `pip install --editable`. It is primarily useful during development when software needs access to the entry points metadata. When [PEP 517](https://peps.python.org/pep-0517/) was adopted, they punted on how to allow for editable installs. [PEP 660](https://peps.python.org/pep-0660/) extended the PEP 517 backend API to support building "editable" wheels. Therefore, there is now a standard way to collect and install the metadata for "editable" installs, using the "editable" wheel as an interchange between the backend (which collects the metadata + builds the editable wheel) and the frontend (which marshals the backend to perform a user-requested "editable" install). ## Why would we need editable installs in pants? I need editable installs in pants-exported virtualenvs so that dev tools outside of pants have access to: - The locked requirements - The editable sources on the python path - The entry points (and any other package metadata that can be loaded from `dist-info`. Entry points is the biggest most impactful example) I need to point all the external dev tooling at a virtualenv, and technically I could export a pex that includes all of the python-distributions pre-installed and use pex-tools to create a virtualenv, but then I would have to recreate that venv for every dev change wouldn't be a good dev experience. One of those dev tools is `nosetest`. I considred using `run` to support running that, but I am leary of adding all the complex BUILD machinery to support running a tool that I'm trying to get rid of. Editable installs is a more generic solution that serves my current needs, while allowing for using it in other scenarios. This PR comes in part from #16621 (comment) ## Overview of this PR ### Scope & Future work This PR focuses on adding editable installs to exported virtualenvs. Two other issues ask for editable installs while running tests: - #11386 - #15481 We can probably reuse a significant portion of this to generate editable wheels for use in testing as well. Parts of this code will need to be refactored to support that use case. But we also have to figure out the UX for how users will define dependencies on a `python_distribution` when they want an editable install instead of the built wheel to show up in the sandbox. Anyway, addressing that is out of scope for this PR. ### New option `[export].py_editables_in_resolves` (a `StrListOption`) This option allows user resolves to opt in to using the editable installs. After [consulting](https://pantsbuild.slack.com/archives/C0D7TNJHL/p1680810411706569?thread_ts=1680809713.310039&cid=C0D7TNJHL) with @kaos, I decided to add an option for this instead of always trying to generate/install the editable wheels. > `python_distribution` does not have a `resolve` field. So figuring out which resolve a `python_distribution` belongs to can be expensive: calculating the owned deps of all distributions, and for each distribution, look through those deps until one of them has a resolve field, and use that for that dist’s resolve. > > Plus there’s the cost of building the PEP-660 wheels - if the configured PEP-517 build backend doesn’t support the PEP-660 methods, then it falls back to a method that is, sadly, optional in PEP-517. If that method isn’t there, then it falls back to telling that backend to build the whole wheel, and then it extracts just the dist-info directory from it and discards the rest. > > So, installing these editable wheels isn’t free. It’ll slow down the export, though I’m not sure by how much. For StackStorm, I plan to set this in `pants.toml` for the default resolve that has python_distributions. Even without this option, I tried to bail out early if there were no `python_distribution`s to install. ### Installing editable wheels for exports I added this feature to the new export code path, which requires using `export --resolve=`. The legacy codepath, which uses cli specs `export <address specs>` did not change at all. I also ignored the `tool` resolves which cannot have any relevant dists (and `tool` resolves are deprecated anyway). Also, this is only for `mutable_virtualenv` exports, as we need modify the virtualenv to install the editable wheels in the venv after pex creates it from the lockfile. When exporting a user resolve, we do a `Get(EditableLocalDists, EditableLocalDistsRequest(resolve=resolve))`: _I'll skip over exactly how this builds the wheels for now so this section can focus on how installing works._ https://github.com/pantsbuild/pants/blob/f3a4620e81713f5022bf9a2dd1a4aa5ca100d1af/src/python/pants/backend/python/goals/export.py#L373-L379 As described in the commit message of b5aa26a, I tried a variety of methods for installing the editable wheels using pex. Ultimately, the best I came up with is telling pex that the digest containing our editable wheels are `sources` when building the `requirements.pex` used to populate the venv, so that they would land in the virtualenv (even though they land as plain wheel files. Then we run `pex-tools` in a `PostProcessingCommand` to create and populate the virtualenv, just as we did before this PR. Once the virtualenv is created, we add 3 more `PostProcessingCommands` to actually do the editable install. In this step, Pants is essentially acting as the PEP-660 front end, though we use pip for some of the heavy lifting. These commands: 1. move the editable wheels out of the virtualenv lib directory to the temp dir that gets deleted at the end of the export 2. use pip to install all of the editable wheels (which contain a `.pth` file that injects the source dir into `sys.path` and a `.dist-info` directory with dist metadata such as entry points). 3. replace some of the pip-generated install metadata (`*.dist-info/direct_url.json`) with our own so that we comply with PEP-660 and mark the install as editable with a file url pointing the the sources in the build_root (vs in a sandbox). Now, anything that uses the exported venv should have access to the standardized package metadata (in `.dist-info`) and the relevant source roots should be automatically included in `sys.path`. ### Building PEP-660 editable wheels The logic that actually builds the editable wheels is in `pants.backend.python.util_rules.local_dists_pep660`. Building these wheels requires the same chroot that pants uses to build regular wheels and sdists. So, I refactored the rule in `util_rules.setup_py` so that I could reuse the part that builds the `DistBuildRequest`. These `local_dists_pep660` rules do approx this, starting with the rule called in export: - `Get(EditableLocalDists, EditableLocalDistsRequest(resolve=resolve))` uses rule `build_editable_local_dists` - injected arg: `ResolveSortedPythonDistributionTargets` comes from rule: `sort_all_python_distributions_by_resolve` - injected arg: `AllPythonDistributionTargets` comes from rule: `find_all_python_distributions` - `Get(LocalDistPEP660Wheels, PythonDistributionFieldSet.create(dist))` for each dist in the resolve uses rule: `isolate_local_dist_pep660_wheels` - create `DistBuildRequest` using the `create_dist_build_request` method I exposed in `util_rules.setup_py` - `Get(PEP660BuildResult, DistBuildRequest)` uses rule: `run_pep660_build` - generates the `.pth` file that goes in the editable wheel - runs a PEP 517 build backend wrapper script I wrote - uses the PEP 517 build backend configured for the `python_distribution` to generate the `.dist-info` directory - generates the `WHEEL` and `RECORD` file to build a conformant wheel file - includes the `.pth` file previously generated (and placed in the sandbox with the wrapper script) - uses `zipfile` to build the wheel (using a vendored+modified function from the `wheel` package). - prints a path to the generated wheel - collects the generated editable wheel into a digest and collects metadata about the digest similar to how the `local_dists` rules do. - merges the editable wheel digests for all of the `python_distribution` targets. This gets wrapped in `EditableLocalDists` Much of the rule logic was based on (copied then modified): `pants.backend.python.util_rules.dists` and `pants.backend.python.util_rules.local_dists`.
|
I'm adding an |
cognifloyd
added a commit
that referenced
this issue
Jun 20, 2024
…from a python_distribution (#21062) This adds a new `entry_point_dependencies` field to `python_tests` and `python_test` targets. This allows tests to depend on a subset (or all) of the `entry_points` defined on `python_distribution` targets (only on the `entry_points` field, not in the `provides` field). For example: ```python python_tests( name="tests", entry_point_dependencies={ "//address/to:python_distro_tgt_1": ["*"], # all entry points # only one group of entry points "//address/to:python_distro_tgt_2": ["console_scripts"], "//address/to:python_distro_tgt_4": ["st2common.runners.runner"], # or multiple groups of entry points "//address/to:python_distro_tgt_5": ["console_scripts", "st2common.runners.runner"], # or 1+ individual entry points "//address/to:python_distro_tgt_6": ["console_scripts/foo-bar"], "//address/to:python_distro_tgt_7": ["console_scripts/foo-bar", "console_scripts/baz"], "//address/to:python_distro_tgt_8": ["st2common.runners.runner/action-chain", "console_scripts/foo-bar"], }, ) ``` A dependency defined in `entry_point_dependencies` emulates an editable install of those `python_distribution` targets. Instead of including all of the `python_distribution`'s sources, only the specified entry points are made available. The entry_points metadata is also installed in the pytest sandbox so that tests (or the code under test) can load that metadata via `pkg_resources` or `importlib.metadata.entry_points` or similar. Misc notes: - I added this to the `pants.backend.experimental.python` backend and also registered the rules I extracted from `pants.backend.experimental.python.framework.stevedore` with the stevedore backend since it needs those rules. - Unlike all other dependencies fields, `entry_point_dependencies` is a `dict[str, list[str]]`. The `.keys()` of the dict are logically similar to standard dependencies fields. Using a dict provides more granular control over the dependencies. - Also unlike other dependencies fields, this field does NOT result in a dependency on the `python_distribution` target. Instead, that target provides the `entry_points` metadata to infer dependencies on the actual code for those entry points. - I extracted most of the logic for this from the experimental stevedore plugin. Enabling the stevedore plugin is not required to use this feature. Closes #11386 Closes #15481
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm trying to use Pants v2 for my project but it relies on some entry points that are defined in the setup.cfg of the package. If I create a wheel using
python_distribution, specify theentry_points, and install that wheel then everything works fine, but when I usepython_tests()to test the code it won't work because when running the tests Pants doesn't seem to expose entry points from the setup.cfg like tox does, or like doingpip install -e .Is there a way to get this working or is this something that isn't supported at the moment?
The text was updated successfully, but these errors were encountered: