Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate entry_points.txt for python_tests that require entry points from a python_distribution #21062

Merged
merged 27 commits into from
Jun 20, 2024

Conversation

cognifloyd
Copy link
Member

@cognifloyd cognifloyd commented Jun 14, 2024

This adds a new entry_point_dependencies field to python_tests and python_test targets.

This allows tests to depend on a subset (or all) of the entry_points defined on python_distribution targets (only on the entry_points field, not in the provides field).

For example:

python_tests(
    name="tests",
    entry_point_dependencies={
        "//address/to:python_distro_tgt_1": ["*"],  # all entry points
        # only one group of entry points
        "//address/to:python_distro_tgt_2": ["console_scripts"],
        "//address/to:python_distro_tgt_4": ["st2common.runners.runner"],
        # or multiple groups of entry points
        "//address/to:python_distro_tgt_5": ["console_scripts", "st2common.runners.runner"],
        # or 1+ individual entry points
        "//address/to:python_distro_tgt_6": ["console_scripts/foo-bar"],
        "//address/to:python_distro_tgt_7": ["console_scripts/foo-bar", "console_scripts/baz"],
        "//address/to:python_distro_tgt_8": ["st2common.runners.runner/action-chain", "console_scripts/foo-bar"],
    },
)

A dependency defined in entry_point_dependencies emulates an editable install of those python_distribution targets. Instead of including all of the python_distribution's sources, only the specified entry points are made available. The entry_points metadata is also installed in the pytest sandbox so that tests (or the code under test) can load that metadata via pkg_resources or importlib.metadata.entry_points or similar.

Misc notes:

  • I added this to the pants.backend.experimental.python backend and also registered the rules I extracted from pants.backend.experimental.python.framework.stevedore with the stevedore backend since it needs those rules.
  • Unlike all other dependencies fields, entry_point_dependencies is a dict[str, list[str]]. The .keys() of the dict are logically similar to standard dependencies fields. Using a dict provides more granular control over the dependencies.
  • Also unlike other dependencies fields, this field does NOT result in a dependency on the python_distribution target. Instead, that target provides the entry_points metadata to infer dependencies on the actual code for those entry points.
  • I extracted most of the logic for this from the experimental stevedore plugin. Enabling the stevedore plugin is not required to use this feature.

Closes #11386
Closes #15481

The new python_tests entry_point_dependencies field will need almost
the same logic as the stevedore plugin, so extract the logic into
a reusable rule.

Also, extract a helper function from the shared logic in infer_python_distribution_dependencies.
to allow short circuiting as much as possible
mypy cannot tell that these Targets are PythonDistributions

And fix several actual typos or errors
A helper function has to be defined before the rule that uses it.
Otherwise, the pants engine cannot detect it.
If there's nothing to generate, stop.
If no entry points were found, skip the group.
If the generated file ends up being empty, skip it.
If no files were generated, use EMPTY_DIGEST.
So register them with the experimental backend for now.
Once util_rules.entry_points graduates to the official backend,
we can drop target_types_rules from the experimental backend.
Copy link
Member

@kaos kaos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. 👍🏽

) -> tuple[
Targets, PythonDistributionEntryPointGroupPredicate, PythonDistributionEntryPointPredicate
]:
assert entry_point_deps.value, "Unexpected empty entry_point_dependencies field"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this manifest to the user? (i.e. is this possible with BUILD file (mis)configuration? or is it merely a sign of an internal pants bug?)

What I'd prefer is to have the error message convey if this is in any way fixable when encountered, or if it's an internal bug and they need to turn to the pants project for assistance/report a bug.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This serves to narrow the types for mypy. Everywhere that could call this rule bails out before calling it if the field is empty (See infer_entry_point_dependencies and generate_entry_points_txt_from_entry_point_dependencies). So, I expect this to only ever happen during development, not a released version.

Maybe I could raise an exception like InvalidField. Would that be better?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I turned this into a more verbose ValueError. Thanks for the suggestion.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, right. In that case, I think an assert is just fine, as we don't expect to ever hit it, and it's presence is only to satisfy mypy. I've done the same, but leaving a hint as to why it's there (and thus, no assert message, as it's not supposed to ever hit):

assert parametrize_group is not None # Type narrowing to satisfy mypy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend: Python Python backend-related issues category:new feature
Projects
None yet
2 participants