Pants: Add `pack_metadata` targets to BUILD files #5871

cognifloyd · 2023-01-23T20:20:51Z

Background

This is another part of introducing pants, as discussed in various TSC meetings.

Related PRs can be found in:

pantsbuild milestone (which includes PRs since Initialize pantsbuild #5713)
pantsbuild label (which also includes preparatory PRs that came before adding pantsbuild)

Overview of this PR

This PR adds pack_metadata targets, introduced in #5868 to our BUILD files.

If someone runs ./pants tailor ::, it will automatically add pack_metadata targets in any directory with a pack.yaml file, so when we add new fixtures, it should not be much work to keep the metadata up-to-date.

There are several fixture packs that don't have a pack.yaml file, so I added those targets manually. Also, for the fixtures, I made the python fixture depend on the pack metadata, so that pants will include all of a pack's metadata files if a test depends on that fixture.

I also added one pack_metadata_in_git_submodule target, which I described in #5868 as:

One of our packs is in a git submodule (st2tests/st2tests/fixtures/packs/test_content_version), so [this PR] also adds a pack_metadata_in_git_submodule target to cleanly handle a case where someone has not checked out the git submodules. This improves the Developer Experience by raising an error as early as possible to let people know that they need to checkout the git submodule and how to do that. Otherwise, they won't get an error until they run tests where we have some code to give the same instructions for the tests that need it.

Here is that target (sources have to be explicitly defined because they target is defined in a parent directory, not alongside the pack.yaml file):

st2/st2tests/st2tests/fixtures/packs/BUILD

Lines 6 to 14 in 26132d1

    
           pack_metadata_in_git_submodule( 
        
               name="test_content_version_metadata", 
        
               sources=[ 
        
                   "test_content_version/pack.yaml", 
        
                   "test_content_version/**/*.yaml", 
        
                   "test_content_version/icon.png", 
        
                   "test_content_version/requirements.txt", 
        
               ], 
        
           )

And here is what that error message looks like in practice:

jafloyd@MB-C02XKFC0JG5H st2 % ./pants tailor update-build-files ::     
14:08:25.96 [INFO] Initializing scheduler...
14:08:26.30 [INFO] Scheduler initialized.
14:08:26.82 [WARN] Unmatched glob from st2tests/st2tests/fixtures/packs:test_content_version_shell's `sources` field: "st2tests/st2tests/fixtures/packs/test_content_version/**/*.sh"

Do the file(s) exist? If so, check if the file(s) are in your `.gitignore` or the global `pants_ignore` option, which may result in Pants not being able to see the file(s) even though they exist on disk. Refer to https://www.pantsbuild.org/v2.14/docs/troubleshooting#pants-cannot-find-a-file-in-your-project.
14:08:27.03 [WARN] Unmatched glob from st2tests/st2tests/fixtures/packs:test_content_version's `sources` field: "st2tests/st2tests/fixtures/packs/test_content_version/**/*.py"

Do the file(s) exist? If so, check if the file(s) are in your `.gitignore` or the global `pants_ignore` option, which may result in Pants not being able to see the file(s) even though they exist on disk. Refer to https://www.pantsbuild.org/v2.14/docs/troubleshooting#pants-cannot-find-a-file-in-your-project.
14:08:27.04 [WARN] Unmatched globs from st2tests/st2tests/fixtures/packs:test_content_version_metadata's `sources` field: ["st2tests/st2tests/fixtures/packs/test_content_version/**/*.yaml", "st2tests/st2tests/fixtures/packs/test_content_version/icon.png", "st2tests/st2tests/fixtures/packs/test_content_version/pack.yaml", "st2tests/st2tests/fixtures/packs/test_content_version/requirements.txt"]

Do the file(s) exist? If so, check if the file(s) are in your `.gitignore` or the global `pants_ignore` option, which may result in Pants not being able to see the file(s) even though they exist on disk. Refer to https://www.pantsbuild.org/v2.14/docs/troubleshooting#pants-cannot-find-a-file-in-your-project.
14:08:27.04 [ERROR] 1 Exception encountered:

  UnmatchedGlobsError: One or more git submodules is not checked out. Make sure to run "git submodule update --init --recursive"in the repository root directory to check out all the submodules.

$ git submodule update --init --recursive
Submodule path 'st2tests/st2tests/fixtures/packs/test_content_version': checked out 'c9f4e7ca35a8c719ff4d017abd896fe146214f17'

And then once you checkout the git submodule, your command will succeed. For example, here is running tailor successfully (this also shows the list of pack.yaml files that tailor could detect automatically):

$ ./pants tailor update-build-files ::   
Created contrib/chatops/BUILD:
  - Add pack_metadata target metadata
Updated contrib/core/BUILD:
  - Add pack_metadata target metadata
Created contrib/debug/BUILD:
  - Add pack_metadata target metadata
Created contrib/default/BUILD:
  - Add pack_metadata target metadata
Updated contrib/examples/BUILD:
  - Add pack_metadata target metadata
Created contrib/hello_st2/BUILD:
  - Add pack_metadata target metadata
Updated contrib/linux/BUILD:
  - Add pack_metadata target metadata
Created contrib/packs/BUILD:
  - Add pack_metadata target metadata
Created contrib/packs/tests/fixtures/stackstorm-test/BUILD:
  - Add pack_metadata target metadata
Created contrib/packs/tests/fixtures/stackstorm-test2/BUILD:
  - Add pack_metadata target metadata
Created contrib/packs/tests/fixtures/stackstorm-test3/BUILD:
  - Add pack_metadata target metadata
Created contrib/packs/tests/fixtures/stackstorm-test4/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/action_chain_tests/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/core/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_1/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_10/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_11/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_12/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_13/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_14/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_15/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_16/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_17/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_18/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_19/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_2/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_20/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_21/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_22/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_3/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_4/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_5/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_6/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_7/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_8/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_9/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_schema_with_additional_items_1/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_schema_with_additional_properties_1/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_schema_with_nested_object_1/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_schema_with_nested_object_2/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_schema_with_nested_object_3/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_schema_with_nested_object_4/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_schema_with_nested_object_5/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_schema_with_pattern_and_additional_properties_1/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/dummy_pack_schema_with_pattern_properties_1/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/orquesta_tests/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/pack_dir_name_doesnt_match_ref/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/pack_invalid_requirements/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs/test_library_dependencies/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs_1/dummy_pack_4/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs_invalid/dummy_pack_17/BUILD:
  - Add pack_metadata target metadata
Updated st2tests/st2tests/fixtures/packs_invalid/dummy_pack_18/BUILD:
  - Add pack_metadata target metadata
14:08:44.28 [INFO] Filesystem changed during run: retrying `UpdateBuildFilesGoal` in 500ms...
14:08:45.30 [INFO] Filesystem changed during run: retrying `UpdateBuildFilesGoal` in 500ms...
14:08:52.90 [INFO] No required changes to BUILD files found. However, there may still be deprecations that `update-build-files` doesn't know how to fix. See https://www.pantsbuild.org/v2.14/docs/upgrade-tips for upgrade tips.

Several fixtures have been adjusted so pants can infer them. So, I dropped the hard-coded dep in the BUILD files.

This reverts commit 4a1b241.

amanda11

A request for clarification...

amanda11 · 2023-01-26T15:49:38Z

st2tests/st2tests/fixtures/aliases/BUILD

@@ -1 +1,7 @@
-python_sources()
+pack_metadata(
+    name="metadata",


Slightly confused. In the dummy_pack_23 we don't include the pack_metadata as there is no pack.yaml in that pack.

For things like this BUILD file and others at this level in the hierarchy, why do they have the pack_metatdata?

pack_metadata includes more than just pack.yaml.

st2/pants-plugins/pack_metadata/target_types.py

Lines 27 to 40 in a73225c

class PackMetadataSourcesField(ResourcesGeneratingSourcesField):

required = False

default = (

# metadata does not include any python, shell, or other sources.

"pack.yaml",

"config.schema.yaml",

"*.yaml.example",

"**/*.yaml",

"**/*.yml",

"icon.png", # used in st2web ui

# "requirements*.txt", # including this causes target conflicts

# "README.md",

# "HISTORY.md",

)

It also includes metadata yaml files for resources that go in packs: actions, aliases, sensors, etc.

Several of the directories at this level do not have pack.yaml, but they do have the other metadata yaml files. Examples:

st2tests/st2tests/fixtures/aliases/

actions/

action1.yaml

action2.yaml

action3.yaml

action4.yaml

aliases/

alias1.yaml

alias2.yaml

alias3.yaml

alias4.yaml

...

runners/ runners are not typically defined in packs, but close enough.

runner1.yaml

st2tests/st2tests/fixtures/backstop/

rules/

backstop.yaml

fail.yaml

success.yaml

triggers/ this is not common in packs, but is allowed.

trigger1.yaml

triggertypes/ this is even less common, and I'm not sure if it can be defined in packs, but close enough.

triggertype1.yaml

st2tests/st2tests/fixtures/descendants/ this is not like a pack, but has db entries in the form of yaml files which is sort of pack-ish

executions/ this would never be defined in a pack

child1_level1

...

Then the other fixtures do similar things. They all seem to have mongo docs in the form of yaml files (after-all the db effectively just holds all of the metadata files in json format, so going in the reverse makes sense).

We need resources for all of these yaml files. PackMetadata subclasses Resources, so everything in the pack_metadata target also counts as a resource. So, we either need to:

add resources(sources=...) targets that encompass all of the yaml files, and make the python fixture.py depend on them

add pack_metadata and make the python fixture.py depend on it

It was quick and easy to just treat these as packs using pack_metadata so that's what I did. Would it make more sense to you if we used resources() instead for these?

Oh. And one more clarification, dummy_pack_23 does not have ANY yaml files, not just pack.yaml. So, including the pack_metadata target there results in warnings because none of the sources globs match anything. All of these other fixtures do have yaml files, so the **/*.yaml glob matches.

It's fine - thanks for clarification.
My point for not reading the resource correctly / I did go and look and must have not scrolled past the pack example yaml line.

All good!

cognifloyd · 2023-01-26T18:12:43Z

st2tests/st2tests/fixtures/aliases/BUILD

+pack_metadata(
+    name="metadata",
+)


This would be functionally equivalent without using the pack_metadata target. Would you prefer this for these fixtures?

Suggested change

pack_metadata(

name="metadata",

)

resources(

name="metadata",

sources=["**/*.yaml"],

)

rush-skills

LGTM

amanda11

LGTM

cognifloyd added 11 commits January 23, 2023 12:59

pants: add pack_metadata targets

e5d4f18

add packs_metadata and related targets

f18e1a0

Add pack_metadata dependencies

8aebc18

Add pack_metadata for more fixtures

d995003

Allow pants to detect packs fixture usage

e1614fc

Cleanup fixtures BUILD files

e2bef63

Several fixtures have been adjusted so pants can infer them. So, I dropped the hard-coded dep in the BUILD files.

add BUILD metadata for st2tests/testpacks

0f2c7cc

depend on pack_metadata in st2tests.fixtures.packs.core.fixture

adf97fa

Add pack_metadata targets in st2reactor pack fixtures

521397b

Enable pants-plugins/pack_metadata

d864817

This reverts commit 4a1b241.

update changelog entry

26132d1

cognifloyd added enhancement infrastructure: ci/cd pantsbuild labels Jan 23, 2023

cognifloyd added this to the pants milestone Jan 23, 2023

cognifloyd self-assigned this Jan 23, 2023

pull-request-size bot added the size/L PR that changes 100-499 lines. Requires some effort to review. label Jan 23, 2023

cognifloyd requested review from amanda11, arm4b, nzlosh, rush-skills, winem and a team January 23, 2023 20:22

cognifloyd changed the title ~~Pants pack metadata~~ Pants: Add pack_metadata targets to BUILD files Jan 23, 2023

cognifloyd added 2 commits January 23, 2023 14:31

disable spurious pack_metadata target where there is no metadata

f4062f7

add pack_metadata to contrib/debug with ./pants tailor ::

e18e3ae

cognifloyd enabled auto-merge (squash) January 23, 2023 20:34

amanda11 reviewed Jan 26, 2023

View reviewed changes

oops. executable bit should not be set on BUILD files

54012ec

cognifloyd commented Jan 26, 2023

View reviewed changes

cognifloyd requested review from amanda11 and a team January 26, 2023 18:17

rush-skills approved these changes Jan 27, 2023

View reviewed changes

amanda11 approved these changes Jan 27, 2023

View reviewed changes

cognifloyd merged commit 6e773e6 into master Jan 27, 2023

cognifloyd deleted the pants-pack_metadata branch January 27, 2023 15:27

cognifloyd mentioned this pull request Jan 27, 2023

Tell pants about tests that use globs to test with many pack fixtures #5874

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pants: Add `pack_metadata` targets to BUILD files #5871

Pants: Add `pack_metadata` targets to BUILD files #5871

cognifloyd commented Jan 23, 2023 •

edited

Loading

amanda11 left a comment

amanda11 Jan 26, 2023

cognifloyd Jan 26, 2023

cognifloyd Jan 26, 2023 •

edited

Loading

amanda11 Jan 27, 2023

cognifloyd Jan 26, 2023

rush-skills left a comment

amanda11 left a comment

	pack_metadata_in_git_submodule(
	name="test_content_version_metadata",
	sources=[
	"test_content_version/pack.yaml",
	"test_content_version/*/.yaml",
	"test_content_version/icon.png",
	"test_content_version/requirements.txt",
	],
	)

	class PackMetadataSourcesField(ResourcesGeneratingSourcesField):
	required = False
	default = (
	# metadata does not include any python, shell, or other sources.
	"pack.yaml",
	"config.schema.yaml",
	"*.yaml.example",
	"*/.yaml",
	"*/.yml",
	"icon.png", # used in st2web ui
	# "requirements*.txt", # including this causes target conflicts
	# "README.md",
	# "HISTORY.md",
	)

Pants: Add pack_metadata targets to BUILD files #5871

Pants: Add pack_metadata targets to BUILD files #5871

Conversation

cognifloyd commented Jan 23, 2023 • edited Loading

Background

Overview of this PR

amanda11 left a comment

Choose a reason for hiding this comment

amanda11 Jan 26, 2023

Choose a reason for hiding this comment

cognifloyd Jan 26, 2023

Choose a reason for hiding this comment

cognifloyd Jan 26, 2023 • edited Loading

Choose a reason for hiding this comment

amanda11 Jan 27, 2023

Choose a reason for hiding this comment

cognifloyd Jan 26, 2023

Choose a reason for hiding this comment

rush-skills left a comment

Choose a reason for hiding this comment

amanda11 left a comment

Choose a reason for hiding this comment

Pants: Add `pack_metadata` targets to BUILD files #5871

Pants: Add `pack_metadata` targets to BUILD files #5871

cognifloyd commented Jan 23, 2023 •

edited

Loading

cognifloyd Jan 26, 2023 •

edited

Loading