Configure pantsbuild source roots

[cognifloyd]

Background

This is part 2 of introducing pants, as discussed in the TSC Meetings on 12 July 2022, 02 Aug 2022 and 06 Sept 2022. Pants has fine-grained per-file caching of results for lint, fmt (like black), test, etc. It also has lockfiles that work well for monorepos that have multiple python packages. With these lockfiles CI should not break when any of our dependencies or our transitive dependencies release new versions, because CI will continue to use the locked version until we explicitly relock with updates.

To keep PRs as manageable/reviewable as possible, introducing pants will take a series of PRs. I do not know yet how many PRs; I will break this up into logical steps with these goals:

• introduce pants to the st2 repo, and
• teach some (but not all) TSC members about pants step-by-step.

Previous pants PRs include:

• Initialize pantsbuild #5713

Overview of this PR

This PR configures pants source roots.
https://www.pantsbuild.org/docs/initial-configuration#configure-source-roots

This is based on a lot of work in a PoC pants branch where I experimented with a several approaches to define source roots. This was the most robust, and includes a comment to document why.

We can probably simplify this config by moving the packs in contrib/ into a separate directory like contrib/packs/. But that is out-of-scope until after we remove the Makefile and other CI infrastructure that expects those packs to be directly under contrib/.

pants.toml config file

Pants configuration lives in a file called pants.toml in the root of the repo.

https://www.pantsbuild.org/docs/initial-configuration#create-pantstoml

Many important things get configured in the pants.toml config file, but this PR only configures [source].root_patterns. I also tried using [source].marker_filenames but that didn't work very well in our repo with how we symlink the core pack into st2tests' fixture packs.

[source].root_patterns

Essentially, [source].root_patterns tells pants where to find python modules (eg to infer dependencies between them) and how to configure PYTHONPATH/sys.path.

Some project layouts use top-level folders for namespace purposes, but have the code live underneath. However, the code's imports will ignore these top-level folders, thanks to mechanisms like the $PYTHONPATH and the JVM classpath. Source roots are a generic equivalent of these concepts.
Run ./pants roots to see what Pants is using

Detailed docs on pants source roots are available here:

• https://www.pantsbuild.org/docs/source-roots
• https://www.pantsbuild.org/docs/reference-source

For StackStorm, we need a root for each python module (each core lib and each runner) and each pack. Plus there are some tool/util directories that we also needed to add to source roots.

Things you can do with pantsbuild after this PR

This PR does not wire up any lockfiles, formatters, linters, etc, yet. But you can test that pants resolves the source roots correctly with ./pants roots.

$ ./pants roots
.
contrib/chatops
contrib/core
contrib/default
contrib/examples
contrib/examples/lib
contrib/hello_st2
contrib/linux
contrib/packs
contrib/runners/action_chain_runner
contrib/runners/announcement_runner
contrib/runners/http_runner
contrib/runners/inquirer_runner
contrib/runners/local_runner
contrib/runners/noop_runner
contrib/runners/orquesta_runner
contrib/runners/python_runner
contrib/runners/remote_runner
contrib/runners/winrm_runner
pylint_plugins
scripts
st2actions
st2api
st2auth
st2client
st2common
st2common/benchmarks/micro
st2reactor
st2stream
st2tests
st2tests/testpacks/checks
st2tests/testpacks/errorcheck
tools