[WIP] bazel: Add python data providers

[phlax]

Signed-off-by: Ryan Northey ryan@synca.io

Commit Message: bazel: Add python data providers
Additional Description:

This is a generalized solution to the problem of getting data from bazel into python tools

It adds a write_json rule that can create a json target from any data serializable by the bazel json parser

It also adds a py_data rule which allows json or yaml to be imported directly into python

The data rule take an optional filters argument which is a chain of mutating rules. The filters deal with python data and are agnostic to the source, but are obv specific to the data being processsed.

This PR adds the required macros and adds rules for ~all of the data currently used in tooling

It doesnt make use of the data - this will be added in a follow up PR

Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Deprecated:]
[Optional API Considerations:]

[phlax]

cc @mattklein123 @htuch

although this adds some code, it will allow us to reduce code overall, and exposes bazel and bazel data in ways that fits both with bazel's workflow and python's idioms, i think

ultimately, it will be very useful shifting any remaining scripts to bazel and tidying up some of the existing bazel rules

[phlax]

cc @envoyproxy/dependency-shepherds

this rule doesnt make sense in the api, which is mounted within an external namespace in envoy' workspace, and can only refer to the envoy targets through the @envoy namespace

perhaps the intention is to prevent accessing the envoy namespace from the api, but then that feels more like a job for visibility

[phlax]

this could be resolved alternately by shifting these rules into a separate namespace

without this change, the current implementation fails format checking - i guess for good reason

[phlax]

the one question im wondering about this is whether it would be better to have the mutators in the write_json and/or as implemented in the python accessors

the benefit to having the mutators in the write method would be

• non-python rules can make use of the mutated data
• if mutation is expensive then it would benefit from bazel caching etc

the downside is added code/complexity - its easier to implement as it is, as there is already a need to add a python template and inject it with data

[phlax]

the other thing is namespacing

with current implementation you can import "from" the repo - ie from source.extensions import extensions_build_config

with #17293 the envoy_repo namespace is introduced - which could allow something like from envoy_repo.source.extensions import ...

potentially, we could create a directory like tools/data (or just data) and mount that into the the envoy_repo namespace

rules would have to depend on @envoy_repo data in that case - which i guess is also more explicit from a bazel pov

[repokitteh-read-only]

CC @envoyproxy/dependency-shepherds: Your approval is needed for changes made to (bazel/.*repos.*\.bzl)|(bazel/dependency_imports\.bzl)|(api/bazel/.*\.bzl)|(.*/requirements\.txt)|(.*\.patch).

🐱

Caused by: #17998 was synchronize by phlax.

see: more, trace.

[phlax]

@lizan @htuch before pushing this PR i would like to go through all of the ways in which i want to use it and prepare the following PRs, mostly so i can think through the pattern

i would v much like this or something like this to land, and believe i can cut quite a bit of code with it, but i would like to think it through again, so its going to take me a few days

[phlax]

Tuple[str, ...]

[github-actions]

This pull request has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in 7 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

[phlax]

with the move to pytooling this should not be necessary, we can use json data from bazel