feat(MDC): Validate configs in CI

[rahul-kumar-saini]

Overview

• Adds an optional step in CI that validates all .yaml files located under the config files directory
• See step "Dataset Config Validation" in some CI runs on this PR to see how it shows up

[codecov-commenter]

Codecov Report

Base: 92.81% // Head: 92.72% // Decreases project coverage by -0.09% ⚠️

Coverage data is based on head (6ad239c) compared to base (89fe3a7).
Patch coverage: 3.57% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3128      +/-   ##
==========================================
- Coverage   92.81%   92.72%   -0.10%     
==========================================
  Files         677      678       +1     
  Lines       30933    30961      +28     
==========================================
- Hits        28712    28709       -3     
- Misses       2221     2252      +31     

Impacted Files	Coverage Δ
snuba/validate_configs.py	0.00% <0.00%> (ø)
snuba/datasets/configuration/json_schema.py	100.00% <100.00%> (ø)
snuba/environment.py	83.33% <0.00%> (-16.67%)	⬇️
snuba/cli/consumer.py	68.62% <0.00%> (-3.93%)	⬇️
snuba/attribution/log.py	87.09% <0.00%> (+1.61%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[evanh]

Nice, this looks good! The only thing I would change is to move the validation script into the snuba app, so we can do snuba validate-configs.

[enochtangg]

The print statements can probably be converted to click.echo to follow convention.

Something I noticed when running snuba validate-configs, the snuba CLI-command by default imports all modules within /cli, which includes the subscriptions-executor command. This command runs get_enabled_dataset_names() in a command-line argument (

snuba/snuba/cli/subscriptions_executor.py

Line 27 in 2d5817d

type=click.Choice(get_enabled_dataset_names()),

) which runs load_configuration_data().

Therefore, when a user has an invalid config, and they run snuba validate-configs, the validation occurs in the loading of the subscriptions-executor module as opposed to snuba validate-configs itself.