Detect included yaml files which are also targets

[jayvdb]

If a mobanfile has a target which is also part of the mobanfile yaml file system, moban should generate that file first if the file is missing, and re-run whenever the generated yaml file is modified.

e.g.

A.yaml includes:

overrides: B.yaml

targets:
  - B.yaml: genB.yaml.jj2

This will allow the yaml data to be built dynamically using templates, either to bootstrap a project with default values based on other input values, or for changes to values to be propagated into other yaml variables which can then be used.

Or, it allows yaml values to contain variables.

e.g. mobanfile:

overrides: derived_data.yaml

app_name: foobar
copyright_start: 2016

targets:
  - derived_data.yaml: gen_derived_data.jj2
  - copyright.js: const_copyright_variables.c.jj2
  - copyright.php: const_copyright_variables.php.jj2

gen_derived_data.jj2:

copyright_string: "(c) {{app_name}} {{copyright_start}}-{% get_current_year() %}"

(very simplified)

In a single case, that could be achieved by putting the logic for copyright_string into the template logic for each template which needs it. Or improved by using a re-usable macro included into each template.

However the use case I have is a large collections of data is needed as inputs, and throughout the string values are constants which will infrequently change.

Another fun experiment is making the mobanfile also a template. This allows a repo to be 100% controlled by an upstream repo.

e.g. mobanfile

requires:
  - https://github.com/foo/bar
targets:
  mobanfile: gen_moban_file.jj2
  other_file: other_file_gen.jj2

If the upstream gen_moban_file.jj2 adds a new file which needs to be deployed, the mobanfile would be updated. Currently the user then needs to run moban again for the new file to be generated. The user might forget to do that.

[jayvdb]

https://pypi.org/project/Templer/ has a slightly different approach, calling it dynamic context.

[jayvdb]

And of course Ansible has jinja variables in yaml transparently https://docs.ansible.com/ansible-container/container_yml/template.html & https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html

The 'problem' with that approach is that any transparent system would need to choose one template language for the yaml variables. We could choose one which is very basic, but IMO "jinja" would not be a good choice. It would need to be a cut down version of jinja, which offers no loop control -- only {{ ... }} but even that probably has too much power. Note that ansible does allow loops and other flow control, but that seems crazy for moban to do internally transparently. Templer docs suggest it only allows variable expansion.

https://github.com/kblomqvist/yasha is a bit like templer, and offers shell & subprocess as a filter, and https://github.com/kblomqvist/yasha#subprocess shows it has if logic in the yaml, so probably also loops and other flow control in the yaml.

https://bitbucket.org/djarvis/yamlp uses $a$ pre-processing.

https://docs.helm.sh/chart_template_guide/ has flow control using go template language.

https://docs.docker.com/compose/compose-file/#variable-substitution uses bash syntax

https://puppet.com/docs/puppet/5.0/hiera_interpolation.html uses python syntax.

Most of the time the "need" is only to reference other yaml values. e.g.

cookiecutter:
  postgresql_version: 10

image:
  name: django-mobans-pg-image
  version: 0.1
  maintainer: [email protected]
  parent: postgres:{{ cookiecutter.postgresql_version }}-alpine
...

https://stackoverflow.com/questions/4150782/use-yaml-with-variables is another great example.

It would be really nice if yaml provided something like this (string interpolation) natively.

I dont see anything in the yaml 1.3 RFCs which might provide something suitable.

I notice ruamel has already begun work on 1.3 https://pypi.org/project/yaml-1.3/

https://github.com/arthurlacoste/tampax uses {{ .. }}

https://github.com/metakirby5/whizkers & https://github.com/metakirby5/zenbu use {{ ... }}

https://github.com/grasmash/yaml-expander uses ${{ }}

Using https://pypi.org/project/HiYaPyCo/ for loading the yaml is another possible approach - it solves quite a few features needed for moban, but it also means we cant solve them ourselves.

The approach I like the best is to define custom yaml constructor, which will be feasible for most tools to build with their yaml parser, or achieve with post-processing.

https://stackoverflow.com/a/23212501 with answer expanded at https://stackoverflow.com/a/30679992 .
This effectively creates a new template language, and not a pretty one.

It is the approach used here, for !py and !sic :

https://pypi.org/project/pypyr/#substitutions

We could define !jinja and others as the constructors, allowing users to explicitly choose their dynamic yaml language.

[jayvdb]

https://github.com/dedalusj/ssm-config & https://github.com/elelsee/pycfn-yaml & https://github.com/EverythingMe/rainbow all use yaml tags

https://github.com/hftools/hftools/blob/master/hftools/file_formats/config.py has some more loaders.

Implicit env var interpolation https://github.com/opensurv/arupy/blob/master/arupy/config.py and https://github.com/SeedVault/rhizome/blob/master/bbot/config.py and https://github.com/snowplow/release-manager/blob/master/release_manager/utils.py and https://github.com/jirutka/yaml-env-tag and https://github.com/spockNinja/py-yaml-builder/blob/master/yaml_builder/loader.py

https://github.com/claylo/yaml-include done using yaml custom tags

Quite a few interesting and relevant custom tags in https://github.com/norwoodj/jconfigure

https://github.com/hchasestevens/bellybutton/blob/master/bellybutton/parsing.py is a very nice way to build a collection of yaml tags.

And finally a library for managing tags https://github.com/meiblorn/pyyaml-tags