Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CPDEV-101596] Rework of inventory compilation #650

Merged
merged 3 commits into from
May 14, 2024

Conversation

ilia1243
Copy link
Contributor

@ilia1243 ilia1243 commented Apr 26, 2024

Description

  • We have own implementation of compilation of jinja templates with both iterative and recursive compilation.
  • Due to iterative compilation, some default features of jinja do not work.
  • Recursive compilation is supported only for some custom filters, but not for the jinja's default ones.
  • To use primitive values (like plugins.install) inside templates we have to use filters like is_true, is_false.
  • Finally, there is no control over the compilation process
    • May complicate development (e.g. changing of inventory format)
    • Does not handle infinite recursion.

Solution

  • Instead of iterative and recursive compilation, only recursive compilation is left.
  • In recursive compilation, any access to the inventory sections inside templates, is proxied.
  • Implemented proxies for sequence, mapping, and slice.
  • The proxies support recursive compilation in a generic way (not only for custom filters), automatically convert primitive values, and support other customization of work with the particular inventory sections.

Breaking changes

  • Compilation result for templates like '{{ "{{ values.variable }}" | b64encode }}' has changed.
    It is necessary to use simpler '{{ values.variable | b64encode }}' with the same result.

Test Cases

TestCase 1

Fix a couple of escaping problems.

values:
  var1: "{{ '{{ .Name }}' }}"
  var2: A
  var3: '{% raw %}{{ raw1 }}{% endraw %}{{ values.var2 }}{% raw %}{{ raw2 }}{% endraw %}{{ values.var2 }}'

ER:

values:
  var1: '{{ .Name }}'
  var2: A
  var3: '{{ raw1 }}A{{ raw2 }}A'

AR:

  • values.var1 is not compiled
  • values.var3 is compiled to {{ raw1 }}AA{{ raw2 }}

TestCase 2

Templates with default filters.

values:
  var1: '{{ values.var2 | int }}'
  var2: '{{ 1 }}'

ER:

values:
  var1: '1'
  var2: '1'

AR:

values:
  var1: '0'
  var2: '1'

TestCase 3

Reference to primitive values.

plugins:
  my_plugin:
    var1: '{% if plugins.my_plugin.install %}unexpected{% else %}ok{% endif %}'
    install: '{{ "false" }}'

ER:

plugins:
  my_plugin:
    install: false
    var1: ok

AR:

plugins:
  my_plugin:
    install: false
    var1: unexpected

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • Integration CI passed
  • Unit tests. If Yes list of new/changed tests with brief description
  • There is no merge conflicts

Unit tests

test_jinja.py - different compilation cases
test_defaults.py - reference to primitive values.
test_cluster.py - compilation at light stage
test_coredns.py, test_templates.py - resolve some gaps in unit testing of compilation.

Sorry, something went wrong.

Verified

This commit was signed with the committer’s verified signature.
jalaziz Jameel Al-Aziz
Remove iterative compilation.

Extend recursive compilation to always automatically compile referenced variables.

Implement on-the-fly conversion of primitive values.

Implement points of extensions, and add custom handling of kubernetesVersion.
@ilia1243 ilia1243 added bug Something isn't working improvement New feature or request labels Apr 26, 2024
@ilia1243 ilia1243 requested a review from Imadzuma April 26, 2024 14:57
@koryaga koryaga requested a review from theboringstuff May 3, 2024 08:25
@vlar0816 vlar0816 changed the title Rework of inventory compilation [CPDEV-101596] Rework of inventory compilation May 13, 2024
@koryaga koryaga requested a review from OleksandrIaresko May 13, 2024 11:22
@koryaga koryaga added this to the v0.30.0 milestone May 14, 2024
@koryaga koryaga merged commit 7099715 into main May 14, 2024
43 checks passed
@koryaga koryaga deleted the improvement/inventory_compilation_rework branch May 14, 2024 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working improvement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants