Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop support for Python 2 unicode string literals in YAML renderer #48309

Merged
merged 27 commits into from
Jun 30, 2018

Conversation

terminalmage
Copy link
Contributor

@terminalmage terminalmage commented Jun 26, 2018

In earlier releases, the below was considered valid usage in Python 2, assuming that data was a dictionary containing keys/values which are unicode types:

/etc/foo.conf:
  file.managed:
    - source: salt://foo.conf.jinja
    - template: jinja
    - context:
        data: {{ data }}

Jinja will render the unicode string types in Python 2 with the "u" prefix (e.g. {u'foo': u'bar'}. While not valid YAML, earlier releases would successfully load these values.

This pull request drops this support, effective in the upcoming Fluorine release. Instead, it will now be necessary to use Jinja's "tojson" filter.

/etc/foo.conf:
  file.managed:
    - source: salt://foo.conf.jinja
    - template: jinja
    - context:
        data: {{ data|tojson }}

The "tojson" filter was added to Jinja in version 2.9, but several LTS releases (RHEL 7, Ubuntu 14.04, etc.) ship with earlier versions in their repositories. Therefore, this PR also adds an implementation of this filter, which will be used if Jinja doesn't already provide it (when it does, the upstream version of the filter will be used).

These changes allow Salt's custom YAML SafeLoader to use the libyaml versions of PyYAML's SafeLoader and SafeDumper (i.e. yaml.CSafeLoader and yaml.CSafeDumper) if PyYAML was built with libyaml support.

Thanks to @jacksontj for working with me on this.

@terminalmage
Copy link
Contributor Author

My new integration test doesn't take into account dict iteration order differences. Derp. Fixing this as well as a lint violation, and there appear to be some other related failures.

@terminalmage
Copy link
Contributor Author

OK, the necessary tests have been updated. I also have deprecated the json_encode_list and json_encode_dict filters, as they don't emit actual JSON, and are superseded by the tojson filter.

@rallytime rallytime requested review from a team June 26, 2018 21:41
salt/utils/jinja.py Show resolved Hide resolved
@terminalmage terminalmage requested a review from isbm June 27, 2018 15:29
Copy link
Contributor

@isbm isbm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@terminalmage thanks 😉

@terminalmage
Copy link
Contributor Author

I backported just the tojson filter to 2018.3 in #48339. I've updated the docs I wrote for this filter to reflect that.

This PR should be good to merge pending test results.

@rallytime
Copy link
Contributor

@terminalmage
Copy link
Contributor Author

@rallytime that should be fixed now. Apparently the jinja sort filter doesn't, you know, sort anything.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants