CEP: Jinja functionality in v2 recipes #71

wolfv · 2024-04-12T15:50:41Z

No description provided.

h-vetinari · 2024-04-12T20:43:19Z

I do think that for the future (ie. rattler-build) we'll have an upcoming CEP to define this feature better.

Is this that CEP? In any case, I don't see {{ stdlib(...) }} in the diff, and I think it needs to be covered in some way, given that we're now rolling this out at scale.

wolfv · 2024-04-13T08:43:04Z

Yep, this is the one. Do you want to have it as a separate function or rolled into compiler? I prefer to roll it into compiler...

h-vetinari · 2024-04-13T23:30:23Z

Yep, this is the one. Do you want to have it as a separate function or rolled into compiler? I prefer to roll it into compiler...

I'd prefer a separate function, because it's less magic to have to understand (i.e. "why do I need to specify c_stdlib_version and how does that affect my build environment?" is much easier to answer if there is a {{ stdlib("c") }} under build:); also, the coupling of compiler and stdlib versions is entirely incidental (at least for C), so I'd like to not conflate them conceptually.

wolfv · 2024-04-14T05:59:57Z

My points:

There are already two variables for compilers (c_compiler and c_compiler_version). Adding additional 2 variables (that are non-mandatory) is fine IMO, especially if it's well-documented.
It complicates things for beginners. I think recipes would continue to work without adding the stdlib (correct?). So it's something they should add but don't have to add. For compiler it's clear - the recipe will not work without a compiler.
In most cases, it will always be added when there is a compiler.

I can imagine two behaviors:

We control the stdlib behavior from the variant_config. This means, if the variant config contains entries for c_stdlib / c_stdlib_version we use them from the compiler function.
We add additional arguments to the jinja function, so that compiler('c', stdlib='glibc').

Note that while you are correctly pointing out a dislike about magical behavior, we are printing a lot more information, clearly displayed, with rattler-build. So we can make it obvious where the stdlib comes from (we already point out that the clangxx_osx-arm64 package comes from the compiler, for example).

And lastly, I just think it's a matter of writing really good documentation.

h-vetinari · 2024-04-14T08:29:36Z

I think recipes would continue to work without adding the stdlib (correct?).

Define "work". When we increase the c_stdlib_version on linux and macos later this year, the artefacts produced by a recipe without {{ stdlib("c") }} would continue to work on modern systems, but would be broken on systems with a too-old C standard library, because it's through the meta-packages pulled in by stdlib that we set the correct run-exports in the artefact metadata.

It would only affect very old systems, but I'd still consider that broken.

So it's something they should add but don't have to add. For compiler it's clear - the recipe will not work without a compiler.

For me it's the opposite here - a compiled recipe needs (in 99.9% of cases) a C standard library just as much as it needs a compiler. Up until now this was just an ambient assumption built-in to our infrastructure, but I'd prefer people to learn this at the same time as the necessity of {{ compiler(...) }} - the two go hand in hand (as implied by having the same "c" argument, and the same way the keys are constructed), even though the relevant versions are completely independent from each other.

I can imagine two behaviors:

We add additional arguments to the jinja function, so that compiler('c', stdlib='glibc').

I really dislike this, because it would replace the nicely universal

    - {{ compiler('c') }}
    - {{ stdlib('c') }}

with a horrible mess of platform-specific selectors (or however the syntax for that ends up looking like in the new recipe format):

    - {{ compiler('c', stdlib='glibc') }}                     # [linux]
    - {{ compiler('c', stdlib='macosx_deployment_target') }}  # [osx]
    - {{ compiler('c', stdlib='vs') }}                        # [win]

We control the stdlib behavior from the variant_config. This means, if the variant config contains entries for c_stdlib / c_stdlib_version we use them from the compiler function.

I agree that this is technically feasible, and with good output and documentation would be a workable solution. When I say I prefer to keep {{ stdlib("c") }} explicit, this is not an all-or-nothing statement, but more a 70:30 kind of preference over the implicit handling through the compiler.

One reason that pushes this from a 60:40 (based on "less magic") to a 70:30 preference is that the C stdlib has a special role, in the sense that it can become necessary even without an explicit C compiler. In other words, we'd have to cover cases where a recipe with only {{ compiler("cxx") }} or {{ compiler("rust") }} also includes c_stdlib{,_version}.

cep-recipe-jinja.md

JeanChristopheMorinPerso · 2024-05-06T16:59:32Z

cep-recipe-jinja.md

+
+## The `cdt` function
+
+cdt stands for "core dependency tree" packages. These are repackaged from Centos.


Does "centos" have to be mentioned here? CDT packages could technically contain binaries from any Linux distribution right? I'm also wondering if we should add a link to some documentation that describe CDT more in details?

+1 to the above.

JeanChristopheMorinPerso · 2024-05-06T17:03:00Z

cep-recipe-jinja.md

+
+Where `cdt_name` and `cdt_arch` are loaded from the variant config. If they are undefined, they default to:
+
+- `cos6` for `cdt_name` on `x86_64` and `x86`, otherwise `cos7`


I feel a little bit uneasy with having default values hardcoded here. Do we really need to define default values in the spec? I feel like this kind of thing should be explicitly defined in an ecosystem (the central variant config in an ecosystem) instead of relying on some defaults.

I agree that we should NOT have default values for CDTs. The choice of CentOS 6 and 7 was made by Anaconda and conda-forge long ago, but I don't think we should necessarily impose such choices on the ecosystem as a whole.

Further, imposing these defaults means we either have to indefinitely support CDTs from EOL Linux distributions (CentOS 6 hit EOL in 2020-Nov, and CentOS 7 will hit EOL in 2024-Jun), or the evaluation of ${{ cdt("...") }} changes depending on the version of {conda,rattler)-build you use. Neither of those outcomes seems ideal.

Sure, we can get rid of this. I think we adopted this from conda-build where these values are also there by default.

FWIW, in the new CDTs for Alma 8, we're planning to get rid of cdt_name.

I'd also leave/move these out (see comment regarding defaults for compilers above).

@h-vetinari - curious what that means exactly? A change to conda-build?

I'll remove the defaults from the CEP!

JeanChristopheMorinPerso · 2024-05-06T17:10:27Z

cep-recipe-jinja.md

+- `cos6` for `cdt_name` on `x86_64` and `x86`, otherwise `cos7`
+- To the `platform::arch` for `cdt_arch`, except for `x86` where it defaults to `i686`.
+
+## The `pin_subpackage` function


pin_subpackage is defined in another section. This section seems to only define the arguments of pin_subpackage and pin_compatible?

JeanChristopheMorinPerso · 2024-05-06T17:11:44Z

cep-recipe-jinja.md

+  host:
+    - numpy
+  run:
+    - ${{ pin_compatible('numpy', exact=True) }}


Does it support min_pin and max_pin? It's unclear right now in this spec.

Yes, it does. Both use the same pin specification as mentioned earlier.

JeanChristopheMorinPerso · 2024-05-06T17:12:09Z

cep-recipe-jinja.md

+      version: "1.2.3"
+    requirements:
+      run:
+        - ${{ pin_subpackage('libfoo', exact=True) }}


Does it support min_pin and max_pin? It's unclear right now in this spec.

They both are derived from the same "pin" function. So yeah, they both support the same inputs.

JeanChristopheMorinPerso · 2024-05-06T17:12:39Z

cep-recipe-jinja.md

+
+## The `cmp` function
+
+The `cmp` function is used to compare two versions. It returns `True` if the comparison is true and `False` otherwise.


Should we add an example of how it can be used?

Yes, what operators are available and what is the syntax?

Added some examples!

Ah so it's more like a version matcher. When I read compare I thought we were comparing two packages together like "is numpy's version greater than python's", which didn't make much sense to me.

But this is more like asking "Does the version of Python match this version expression?". So in that sense the name could be better expressed as:

version(python, "<3.8")

version_match(python, "<3.8")

match_version(python, "<3.8")

satisfy(python, "<3.8")

Also this assumes that python will be represented by a concrete version x.y.z and not a spec like x.y.* right? IOW, it's already resolved to a specific package record.

If that's the case, should we specify that this function can only return after the solve has completed?

JeanChristopheMorinPerso · 2024-05-06T17:15:21Z

cep-recipe-jinja.md

+
+You can also check for the existence of an environment variable:
+
+- `${{ env.exists("MY_ENV_VAR") }}` will return `true` if the environment variable `MY_ENV_VAR` is set and `false` otherwise.


Nitpick. false and False are both used in the spec. It's hard to know if it returns a string or a bool.

I added the word boolean true to make it clear that this is a bool value.

cep-recipe-jinja.md

chenghlee · 2024-05-08T16:25:32Z

cep-recipe-jinja.md

+The variant config could look something like:
+
+```
+c_compiler:
+  - gcc
+c_compiler_version:
+  - "8.9"
+cxx_compiler:
+  - clang
+cxx_compiler_version:
+  - "12"
+```


This seems a bit out of place; I think the example variant config should be provided before using its values to evaluate the function.

Additionally, we should provide an example variant config showing how to select different compilers for different platforms; e.g., are we keeping the conda_build_config.yaml mechanism?

c_compiler: - gcc # [linux] - clang # [osx] - vc # [win] c_compiler_version: - "14.1" # [linux] - "17.0" # [osx] - "2022" # [win]

yeah we need to write a spec for the variant config (or even more global "config"). But IMO that shouldn't be part of this CEP.

chenghlee · 2024-05-08T16:28:12Z

cep-recipe-jinja.md

+
+## The compiler function
+
+The compiler function is used to stick together a compiler from {lang}_compiler and {lang}_compiler_version


Which, if any, of {lang}_compiler and {lang}_compiler_version must be specified in the variant configuration? If either or both are optional, the CEP should describe what happens in the case when the user/recipe does not provide a value (i.e., are there default values?).

chenghlee · 2024-05-08T16:28:55Z

cep-recipe-jinja.md

+
+## The `cdt` function
+
+cdt stands for "core dependency tree" packages. These are repackaged from Centos.


+1 to the above.

chenghlee · 2024-05-08T16:45:52Z

cep-recipe-jinja.md

+
+Where `cdt_name` and `cdt_arch` are loaded from the variant config. If they are undefined, they default to:
+
+- `cos6` for `cdt_name` on `x86_64` and `x86`, otherwise `cos7`


I agree that we should NOT have default values for CDTs. The choice of CentOS 6 and 7 was made by Anaconda and conda-forge long ago, but I don't think we should necessarily impose such choices on the ecosystem as a whole.

Further, imposing these defaults means we either have to indefinitely support CDTs from EOL Linux distributions (CentOS 6 hit EOL in 2020-Nov, and CentOS 7 will hit EOL in 2024-Jun), or the evaluation of ${{ cdt("...") }} changes depending on the version of {conda,rattler)-build you use. Neither of those outcomes seems ideal.

cep-recipe-jinja.md

chenghlee · 2024-05-08T17:01:00Z

cep-recipe-jinja.md

+- `min_pin`: The lower bound of the dependency spec. This is expressed as a `x.x....` version where the `x` are filled in from the corresponding version of the package. For example, `x.x` would be `1.2` for a package version `1.2.3`. The resulting pin spec would look like `>=1.2` for the `min_pin` argument of `x.x` and a version of `1.2.3` for the package.
+- `max_pin`: This defines the upper bound and follows the same `x.x` semantics but adds `+1` to the last segment. For example, `x.x` would be `1.(2 + 1)` for a package version `1.2.3`. The resulting pin spec would look like `<1.3` for the `max_pin` argument of `x.x` and a version of `1.2.3` for the package.


There seems a strong but unstated assumption here that version strings are integers separated by dots (.). If that is the case, that assumption should be explicitly stated and guidance provided as to what should happen if the assumption is violated. (Just sayin', because 1!24.4.0+91_gbfa8ccdce is a perfectly valid PEP-440/conda package version string.)

Also, as written, it seems like min_pin and max_pin do not support an explicit version; e.g., recipes cannot simply have max_pin="1.2.3".

Also, as written, it seems like min_pin and max_pin do not support an explicit version; e.g., recipes cannot simply have max_pin="1.2.3".

Conda's existing pin_compatible jinja has both {min,max}_pin as well as {lower,upper}_bound, in order to be able to distinguish 'x.x' from an explicit version limit.

Perhaps it's easier to understand to overload both styles into two kwargs (one for lower, one for upper), and allow either 'x(.x)*' or an explicit version number? It's more work to implement, but less confusing IMO (otherwise it's easy to specify conflicting things e.g. by setting both max_pin and upper_bound)

Wow, I wasn't aware! I found a few places where it's used but I don't know how it works. For example, in the onnx-feedstock it says:

- {{ pin_compatible('numpy', lower_bound='1.19', upper_bound='3.0') }} # [py>38]

Does this still take the lower bound from the build-time resolution? Or is this equivalent to just writing numpy >=1.19,<3.0?

(https://github.com/conda-forge/onnx-feedstock/blob/49b7f95ea64d2b0e174f23ac62fe0f59441491d7/recipe/meta.yaml#L49)

I also don't really understand why we would want this in the function. One can just as well write something like:

- numpy >=1.5 - ${{ pin_compatible("numpy", max_pin="x.x", min_pin="x.x.x") }}

If one needs additional (hard-coded) constraints. The solver doesn't really care about more constraints.

@h-vetinari I'm on board with your proposal.

I suppose that could in theory create a problem if some upstream project decided to the string literals x.x, x.y, and x.z as their versions, but if that ever happens, I'm happy to say "NOPE" to attempting to package it. 😆

@wolfv, given your changes in 0fda213, I presume you're not on board with this idea (or didn't see this thread)?

If you want to keep {min,max}_pin separate from {lower,upper}_bound, I think we need to enforce that at most one {min_pin,lower_bound} resp. {max_pin,upper_bound} each may be specified, everything else should raise an error.

I'd still prefer a unified interface, because I don't see the gain to keep these arguments separate (but rather requires a lot of complexity to ensure it is unambiguous/consistent)

I indeed didn't see it or didn't read carefully enough. I also think this is a good change. The logic is already in place, so I don't think it will be a big change on the rattler-build side.

OK, I just implemented that we only use lower_bound and upper_bound, but now I am wondering if we shouldn't have used min_pin and max_pin instead (as more people are using that these days). But this PR has the implementation: prefix-dev/rattler-build#918

I proposed the API that I think is more natural and IMO more desirable long-term.

Obviously the naming choice is just bike shedding and both would work, but max_pin=1.2.3 seems off to me, whereas upper_bound='x.x' is fine because it's a more generic term, where users don't have to guess what a pin is or isn't.

chenghlee · 2024-05-08T17:03:49Z

cep-recipe-jinja.md

+
+## The `hash` variable
+
+`${{ hash }}` is the variant hash and is useful in the build string computation. This used to be `PKG_HASH` in the old recipe format. Since the `hash` variable depends on the variant computation, it is only available in the `build.string` field and is computed after the entire variant computation is finished.


Should we explicitly say that this is a read-only variable? E.g., is providing "hash: deadbeef123455" in the variant config file explicitly prohibited?

Shall we call it variant_hash?

it's not really prohibited in the implementation right now. But we can say that it should be...

I don't have strong feelings about the name. We can change it to variant_hash if that's preferred!

chenghlee · 2024-05-08T17:04:12Z

cep-recipe-jinja.md

+
+## The `version_to_buildstring` function
+
+- `${{ python | version_to_buildstring }}` converts a version from the variant to a build string (it removes the `.` character and takes only the first two elements of the version).


Should we provide an example here?

This is the first time we introduce the | syntax as well. Can we do version_to_buildstring(python)?

Yeah using a function would also work. But we also expose the default minijinja filters (such as lower, replace, ...), that can be used like ${{ "mystring" | upper }}

or ${{ val | replace('foo', 'moo') }}

@jaimergp I added some more text regarding jinja filters to the bottom of the CEP.

jaimergp · 2024-05-09T11:08:09Z

cep-recipe-jinja.md

@@ -0,0 +1,125 @@
+# Jinja functions in recipes
+
+The new recipe format has some Jinja functionalities. We want to specify what functions exist and their expected behavior.


I feel this is lacking a short intro. At least we should link into the previous CEPs.

Suggested change

The new recipe format has some Jinja functionalities. We want to specify what functions exist and their expected behavior.

The new recipe format (introduced by CEP-XX, CEP-XY) has some Jinja functionalities. We want to specify what functions exist and their expected behavior.

cep-recipe-jinja.md

jaimergp · 2024-05-09T11:18:00Z

cep-recipe-jinja.md

+- `${{ env.get("MY_ENV_VAR") }}` will return the value of the environment variable `MY_ENV_VAR` or throw an error if it is not set.
+- `${{ env.get_default("MY_ENV_VAR", "default_value") }}` will return the value of the environment variable `MY_ENV_VAR` or `"default_value"` if it is not set.


Could this be overloaded to a single function? With the 2nd param being optional.

yeah, that can work. We can also make it a named argument, e.g.

env.get("FOO", default="baz")

What do you think?

@jaimergp I implemented this change: prefix-dev/rattler-build#917

Let me know what you think.

cep-recipe-jinja.md

Co-authored-by: jaimergp <[email protected]>

Co-authored-by: Jean-Christophe Morin <[email protected]>

Co-authored-by: jaimergp <[email protected]>

cep-recipe-jinja.md

wolfv · 2024-05-20T17:08:04Z

@h-vetinari since we don't have a CEP for stdlib it might be good to define it here.

What are the default values for the different operating systems?

h-vetinari · 2024-05-20T22:15:55Z

since we don't have a CEP for stdlib it might be good to define it here.

Sounds good!

What are the default values for the different operating systems?

I was under the impression that defaults here aren't necessarily a good thing? Perhaps even more than the compiler packages, the stdlib-metapackages don't have a "canonical" naming. Are you planning to set defaults for {{ compiler(...) }} as well? Looking at the diff here it doesn't seem like there are default being proposed?

In any case, here are the specs that we're using in conda-forge. They won't change substantially anymore, barring unforeseen events.

c_stdlib:
  - sysroot                    # [linux]
  - macosx_deployment_target   # [osx]
  - vs                         # [win]
m2w64_c_stdlib:                # [win]
  - m2w64-toolchain            # [win]              <-- might change to m2w64-sysroot in the future
c_stdlib_version:              # [unix]
  - 2.17                       # [linux]            <-- per July 2024
  - 10.13                      # [osx and x86_64]
  - 11.0                       # [osx and arm64]

wolfv · 2024-05-21T16:17:36Z

I expanded the content a bit.

Some things that are left to decide:

Limit the filters from Jinja to a default subset?
Should we add a split(str) filter (useful for version numbers, so that one can do "${{ version | split(".") | slice(0, 2) | join(".") }} or something along those lines. We could also create a more specialized function/filter for version string manipulation of this sort.
Should env work more like Python env
- env["FOO"] instead of env.get("FOO")
- env.get("FOO", "default") instead of env.get_default("FOO", "default")
Or should it be more like Github Actions
- env.FOO instead of env.get("FOO")
- env.FOO or "default" for default values
Should ${{ compiler('c') }} evaluate directly to gcc_linux-64 or should there be an intermediate representation (as is now)
For pin_subpackage it will be impossible to evaluate it directly (due to self-referential nature of the function).
Should we add lower_bound and upper_bound to the pin functions? And how should they work (in conjunction with min_pin and max_pin)?
The final name for the cmp function.

mbargull · 2024-06-19T12:37:09Z

cep-recipe-jinja.md

+Context evaluation must happen from top-to-bottom. That means a later value can
+reference an earlier one like so:


This is in conflict with YAML's representation graph semantics and CEP-14.
See
https://yaml.org/spec/1.2.2/#3221-mapping-key-order
and

ceps/cep-14.md

Lines 73 to 78 in 21aff84

The variables can reference other variables from the context section.

> [!NOTE]

> The order of the keys is not enforced by the YAML standard. We expect parsers to parse maps (especially the context section)

> in the order they are defined in the file. However, we do not require this behavior for the recipe to be valid to conform to the YAML standard.

> Given this, implementations need to ensure topological sorting is done before string interpolation.

. If the semantics from CEP-14 shall be repeated here, you'd have to paste in the blob from there, i.e.:

Suggested change

Context evaluation must happen from top-to-bottom. That means a later value can

reference an earlier one like so:

The variables can reference other variables from the context section.

> [!NOTE]

> The order of the keys is not enforced by the YAML standard. We expect parsers to parse maps (especially the context section)

> in the order they are defined in the file. However, we do not require this behavior for the recipe to be valid to conform to the YAML standard.

> Given this, implementations need to ensure topological sorting is done before string interpolation.

It would be good to instead or at least additionally reference CEP-14 here.

I added this text. Rattler build will need the user to use the correct order for the recipe to work but you'll get a nice error :)

That's totally fine; meaning, it's then just a deficiency in rattler-build for now that we can resolve later.
The spec-like document only can't say "we need sequential processing of non-sequence data type".

We could make another CEP also allowing for these ordered context fields to be specified as

context: - key: valueExpr

This is similar to how env vars are specified in kubernetes for example

We could make another CEP also allowing for these ordered context fields to be specified as

FYI, this had also been suggested in #56 (comment) .

mbargull · 2024-06-19T12:45:23Z

cep-recipe-jinja.md

+Any variable specified in the variant configuration file can be used in a recipe
+by using it in a Jinja expression. For example, with a variant config like:


It would make sense to note that variant variable values are always strings -- at least in current conda_build_config.yaml semantics it is that way, IIRC.
E.g., a some_integer: [123, 124] entry from the variant config would have to be used like ${{ "conditional_dep" if int(some_integer) > 123 }} in the Jinja contexts.
(I can't recall if that semantic changed for the new recipe format -- if it did, we should still put in some example to indicate such difference.)

Actually we decided to allow string, integer and boolean values.

Oh... we may want to discuss that (and the format of the variant config file in general).
That's a change that has its pros and cons and we should be careful on how to handle it.

Yeah I was planning to leave that for another document :)

mbargull · 2024-06-19T12:51:55Z

cep-recipe-jinja.md

+> [!NOTE] Default values for the `compiler` function
+>
+> If not further specified, the following values are used as default values:


I'd put such implementation/ecosystem-specific defaults either into a separate document or at least some separate/supplemental section (since they can or should be easily changed and not defined in a conceptual/"specification precursor" document, IMO).

That's probably resolved by the commit from some minutes ago (haven't yet checked those changes).

cep-recipe-jinja.md

mbargull · 2024-06-19T12:55:15Z

cep-recipe-jinja.md

+
+Where `cdt_name` and `cdt_arch` are loaded from the variant config. If they are undefined, they default to:
+
+- `cos6` for `cdt_name` on `x86_64` and `x86`, otherwise `cos7`


I'd also leave/move these out (see comment regarding defaults for compilers above).

cep-recipe-jinja.md

mbargull · 2024-06-19T13:05:05Z

cep-recipe-jinja.md

+
+You can also check for the existence of an environment variable:
+
+- `${{ env.exists("MY_ENV_VAR") }}` will return a boolean `true` if the


Is the .exists a standard Jinja method?
(Sorry, I didn't look this up. Just want to make sure we are conscious this in case it's some custom thing vs a plain Python "MY_ENV_VAR" in env.)

No, it's not standard jinja. You're right, we might be able to implemnetn the functions to make "FOO" in env work, but I am not sure if its worth it (also seems a bit more magical?).

We also don't use a regular "dict" or "sequence" object to be able to throw proper errors etc.

Ah, I also missed that env.get("MY_ENV_VAR") should throw an error in case MY_ENV_VAR isn't set -- meaning I totally missed that env isn't meant to be a dict.
TBH, I don't have a strong opinion then (I actually like a method better; no idea what people would prefer syntax- and/or naming-wise).

get just returns None for undefined keys in Python and Jinja

https://docs.python.org/3/library/stdtypes.html#dict.get

mbargull · 2024-06-19T13:10:33Z

cep-recipe-jinja.md

+- `${{ python | version_to_buildstring }}` converts a version from the variant
+  to a build string (it removes the `.` character and takes only the first two
+  elements of the version).


I'm not so sure about the utility of this one.
Yes, this how we currently compose parts of build strings.
But it's also a hacky thing to do, really.
This might just support dragging on such build string massaging for longer than we should.

(That is not to say that I'm completely opposed to such functionality; I just want to caution against adding potential future burden from the get go.)

It is currently relatively cumbersome to recreate this functionality in jinja, but I think it might be doable iwth split, join and slice. So we could also remove it.

Yes, it'd be a python.split(".")[:2] | join("") -- so, not too complicated, but also error-prone.
My remark was more about why we do these stuff in general, though.
I think turning a 11.2.0 into 112 is just not wise but just a historic habit that we might want to weed out...

Actually, this syntax doesn't work in the new recipes (at least not in the way implemneted in rattler-build since we're using minijinja that doesn't have all the Python builtins implemented).

using '' as the join seperator is almost certainly a bad idea even though we do it all the time. _ is a FAR better solution since it avoids the 1.12.0 == 11.2.0 problem that this has

jezdez · 2024-07-02T13:18:44Z

@conda/steering-council

This vote falls under the "Enhancement Proposal Approval" policy of the conda governance policy,
please vote and/or comment on this proposal at your earliest convenience.

It needs 60% of the Steering Council to vote yes to pass.

To vote, please leave yes, no or abstain as comments below.

If you have questions concerning the proposal, you may also leave a comment or code review.

This vote will end on 2024-07-16, End of Day, Anywhere on Earth (AoE). This is an extended voting period due to summer holiday time in the Northern Hemisphere.

baszalmstra · 2024-07-02T13:23:08Z

yes

wolfv · 2024-07-02T13:43:11Z

mariusvniekerk · 2024-07-02T14:40:50Z

cep-recipe-jinja.md

+```yaml
+requirements:
+  host:
+    - ${{ "cudatoolkit" if cuda == "have_cuda" }}


Should we add an explicit callout for the case like this where expression can evaluate into "" or i guess null and that means this is excluded.

We have something about that in line 500 - 510: https://github.com/conda/ceps/pull/71/files/c1028142a7f6e3e8a5d011667e0bec3b3f67ab3a#diff-0154c4169103373897853cfe5287bb7a1101289dd78c7fa30d2acba92b644938R500-R510

JeanChristopheMorinPerso · 2024-07-02T17:51:47Z

cep-recipe-jinja.md

+
+## Error handling
+
+Build tools should be aggressive about Jinja errors:


Is "should" defined somewhere in the CEP repo? If not, an implementer could interpret it as optional compared to "must".

Ideally we determine these in a provisional CEP that others use

cep-recipe-jinja.md

wolfv · 2024-07-16T10:24:27Z

@marcelotrevisani @jakirkham @mbargull @CJ-Wright last chance to vote - you can also choose abstain which would still help with the quorum.

schuylermartin45 · 2024-07-16T12:29:57Z

NP on the title of this PR: Since this is schema_version: 1 I've been referring to this as the V1 recipe format and the pre-CEP-13 format as V0. Do we want to keep consistent with that nomenclature?

I'm also not sure if it is worth codifying somewhere to prevent confusion. Maybe it is also worth it to add a section to the top of the CEP that specifies which schema_version a CEP belongs to, to prevent future confusion?

chenghlee · 2024-07-16T16:24:45Z

For the record: voted "no" because while I like the ideas presented in this CEP, I don't think the CEP as written is ready to be adopted as a specification. IMO, various unanswered questions/comments need to be addressed before adoption (e.g., top-down evaluation of the context object).

mbargull · 2024-07-17T04:53:22Z

cep-recipe-jinja.md

+- `target_platform` is a string that represents the platform for which the package is built. It is a string of the form `os-arch` (e.g. `linux-64`, `osx-64`, `win-64`, `linux-aarch64`, ...).
+- `build_platform` is a string that represents the platform on which the package is built. Same format as `target_platform`.
+- `linux`, `osx`, `win`, `emscripten`: These are boolean variables that are `true` if the target platform is a Linux, macOS, Unix, or Windows platform, respectively. Note that this is the first part of the `target_platform` string.
+- `x86_64`, `aarch64`, `armv7l`, `ppc64le`, `s390x`, `sparc64`, `riscv64`: These are boolean variables that are `true` if the target platform is the respective architecture. Note that, except for `x86_64`, these are the second part of the `target_platform` string.


This does not capture the osx-arm64/win-arm64 cases.
Not a huge deal, but it would be good to mention here how those are handled.

added arm64.

mbargull · 2024-07-17T05:13:25Z

cep-recipe-jinja.md

+
+Context evaluation must happen from top-to-bottom. That means a later value can
+reference an earlier one like so:
+
+```yaml
+context:
+  version: "1.0.5"
+  name_and_version: "pkg_${{ version | replace('.', '_') }}"  # evaluates to "pkg_1_0_5"
+```


Suggested change

Context evaluation must happen from top-to-bottom. That means a later value can

reference an earlier one like so:

```yaml

context:

version: "1.0.5"

name_and_version: "pkg_${{ version | replace('.', '_') }}" # evaluates to "pkg_1_0_5"

```

It would be better to leave this under-specified here instead of specifying it in conflict with the CEP-14.
This exact issue with the specification has been brought up by me in both previous CEPs for the new format and the wording here regresses on the issue instead of finally addressing it.
I do not understand why this simple issue is not being addressed, given that solutions have already been proposed.

jakirkham

Thanks Wolf! 🙏

Generally seems reasonable. Included some comments/suggestions. Otherwise would be good to add

cep-recipe-jinja.md

jakirkham · 2024-07-17T10:21:46Z

cep-recipe-jinja.md

+- `lower_bound`, defaults to `x.x.x.x.x.x`: the lower bound, either as a version
+  or as a "pin expression", or `None`
+- `upper_bound`, defaults to `x`: the upper bound, either as a version or as a
+  "pin expression" or `None`


This is specifying bounds...

jakirkham · 2024-07-17T10:22:53Z

cep-recipe-jinja.md

+- The epoch is left untouched by the `max_pin` (or `min_pin`). If the epoch is
+  set, it will be included in the final version. E.g. `1!1.2.3` with a
+  `max_pin='x.x'` will result in `<1!1.3.0a0`.
+- When bumping the version with a `max_pin` the local version part is removed.
+  For example, `1.2.3+local` with a `max_pin='x.x'` will result in `<1.3.0a0`.


Whereas this is specifying pins. Should we consolidate to one terminology or should we define both? Are there cases we need both?

jakirkham · 2024-07-17T10:26:06Z

cep-recipe-jinja.md

+> `conda-build` uses the `lower_bound` for the version that is used in
+> the `max_pin` pinning expression. `conda-build` also ignores the `min_pin`
+> expression when a `upper_bound` is used.


Am curious about the motivation for the differences here

FWIW do find users confuse or forget about the options (usually the bounds). So consolidating somehow would makes sense

jakirkham · 2024-07-17T10:39:01Z

cep-recipe-jinja.md

+
+You can also check for the existence of an environment variable:
+
+- `${{ env.exists("MY_ENV_VAR") }}` will return a boolean `true` if the


get just returns None for undefined keys in Python and Jinja

https://docs.python.org/3/library/stdtypes.html#dict.get

jakirkham · 2024-07-17T10:41:25Z

cep-recipe-jinja.md

+
+## Error handling
+
+Build tools should be aggressive about Jinja errors:


Ideally we determine these in a provisional CEP that others use

Co-authored-by: jakirkham <[email protected]>

wolfv · 2024-07-22T15:26:48Z

The vote has been closed with the following result:

Total voters: 15 (valid: 13 = 86.67%)

Yes votes (11 / 84.62%):

@xhochy (Uwe Korn)
@mariusvniekerk (Marius van Niekerk)
@goanpeca (Gonzalo Peña-Castellanos)
@ocefpaf (Filipe Fernandes)
@msarahan (Michael Sarahan)
@jakirkham (John Kirkham)
@jezdez (Jannis Leidel)
@wolfv (Wolf Vollprecht)
@jaimergp (Jaime Rodríguez-Guerra)
@kkraus14 (Keith Kraus)
@baszalmstra (Bas Zalmstra)

No votes (1 / 7.69%)):

@chenghlee (Cheng H. Lee)

Abstain votes (1 / 7.69%):

@mbargull (Marcel Bargull)

Not voted (2):

@CJ-Wright (Christopher J. 'CJ' Wright)
@marcelotrevisani (Marcelo Duarte Trevisani)

Invalid votes (0):

We reached quorum, and enough YES votes for this CEP to be accepted. 🎉

beeankha · 2024-10-03T21:11:24Z

cep-17.md

+The "old" recipe format has allowed arbitrary Jinja syntax (including set,
+if/else or for loops). The new recipe format only allows a subset of Jinja with


Suggested change

The "old" recipe format has allowed arbitrary Jinja syntax (including set,

if/else or for loops). The new recipe format only allows a subset of Jinja with

The "old" (`V0`) recipe format has allowed arbitrary Jinja syntax (including `set`,

`if`/`else` or `for` loops). The "new" (`V1`) recipe format only allows a subset of Jinja with

We should probably also change old to V0 while we're at it.

The title of this PR should be changed to reference V1 recipe format vs V2

beeankha · 2024-10-03T21:44:13Z

cep-17.md

+## The `hash` variable
+
+`${{ hash }}` is the variant hash and is useful in the build string computation.
+This used to be `PKG_HASH` in the old recipe format. Since the `hash` variable


Suggested change

This used to be `PKG_HASH` in the old recipe format. Since the `hash` variable

This used to be `PKG_HASH` in the `V0` recipe format. Since the `hash` variable

beeankha · 2024-10-03T21:44:33Z

cep-17.md

+```
+
+In this case the value from the `python` _variant_ is used to add or remove
+optional dependencies. Note that generalizes and replaces selectors from old


Suggested change

optional dependencies. Note that generalizes and replaces selectors from old

optional dependencies. Note that generalizes and replaces selectors from `V0`

beeankha · 2024-10-03T21:53:53Z

cep-17.md

+# compiler: "superfoo_linux-64 1.2.3"
+```
+
+> [!NOTE] Default values for `<lang>_compiler`


Suggested change

> [!NOTE] Default values for `<lang>_compiler`

> [!NOTE]

>

> Default values for `<lang>_compiler`

add cep for all jinja functionality

f1ad3f2

JeanChristopheMorinPerso reviewed May 6, 2024

View reviewed changes

chenghlee requested changes May 8, 2024

View reviewed changes

jaimergp reviewed May 9, 2024

View reviewed changes

cep-recipe-jinja.md Outdated Show resolved Hide resolved

jaimergp reviewed May 9, 2024

View reviewed changes

cep-recipe-jinja.md Outdated Show resolved Hide resolved

jaimergp reviewed May 9, 2024

View reviewed changes

cep-recipe-jinja.md Outdated Show resolved Hide resolved

jaimergp reviewed May 9, 2024

View reviewed changes

cep-recipe-jinja.md Outdated Show resolved Hide resolved

jaimergp reviewed May 9, 2024

View reviewed changes

cep-recipe-jinja.md Outdated Show resolved Hide resolved

jaimergp reviewed May 9, 2024

View reviewed changes

cep-recipe-jinja.md Outdated Show resolved Hide resolved

jaimergp reviewed May 9, 2024

View reviewed changes

cep-recipe-jinja.md Outdated Show resolved Hide resolved

jaimergp reviewed May 9, 2024

View reviewed changes

cep-recipe-jinja.md Outdated Show resolved Hide resolved

jaimergp reviewed May 9, 2024

View reviewed changes

cep-recipe-jinja.md Outdated Show resolved Hide resolved

wolfv and others added 5 commits May 10, 2024 09:31

Update cep-recipe-jinja.md

07da5e7

Co-authored-by: jaimergp <[email protected]>

Update cep-recipe-jinja.md

e4e0a42

Co-authored-by: Jean-Christophe Morin <[email protected]>

Update cep-recipe-jinja.md

81bdf68

Co-authored-by: jaimergp <[email protected]>

Update cep-recipe-jinja.md

2694476

Co-authored-by: jaimergp <[email protected]>

Update cep-recipe-jinja.md

62ea3d4

Co-authored-by: jaimergp <[email protected]>

h-vetinari suggested changes May 10, 2024

View reviewed changes

cep-recipe-jinja.md Outdated Show resolved Hide resolved

add more text regarding jinja filters

0d3f301

add more content and context

dde40b1

wolfv added 2 commits June 19, 2024 14:53

update default values for compiler

fa7dbdb

format headers more appropriately

c3c03c7

mbargull requested changes Jun 19, 2024

View reviewed changes

wolfv added 2 commits June 19, 2024 15:42

fix syntax

fba70b6

add is_... functions and default variables

c102814

jezdez added the vote Voting following governance policy label Jul 2, 2024

mariusvniekerk reviewed Jul 2, 2024

View reviewed changes

JeanChristopheMorinPerso reviewed Jul 2, 2024

View reviewed changes

mariusvniekerk approved these changes Jul 3, 2024

View reviewed changes

jaimergp reviewed Jul 6, 2024

View reviewed changes

cep-recipe-jinja.md Outdated Show resolved Hide resolved

schuylermartin45 mentioned this pull request Jul 16, 2024

Upgrades commonly used selectors conda-incubator/conda-recipe-manager#80

Merged

mbargull reviewed Jul 17, 2024

View reviewed changes

h-vetinari mentioned this pull request Jul 17, 2024

CEP: Recipe serialization in packages #74

Open

jakirkham approved these changes Jul 17, 2024

View reviewed changes

Update cep-recipe-jinja.md

81d9515

Co-authored-by: jakirkham <[email protected]>

baszalmstra mentioned this pull request Aug 1, 2024

fix: omit also jinja functions prefix-dev/rattler-build-conda-compat#37

Merged

wolfv added 2 commits August 26, 2024 11:24

fix and finalize CEP

822558a

rename to cep-17

4141b17

beeankha reviewed Oct 3, 2024

View reviewed changes


		## The `cdt` function

		cdt stands for "core dependency tree" packages. These are repackaged from Centos.


		Where `cdt_name` and `cdt_arch` are loaded from the variant config. If they are undefined, they default to:

		- `cos6` for `cdt_name` on `x86_64` and `x86`, otherwise `cos7`


		## The `cmp` function

		The `cmp` function is used to compare two versions. It returns `True` if the comparison is true and `False` otherwise.


		You can also check for the existence of an environment variable:

		- `${{ env.exists("MY_ENV_VAR") }}` will return `true` if the environment variable `MY_ENV_VAR` is set and `false` otherwise.


		## The compiler function

		The compiler function is used to stick together a compiler from {lang}_compiler and {lang}_compiler_version

		- `min_pin`: The lower bound of the dependency spec. This is expressed as a `x.x....` version where the `x` are filled in from the corresponding version of the package. For example, `x.x` would be `1.2` for a package version `1.2.3`. The resulting pin spec would look like `>=1.2` for the `min_pin` argument of `x.x` and a version of `1.2.3` for the package.
		- `max_pin`: This defines the upper bound and follows the same `x.x` semantics but adds `+1` to the last segment. For example, `x.x` would be `1.(2 + 1)` for a package version `1.2.3`. The resulting pin spec would look like `<1.3` for the `max_pin` argument of `x.x` and a version of `1.2.3` for the package.


		## The `hash` variable

		`${{ hash }}` is the variant hash and is useful in the build string computation. This used to be `PKG_HASH` in the old recipe format. Since the `hash` variable depends on the variant computation, it is only available in the `build.string` field and is computed after the entire variant computation is finished.


		## The `version_to_buildstring` function

		- `${{ python \| version_to_buildstring }}` converts a version from the variant to a build string (it removes the `.` character and takes only the first two elements of the version).

		@@ -0,0 +1,125 @@
		# Jinja functions in recipes

		The new recipe format has some Jinja functionalities. We want to specify what functions exist and their expected behavior.

	The new recipe format has some Jinja functionalities. We want to specify what functions exist and their expected behavior.
	The new recipe format (introduced by CEP-XX, CEP-XY) has some Jinja functionalities. We want to specify what functions exist and their expected behavior.

		- `${{ env.get("MY_ENV_VAR") }}` will return the value of the environment variable `MY_ENV_VAR` or throw an error if it is not set.
		- `${{ env.get_default("MY_ENV_VAR", "default_value") }}` will return the value of the environment variable `MY_ENV_VAR` or `"default_value"` if it is not set.

		Context evaluation must happen from top-to-bottom. That means a later value can
		reference an earlier one like so:

	The variables can reference other variables from the context section.

	> [!NOTE]
	> The order of the keys is not enforced by the YAML standard. We expect parsers to parse maps (especially the context section)
	> in the order they are defined in the file. However, we do not require this behavior for the recipe to be valid to conform to the YAML standard.
	> Given this, implementations need to ensure topological sorting is done before string interpolation.

-Context evaluation must happen from top-to-bottom. That means a later value can
-reference an earlier one like so:
+The variables can reference other variables from the context section.
+> [!NOTE]
+> The order of the keys is not enforced by the YAML standard. We expect parsers to parse maps (especially the context section)
+> in the order they are defined in the file. However, we do not require this behavior for the recipe to be valid to conform to the YAML standard.
+> Given this, implementations need to ensure topological sorting is done before string interpolation.

		Any variable specified in the variant configuration file can be used in a recipe
		by using it in a Jinja expression. For example, with a variant config like:


		## Error handling

		Build tools should be aggressive about Jinja errors:

		The "old" recipe format has allowed arbitrary Jinja syntax (including set,
		if/else or for loops). The new recipe format only allows a subset of Jinja with

	This used to be `PKG_HASH` in the old recipe format. Since the `hash` variable
	This used to be `PKG_HASH` in the `V0` recipe format. Since the `hash` variable

	optional dependencies. Note that generalizes and replaces selectors from old
	optional dependencies. Note that generalizes and replaces selectors from `V0`

CEP: Jinja functionality in v2 recipes #71

Are you sure you want to change the base?

CEP: Jinja functionality in v2 recipes #71

Conversation

wolfv commented Apr 12, 2024

h-vetinari commented Apr 12, 2024

wolfv commented Apr 13, 2024

h-vetinari commented Apr 13, 2024 • edited Loading

wolfv commented Apr 14, 2024

h-vetinari commented Apr 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wolfv May 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

h-vetinari Jun 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wolfv commented May 20, 2024

h-vetinari commented May 20, 2024

wolfv commented May 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mariusvniekerk Jul 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

h-vetinari commented Apr 13, 2024 •

edited

Loading

h-vetinari commented Apr 14, 2024 •

edited

Loading

wolfv May 10, 2024 •

edited

Loading

h-vetinari Jun 10, 2024 •

edited

Loading

wolfv commented May 21, 2024 •

edited

Loading

mariusvniekerk Jul 2, 2024 •

edited

Loading