Add Grain feedstock

[mtsokol]

Checklist

• Title of this PR is meaningful: e.g. "Adding my_nifty_package", not "updated meta.yaml".
• License file is packaged (see here for an example).
• Source is from official source.
• Package does not vendor other packages. (If a package uses the source of another package, they should be separate packages or the licenses of all packages need to be packaged).
• If static libraries are linked in, the license of the static library is packaged.
• Package does not ship static libraries. If static libraries are needed, follow CFEP-18.
• Build number is 0.
• A tarball (url) rather than a repo (e.g. git_url) is used in your recipe (see here for more details).
• GitHub users listed in the maintainer section have posted a comment confirming they are willing to be listed there.
• When in trouble, please check our knowledge base documentation before pinging a team.

[conda-forge-admin]

Hi! This is the friendly automated conda-forge-linting service.

I failed to even lint the recipe, probably because of a conda-smithy bug 😢. This likely indicates a problem in your meta.yaml, though. To get a traceback to help figure out what's going on, install conda-smithy and run conda smithy recipe-lint --conda-forge . from the recipe directory. You can also examine the workflow logs for more detail.

_{This message was generated by GitHub Actions workflow run https://github.com/conda-forge/conda-forge-webservices/actions/runs/15568776179. Examine the logs at this URL for more detail.}

[gapplef]

@mtsokol Any update?

[mtsokol]

Right, I'll have more time in the next two weeks to finish it.

[gapplef]

Thanks a lot, really appreciate your effort.

[conda-forge-admin]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/grain/meta.yaml) and found it was in an excellent condition.

[github-actions]

Hi! This is the staged-recipes linter and I found some lint.

File-specific lints and/or hints:

• recipes/grain/meta.yaml:
  • lints:
    • The following maintainers have not yet confirmed that they are willing to be listed here: iindyk. Please ask them to comment on this PR if they are.

[mtsokol]

Hi @iindyk,
I come back with some findings: In the Grain recipe bazel is a host and build dependency, and it's taken from its conda-forge feedstock. It looks like there's no 7.1.1 published: https://anaconda.org/conda-forge/bazel/files?page=3, there's only 7.2.1 which is the closest, but then the build complains about platforms dependency being 0.0.8, so AFAIC we need to update Grain to 7.2.1 to make it work here in the conda forge. Or is closest lower version also considerable? (a.k.a. 6.5.0)

[iindyk]

Hi, I'd like to be a maintainer of this feedstock

[iindyk]

Hi @iindyk, I come back with some findings: In the Grain recipe bazel is a host and build dependency, and it's taken from its conda-forge feedstock. It looks like there's no 7.1.1 published: https://anaconda.org/conda-forge/bazel/files?page=3, there's only 7.2.1 which is the closest, but then the build complains about platforms dependency being 0.0.8, so AFAIC we need to update Grain to 7.2.1 to make it work here in the conda forge. Or is closest lower version also considerable? (a.k.a. 6.5.0)

7.2.1 is ok as long as the chain of dep updates will not force us to use a new protobuf dependency -- that is going to be painful. I think downgrade may be even more painful

[mtsokol]

Thanks! Let me try to bump bazel to 7.2.1 then.

[github-actions]

Hi! This is the staged-recipes linter and your PR looks excellent! 🚀

[mtsokol]

This is pure C++ code, whose only dependency is pybind11, so it should only required 2 invocation of the compiler for each Python version, so more and less ~10 compiler invocation in total. Furthermore, as only a function is exposed (and not a class) I guess it is perfectly ok to avoid depending on pybind11-abi. However, for some reason the setuptools+bazel decides that it needs to rebuild bits of abseil and protobuf for each Python version, resulting in a total build time of ~1 hour. As CI resources available to conda-forge are kindly donated by conda-forge sponsors are not infinite, do you think there is some way to avoid protobuf to be compiled everytime as it is not required to actually build the output of the packages?

@iindyk Do you know if that's something we have influence over?

[traversaro]

As CI resources available to conda-forge are kindly donated by conda-forge sponsors are not infinite, do you think there is some way to avoid protobuf to be compiled everytime as it is not required to actually build the output of the packages?

For example, perhaps we can run bazel build //grain/_src/python/experimental/index_shuffle/python:index_shuffle_module instead of bazel build ... to only build what we actually need?

[traversaro]

See #30294 (comment) .

[mtsokol]

For example, perhaps we can run bazel build //grain/_src/python/experimental/index_shuffle/python:index_shuffle_module instead of bazel build ... to only build what we actually need?

@traversaro Unfortunately we can't narrow the build to this cpp extension only. There is also https://github.com/google/grain/blob/main/grain/proto/execution_summary.proto protobuf file which needs to be build with bazel and I think it's the delinquent of abseil and protobuf builds that you reported.

Is it a deal-breaker for conda-forge?

[traversaro]

For example, perhaps we can run bazel build //grain/_src/python/experimental/index_shuffle/python:index_shuffle_module instead of bazel build ... to only build what we actually need?

@traversaro Unfortunately we can't narrow the build to this cpp extension only. There is also https://github.com/google/grain/blob/main/grain/proto/execution_summary.proto protobuf file which needs to be build with bazel and I think it's the delinquent of abseil and protobuf builds that you reported.

Ah, I see! It would be cool to have some way to let bazel use the conda-forge's precompiled protobuf and abseil (I played around with this in https://github.com/ami-iit/bazel-cmake-deps-override in the past) but I guess this is not trivial.

Is it a deal-breaker for conda-forge?

As long as the releases are not so often, I guess this is not a big problem, even if the planet would probably be happier if we avoid wasting CI time. : )

However, this reference to grain/proto/execution_summary.proto leads me to another question. The package contains the generated grain/proto/execution_summary_pb2.py , but as far as protobuf-generated Python code has a strict dependency on the major version of protobuf used to generated it (I encountered this in https://github.com/conda-forge/gz-msgs-feedstock). By looking in the docs, in https://protobuf.dev/support/cross-version-runtime-guarantee/#python I found:

Python may break generated code compatibility in specific future major version releases, but it will be coupled with poison pill warnings and errors in advance. As of 6.32.0, all generated code since 3.20.0 will be supported until at least 8.x.y.

So if are sure that protobuf used by bazel to generate the Python is at least 6.32.0 should be fine?

[mtsokol]

As long as the releases are not so often

It looks like Grain has been being released once a month so far and it shouldn't increase IMO.

The argument is undoubtedly correct - I can take a look at the precompiled bazel approach, but first I'll reach out to bazel team asking about it - here the bazel setup is relatively minimal, just one proto file. I'll see in case they have suggestions. The feedstock PR can either wait or we can ship it and have precompiled attempt separately.

So if are sure that protobuf used by bazel to generate the Python is at least 6.32.0 should be fine?

Here we can't afford selecting protobuf version - we use exactly the one that TensorFlow uses to avoid version conflicts link. It's 28.3 right now and in some future version TF will surely reach 32.

[traversaro]

Here we can't afford selecting protobuf version - we use exactly the one that TensorFlow uses to avoid version conflicts link. It's 28.3 right now and in some future version TF will surely reach 32.

In that case, can you look into adding an appropriate upper bound for the protobuf version used at runtime? To avoid errors like https://github.com/grpc/grpc/issues/39912?utm_source=chatgpt.com . It may also make sense to import the generated protobuf file in the tests, to ensure that at least the protobuf used for testing is compatible with the generated file.

[mtsokol]

In that case, can you look into adding an appropriate upper bound for the protobuf version used at runtime? To avoid errors like https://github.com/grpc/grpc/issues/39912?utm_source=chatgpt.com . It may also make sense to import the generated protobuf file in the tests, to ensure that at least the protobuf used for testing is compatible with the generated file.

@traversaro Sure, I added protobuf generated module import in test: section.

About an upper bound constraint: I would advocate to keep protobuf version constraint as >=5.28.3, as it is right now. The strict version pin regards MODULE.bazel file in Grain only, because that's the version used for building proto generated module during bazel build (and in that generated module there's protobuf version hardcoded). And there it must match TF protobuf version.

For runtime dependencies upper bound isn't necessarily suitable:

1. JAX version that Grain requires in runtime is >=0.6.0 which needs protobuf 31.x (just tried it here) so fixing protobuf to 5.28 in runtime is unsatisfiable, and in runtime version mismatch doesn't cause issues here.
2. Grain authors also prefer >= for protobuf runtime dependency Update ArrayRecord google/grain#999 (comment) to not constrain downstream to much.

I think the root cause here is the duality of using bazel within conda-forge build tools/setup which was already painful for array_record.

Would you agree to leave it >= for protobuf then?

[traversaro]

JAX version that Grain requires in runtime is >=0.6.0 which needs protobuf 31.x (just tried it here) so fixing protobuf to 5.28 in runtime is unsatisfiable, and in runtime version mismatch doesn't cause issues here.

Sorry, why would you want to fox protobuf to 5.28 ? From what I understand from https://protobuf.dev/support/cross-version-runtime-guarantee/#python, shouldn't we just constraint protobuf to be < 8.* ?

Grain authors also prefer >= for protobuf runtime dependency google/grain#999 (comment) to not constrain downstream to much.

I think the pro/cons evaluation may be a bit different in conda-forge, where we have repodata patches. On PyPI the metadata is effectively immutable, while in conda-forge we can patch it after the release.

[mtsokol]

Ah, sorry! I misunderstood your request (for some reason I thought about fixing a specific version for it). Right, this upper bound makes sense - I added it.

[traversaro]

Can you add a comment pointing to the protobuf docs? Otherwise it is quite difficult to track where this <8.0.0 comes from.

[mtsokol]

Done!

[jaimergp]

Not to be super pedantic, but usually these upper bounds are written like <8.0dev0 or similar to avoid RCs and pre-releases getting in the way.

Suggested change

	- protobuf >=5.28.3,<8.0.0
	- protobuf >=5.28.3,<8.0dev0

[traversaro]

Thanks for the observation! @mtsokol probably it is better to do this before merging so the first build already have the correct metadata, thanks!

[mtsokol]

Done! I included .rc0 that is the earliest RC which covers all potential protobuf releases of 8.x.

[traversaro]

@mtsokol sorry for being pedantic, but the @jaimergp suggestion was to use 8.0dev0 as upper bound, do you have any specific reason for using 8.0.0.rc0 instead?

[mtsokol]

I can make it 8.0dev0, I haven't found any dev releases for protobuf in its conda-forge feedstock or pypi.

[mtsokol]

Now it also considers dev in the release.

[traversaro]

I can make it 8.0dev0, I haven't found any dev releases for protobuf in its conda-forge feedstock or pypi.

That make sense, but please explain changes, as you may imagine reviewers can't read what you have in mind. : )

Anyhow, I guess the @jaimergp suggestion should cover also the rc case, so I think probably sticking to 8.0dev0 make sense.

[mtsokol]

Sorry about that! And thank you for all your assistance for this PR!

[traversaro]

Not sure if @jaimergp has more comments, but this is ok for me! If you want to squash and rebase on the latest master that is welcome!

[conda-forge-admin]

Hi! This is the friendly automated conda-forge-linting service.

I failed to even lint the recipe, probably because of a conda-smithy bug 😢. This likely indicates a problem in your meta.yaml, though. To get a traceback to help figure out what's going on, install conda-smithy and run conda smithy recipe-lint --conda-forge . from the recipe directory. You can also examine the workflow logs for more detail.

_{This message was generated by GitHub Actions workflow run https://github.com/conda-forge/conda-forge-webservices/actions/runs/19272559607. Examine the logs at this URL for more detail.}

[conda-forge-admin]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/grain/meta.yaml) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipes/grain/meta.yaml:

• ℹ️ The recipe is not parsable by parser conda-souschef (grayskull). This parser is not currently used by conda-forge, but may be in the future. We are collecting information to see which recipes are compatible with grayskull.

_{This message was generated by GitHub Actions workflow run https://github.com/conda-forge/conda-forge-webservices/actions/runs/19292653517. Examine the logs at this URL for more detail.}

[jaimergp]

Not sure if @jaimergp has more comments

Nothing blocking, go ahead!

[traversaro]

For example, perhaps we can run bazel build //grain/_src/python/experimental/index_shuffle/python:index_shuffle_module instead of bazel build ... to only build what we actually need?

@traversaro Unfortunately we can't narrow the build to this cpp extension only. There is also https://github.com/google/grain/blob/main/grain/proto/execution_summary.proto protobuf file which needs to be build with bazel and I think it's the delinquent of abseil and protobuf builds that you reported.

Is it a deal-breaker for conda-forge?

Just a cross-ref to a couple bazel issues that could help in the future:

• ENH: equivalent of local_cc_dependency override in bzlmod bazelbuild/bazel#20947
• bzlmod: Mechanism for optionally using a system-provided library in place of a version fetched by bazel bazelbuild/bazel#20581

Even if the protobuf/absl case is definitely a difficult, as it requires somehow to also handle code generators, so probably in any case it would always require some glue code as not all the information required by bazel is present in pkgconfig, CMake config or cps files.