Better error messages

[codebreach]

Thanks so much for this repository! we were able to do cross-compiled docker builds using this :)

a couple of Developer Experience issues we identified:

• improve messaging for The files field must not be empty. This is caused by metadata.files being empty in package.lock which can happen for poetry <= 1.1.3 Missing wheels (metadata.files all empty lists) python-poetry/poetry#6045. At least outputting the package name in the assert would help understand what to look for.

• KeyError: None

  bzl_lock_generator.runfiles/jvolkman_rules_pycross/pycross/private/tools/bzl_lock_generator.py", line 261, in 
  get_package_sources_by_environment
      environment_sources[environment.name] = candidates_to_package_sources[
  KeyError: None
  

  this can be caused if there is no wheel or even target available for a os + arch mix. This happens for tensorflow@~2 on Mac M1

For both these I have local changes that make the error clearer. Let me know if you are open to a PR.

[jvolkman]

Yes, please send over a PR!

For the second issue, is there also not an sdist available? I would have expected that it would try to generate a pycross_wheel_build target.

[codebreach]

I will share the PR soon...

The tensorflow problem I think its because they dont publish a source wheel anyway. So it cant be compiled on M1. There is a different package tensorflow-macos which exports the same symbols but works on macos.

I was able to get poetry to resolve by doing

tensorflow = {version = "2.10.0", platform = "linux"}
tensorflow-macos = {version = "2.10.0", platform = "darwin"}

now i think i need to manually write a pycross_wheel_library target which uses pypi_file but haven't yet figured that out.

was thinking of something like:

pycross_wheel_library(
        name = "tensorflow_2.11.0",
        deps = [], # TODO
        wheel = select({
            ":_env_python_darwin_arm64": "@lock_wheel_tenso_macos_cp39_cp39_macosx_11_0_arm64//file",
            ":_env_python_darwin_x86_64": "@lock_wheel_tenso_cp39_cp3_macosx_10_14_x86_64//file",
            ":_env_python_linux_x86_64": "@lock_wheel_ tensorflow_2_11_0_cp39_cp39_manylinux_2_17_x86_64.manylinux2014_x86_64//file",
        }),
    )

i think this should work but havent gotten a chance to try it yet

[codebreach]

actually the pycross_wheel_library with override doesn't work...

pyproject has

tensorflow = {version = "2.10.0", platform = "linux"}
tensorflow-macos = {version = "2.10.0", platform = "darwin"}

with the following lock file

pycross_lock_file(
    name = "lock",
    out = "lock.bzl",
    build_target_overrides = {
        "[email protected]": "@//utils/pypi:overridden_tensorflow_2.10.0",
    },
   ...

the override is defined as follows

maybe(
    pypi_file,
    name = "lock_wheel_tensorflow_macos_2.10.0_cp39_cp39_macosx_12_0_arm64",
    package_name = "tensorflow-macos",
    package_version = "2.10.0",
    filename = "tensorflow_macos-2.10.0-cp39-cp39-macosx_12_0_arm64.whl",
    sha256 = "f2ec603c5496c25fb1bcda8eb4166423bf023bfb7ae6cbdec0be8796ca67e866",
    index = "https://pypi.org",
)

maybe(
    pypi_file,
    name = "lock_wheel_tensorflow_2.10.0_cp39_cp39_macosx_10_14_x86_64",
    package_name = "tensorflow",
    package_version = "2.10.0",
    filename = "tensorflow-2.10.0-cp39-cp39-macosx_10_14_x86_64.whl",
    sha256 = "4b542af76d93c43e9d24dcb69888793831e434dc781c9533ee07f928fce84a15",
    index = "https://pypi.org",
)


maybe(
    pypi_file,
    name = "lock_wheel_tensorflow_2.10.0_cp39_cp39_linux_2_17_x86_64",
    package_name = "tensorflow",
    package_version = "2.10.0",
    filename = "tensorflow-2.10.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
    sha256 = "487918f4074685e213ba247387faab34933df76939134008441cb9d3e2c95cab",
    index = "https://pypi.org",
)

_tensorflow_2_10_0_deps = [
        "//:absl_py_1.4.0",
        "//:astunparse_1.6.3",
        "//:flatbuffers_23.1.21",
        "//:gast_0.4.0",
        "//:google_pasta_0.2.0",
        "//:grpcio_1.51.3",
        "//:h5py_3.8.0",
        "//:keras_2.10.0",
        # incomplete...
    ]

pycross_wheel_library(
    name = "overridden_tensorflow_2.10.0",
    deps = _tensorflow_2_10_0_deps,
    wheel = select({
        ":_env_python_darwin_arm64": "@lock_wheel_tensorflow_macos_2.10.0_cp39_cp39_macosx_12_0_arm64//file",
        ":_env_python_darwin_x86_64": "@lock_wheel_tensorflow_2.10.0_cp39_cp39_macosx_10_14_x86_64//file",
        ":_env_python_linux_x86_64": "@lock_wheel_tensorflow_2.10.0_cp39_cp39_linux_2_17_x86_64//file",
    }),
)

here is the output with my added assert

bazel-out/host/bin/external/jvolkman_rules_pycross/pycross/private/tools/bzl_lock_generator.runfiles/jvolkman_rules_pycross/pycross/private/tools/bzl_lock_generator.py", line 261, in get_package_sources_by_environment
    assert compute_result.best_candidate is not None, f"""compute_result.best_candidate is none for {package.name}
AssertionError: compute_result.best_candidate is none for tensorflow
            candidates: []
            package_soruces: {'tensorflow-2.10.0-cp310-cp310-macosx_10_14_x86_64.whl': PackageSource(label=None, file=PackageFile(name='tensorflow-2.10.0-cp310-cp310-macosx_10_14_x86_64.whl', sha256='60d5b4fbbb7a1304d96352372fa032e861e98bb3f23aced7ce53bc475a2df97d', urls=None)), 'tensorflow-2.10.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl': PackageSource(label=None, file=PackageFile(name='tensorflow-2.10.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl', sha256='25e1e898bc1df521af9a8bfe0e511124379a6414083234ec67c6ab212ad12b2f', urls=None)), 'tensorflow-2.10.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl': PackageSource(label=None, file=PackageFile(name='tensorflow-2.10.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl', sha256='e129114dc529e63af9c419b5917b3407d0d26a4c8b73e114f601a175a7eb0477', urls=None)), 'tensorflow-2.10.0-cp310-cp310-win_amd64.whl': PackageSource(label=None, file=PackageFile(name='tensorflow-2.10.0-cp310-cp310-win_amd64.whl', sha256='0a3b58d90fadb5bdf81a964bea73bb89019a9d1e9ac12de75375c8f65e0d7570', urls=None)), 'tensorflow-2.10.0-cp37-cp37m-macosx_10_14_x86_64.whl': PackageSource(label=None, file=PackageFile(name='tensorflow-2.10.0-cp37-cp37m-macosx_10_14_x86_64.whl', sha256='0701da16a3d6d34763cd9ced6467cee24c02c9abf0d1a48ba59ea5a8d0421cec', urls=None)), 'tensorflow-2.10.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl': PackageSource(label=None, file=PackageFile(name='tensorflow-2.10.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl', sha256='64cc999ae83ddd891083141d3e5d718e3d799501a1b56c544f2ca648a8396c3e', urls=None)), 'tensorflow-2.10.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl': PackageSource(label=None, file=PackageFile(name='tensorflow-2.10.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl', sha256='d9f711c5ff04333355c83eb96ca2e1db57c9663c6fa01d68b5953a040a602a3c', urls=None)), 'tensorflow-2.10.0-cp37-cp37m-win_amd64.whl': PackageSource(label=None, file=PackageFile(name='tensorflow-2.10.0-cp37-cp37m-win_amd64.whl', sha256='9f4677e9ab7104e73710a94ff5d2ed4b335378dcd2ac7402a68c31802a680911', urls=None)), 'tensorflow-2.10.0-cp38-cp38-macosx_10_14_x86_64.whl': PackageSource(label=None, file=PackageFile(name='tensorflow-2.10.0-cp38-cp38-macosx_10_14_x86_64.whl', sha256='8773858cbf37aaad444b07605d29f5b2d8f7cd1ecbf1cce2777931b96884589c', urls=None)), 'tensorflow-2.10.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl': PackageSource(label=None, file=PackageFile(name='tensorflow-2.10.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl', sha256='5806d4645bce5eb415863d757b5f056364b9d1cfa2c34f711f69d46cac605eee', urls=None)), 'tensorflow-2.10.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl': PackageSource(label=None, file=PackageFile(name='tensorflow-2.10.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl', sha256='e85f89bc23c62d4243fad70bac902f00a234b33da8b91e2967eeef0f4b75b1e3', urls=None)), 'tensorflow-2.10.0-cp38-cp38-win_amd64.whl': PackageSource(label=None, file=PackageFile(name='tensorflow-2.10.0-cp38-cp38-win_amd64.whl', sha256='d9b19b5120c0b393d9e2fc72561cfa3a454ef7f1ac649d8ad0dcc98817a086a4', urls=None)), 'tensorflow-2.10.0-cp39-cp39-macosx_10_14_x86_64.whl': PackageSource(label=None, file=PackageFile(name='tensorflow-2.10.0-cp39-cp39-macosx_10_14_x86_64.whl', sha256='4b542af76d93c43e9d24dcb69888793831e434dc781c9533ee07f928fce84a15', urls=None)), 'tensorflow-2.10.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl': PackageSource(label=None, file=PackageFile(name='tensorflow-2.10.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl', sha256='c588a1f34d9db51ea856aff07da9aa877c1d1d109336eee2c3bbb16dabd3f605', urls=None)), 'tensorflow-2.10.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl': PackageSource(label=None, file=PackageFile(name='tensorflow-2.10.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl', sha256='487918f4074685e213ba247387faab34933df76939134008441cb9d3e2c95cab', urls=None)), 'tensorflow-2.10.0-cp39-cp39-win_amd64.whl': PackageSource(label=None, file=PackageFile(name='tensorflow-2.10.0-cp39-cp39-win_amd64.whl', sha256='741a74278f471dc21991a6c7dc802d454d42fd39515900c6363b8c38a898fb0f', urls=None))}
----------------
Note: The failure of target @jvolkman_rules_pycross//pycross/private/tools:bzl_lock_generator (with exit code 1) may have been caused by the fact that it is running under Python 3 instead of Python 2. Examine the error to determine if that appears to be the problem. Since this target is built in the host configuration, the only way to change its version is to set --host_force_python=PY2, which affects the entire build.

If this error started occurring in Bazel 0.27 and later, it may be because the Python toolchain now enforces that targets analyzed as PY2 and PY3 run under a Python 2 and Python 3 interpreter, respectively. See https://github.com/bazelbuild/bazel/issues/7899 for more information.

[jvolkman]

I'll try to take a look at this sometime this week.

[jvolkman]

Does #31 fix your issue? I feel like the full solution is something larger, but hopefully this unblocks you.

[codebreach]

it doesn't fix in that it allows me to use tensorflow-macos as an alias for tensorflow on macs but i think the failure is a bit cleaner.

I am travelling but will try to run with this commit and let you know

cc @dheerajiiitv

[jvolkman]

Closing; feel free to reopen if this is still an issue.