Build from source succeeds but subsequent imports fail - also ERROR: Configuration file not found: grain/third_party/py/grain/oss/copy.bara.sky

[SamuelMarks]

@ 77c56dc with Python (CPython) 3.10.16, setuptools 75.6.0, wheel 0.45.1 on x86_64 Linux:

$ python -m pip install .
$ python -m pip install jax
$ cd /tmp # to ensure PATH isn't being used
$ python -c 'import grain.python as grain'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "grain/grain/python.py", line 21, in <module>
    from . import python_experimental as experimental
  File "grain/grain/python_experimental.py", line 53, in <module>
    from ._src.python.dataset.transformations.shuffle import WindowShuffleMapDataset
  File "grain/grain/_src/python/dataset/transformations/shuffle.py", line 22, in <module>
    from grain._src.python.experimental.index_shuffle.python import index_shuffle_module as index_shuffle
ImportError: cannot import name 'index_shuffle_module' from 'grain._src.python.experimental.index_shuffle.python' (unknown location)

Ok so let's dig into your GitHub Actions to see if there's a hint on how to build; since the standard method didn't err:

# from repository root:
$ bazel build ... --action_env PYTHON_BIN_PATH="$(which python)"

ERROR: Skipping 'grain': error loading package 'grain': Unable to find package for @@[unknown repo 'bazel_skylib' requested from @@]//:bzl_library.bzl: The repository '@@[unknown repo 'bazel_skylib' requested from @@]' could not be resolved: No repository visible as '@bazel_skylib' from main repository. Was the repository introduced in WORKSPACE? The WORKSPACE file is disabled by default in Bazel 8 (late 2024) and will be removed in Bazel 9 (late 2025), please migrate to Bzlmod. See https://bazel.build/external/migration.
ERROR: error loading package 'grain': Unable to find package for @@[unknown repo 'bazel_skylib' requested from @@]//:bzl_library.bzl: The repository '@@[unknown repo 'bazel_skylib' requested from @@]' could not be resolved: No repository visible as '@bazel_skylib' from main repository. Was the repository introduced in WORKSPACE? The WORKSPACE file is disabled by default in Bazel 8 (late 2024) and will be removed in Bazel 9 (late 2025), please migrate to Bzlmod. See https://bazel.build/external/migration

Ok that failed, so let's try the scripted approach recommended in your build.md:

$ ./grain/oss/runner.sh
+ SOURCE_DIR=grain
+ OUTPUT_DIR=/tmp/grain
+ mkdir -p /tmp/grain
+ copybara grain/third_party/py/grain/oss/copy.bara.sky local .. --init-history --folder-dir=/tmp/grain --ignore-noop
Dec 20, 2024 9:45:46 P.M. com.google.copybara.Main configureLog
INFO: Setting up LogManager
Copybara source mover (Version: Unknown version)
Task: Running migrate
ERROR: Configuration file not found: grain/third_party/py/grain/oss/copy.bara.sky

ERROR: Try 'copybara help'.

[iindyk]

you can build using these commands: https://github.com/google/grain/blob/main/.github/workflows/tests.yml#L20-L47

why do you need to build from source? grain is available through pip install grain or grain-nightly

[SamuelMarks]

@iindyk Sure, but I'm testing with different versions of Python debugging multiple Google dependencies concurrently to enable a few choice ones to run on the last many versions of Python.

I note this line: https://github.com/google/grain/blob/b54c76c/grain/oss/runner_common.sh#L18

Also taking a quick glance at your Bash scripts, aside from pushd / popd these are shell scripts with the wrong shebang. Would you accept a PR to fix this? - Plenty of distributions don't ship Bash, e.g., ones with musl. - Planning to do interoperability contributions through the dependency chain…

[iindyk]

Would you accept a PR to fix this?

sure, but I'll have to export it into internal change and submit from our internal version control system

[SamuelMarks]

Cool; sent #695