rules_python is broken with Bazel@HEAD

[sgowroji]

https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2972#0187a200-864b-46a7-987d-75766ba7fe83

ERROR: /Users/sgowroji/bazelci/bazelhead_downstream/rules_python/rules_python/docs/BUILD.bazel:36:12: no such target '@bazel_tools//tools/python:private/defs.bzl': target 'private/defs.bzl' not declared in package 'tools/python' defined by /private/var/tmp/_bazel_sgowroji/a5aaba8bdde99d8862842eaaed117753/external/bazel_tools/tools/python/BUILD (Tip: use `query "@bazel_tools//tools/python:*"` to see all the targets in that package) and referenced by '//docs:bazel_python_tools'
ERROR: Analysis of target '//docs:pip_install_bzl' failed; build aborted: 
INFO: Elapsed time: 0.666s
INFO: 0 processes.
ERROR: Build did NOT complete successfully

Steps followed to reproduce :

1. git clone -v https://github.com/bazelbuild/rules_python.git
2. git reset c72c7bcf4e0899c275042328e8233e3124ccae86 --hard
3. export USE_BAZEL_VERSION=d3ebbb7495e65747ac5f512c67a790e9dec12373
4. bazel build ...

Culprit : 1895585

CC @meteorcloudy

[rickeylev]

I'm skeptical that 1895585 is the actual cause. I suspect #17545 is more likely the cause (it removed the @bazel_tools//tools/python:private/defs.bzl file that the error complains about).

This sounds like a dupe of #17874, too, but it's not clear.

On the rules_python side, bazelbuild/rules_python#1173 removed the reference to that removed file.

[meteorcloudy]

SG, let's check if the downstream turns green for python tomorrow

[meteorcloudy]

rules_python still fails in downstream:
https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2980#0187b697-43a4-4b00-9a8c-c3fb91364cbf

Looks like there are more places to fix:
https://github.com/bazelbuild/rules_python/blob/main/BUILD.bazel#L54-L59

[rickeylev]

Ok, fixing that error revealed another. Both are fixed at rules_python head now. I ran all the rules_python tests with a build from head and it looked ok (I think; there were failures, but they also happen with non-bazel head, so I think my invocation just wasn't ignoring something it should have).

Let me know if the next downstream build has rules_python failures again.

[sgowroji]

Failures in downstream rules_python (latest pipeline ) : https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2982#0187bb51-ca68-4dae-9701-608fcc7178b1

ERROR: /var/lib/buildkite-agent/builds/bk-docker-41r9/bazel-downstream-projects/rules_python/BUILD.bazel:49:10: no such target '@bazel_tools//tools/python:private/defs.bzl': target 'private/defs.bzl' not declared in package 'tools/python' defined by /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/09bcd3e2e52b893a1df06cbb16f91c53/external/bazel_tools/tools/python/BUILD (Tip: use `query "@bazel_tools//tools/python:*"` to see all the targets in that package) and referenced by '//:bzl'
--
  | (02:22:06) ERROR: /var/lib/buildkite-agent/builds/bk-docker-41r9/bazel-downstream-projects/rules_python/tools/build_defs/python/tests/py_test/BUILD.bazel:18:19: in test_mac_requires_darwin_for_execution_test rule //tools/build_defs/python/tests/py_test:test_mac_requires_darwin_for_execution:
  | Traceback (most recent call last):
  | File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/09bcd3e2e52b893a1df06cbb16f91c53/external/rules_testing/lib/analysis_test.bzl", line 191, column 13, in wrapped_impl
  | impl(env, target)
  | File "/var/lib/buildkite-agent/builds/bk-docker-41r9/bazel-downstream-projects/rules_python/tools/build_defs/python/tests/py_test/py_test_tests.bzl", line 53, column 44, in _test_mac_requires_darwin_for_execution_impl
  | env.expect.that_target(target).provider(
  | File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/09bcd3e2e52b893a1df06cbb16f91c53/external/rules_testing/lib/truth.bzl", line 2323, column 60, in lambda
  | provider = lambda *a, **k: _target_subject_provider(self, *a, **k),
  | File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/09bcd3e2e52b893a1df06cbb16f91c53/external/rules_testing/lib/truth.bzl", line 2499, column 23, in _target_subject_provider
  | info = self.target[provider_key]
  | Error: <merged target //tools/build_defs/python/tests/py_test:test_mac_requires_darwin_for_execution_subject> (rule 'py_test') doesn't contain declared provider 'ExecutionInfo'
  | (02:22:06) ERROR: /var/lib/buildkite-agent/builds/bk-docker-41r9/bazel-downstream-projects/rules_python/tools/build_defs/python/tests/py_test/BUILD.bazel:18:19: Analysis of target '//tools/build_defs/python/tests/py_test:test_mac_requires_darwin_for_execution' failed

[meteorcloudy]

Failures in downstream rules_python (latest pipeline ) : https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2982#0187bb51-ca68-4dae-9701-608fcc7178b1

@sgowroji If you check the rules_python commit in the log, it was still building at bazelbuild/rules_python@ba2a903, which is before the fix was submitted bazelbuild/rules_python@600dbe1

[meteorcloudy]

https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/2983#0187c078-0aeb-4374-b54d-c4073136627d
The @bazel_tools//tools/python:private/defs.bzl issue has been fixed now. But rules_python is still failing with Bazel@HEAD with

(04:22:55) ERROR: /Users/buildkite/builds/bk-imacpro-10/bazel-downstream-projects/rules_python/tools/build_defs/python/tests/py_test/BUILD.bazel:18:19: in test_non_mac_doesnt_require_darwin_for_execution_test rule //tools/build_defs/python/tests/py_test:test_non_mac_doesnt_require_darwin_for_execution:
Traceback (most recent call last):
	File "/private/var/tmp/_bazel_buildkite/8e3e233c9482a5bb2bc225479d1df699/external/rules_testing/lib/analysis_test.bzl", line 191, column 13, in wrapped_impl
		impl(env, target)
	File "/Users/buildkite/builds/bk-imacpro-10/bazel-downstream-projects/rules_python/tools/build_defs/python/tests/py_test/py_test_tests.bzl", line 89, column 28, in _test_non_mac_doesnt_require_darwin_for_execution_impl
		).requirements().keys().not_contains("requires-darwin")
Error: 'struct' value has no field or method 'not_contains'
Available attributes: actual, contains, contains_at_least, contains_at_least_predicates, contains_exactly, contains_exactly_predicates, contains_none_of, contains_predicate, has_size, not_contains_predicate

looks like more API changes we need to adapt?

[meteorcloudy]

@sgowroji Can you file an issue against rules_python and close this one?

[rickeylev]

Thanks, I've sent bazelbuild/rules_python#1183 to fix that failure. How it's manifesting is a bit weird, but the bug is indeed on rules_python's side.