Dependencies: refactor with rules_python update by moderation · Pull Request #14329 · envoyproxy/envoy

moderation · 2020-12-08T20:12:42Z

Commit Message: Dependencies: refactor to remove bazel/repositories_extra.bzl with rules_python update
Additional Description:

rules_python 0.1.0 (release notes). Move pip imports from bazel/repositories_extra.bzl and consolidate into bazel/dependency_imports.bzl
change six from pythonhosted to Github location so we can verify release date
change layout of bazel/repository_locations.bzl so that release_date follows version and sha256 to aid with dependency maintainance
updated tools/dependency/release_dates.py to check for release metadata vs. current approach of checking for latest commit in a release which may not equal release date. Makes additional calls to the Github API and not super efficient but works. /cc @htuch for Python review.

Risk Level: Low
Testing: bazel --nohome_rc test //test/..., bazel --nohome_rc test @envoy_api_canonical//test/... @envoy_api_canonical//tools/..., bazel --nohome_rc build @envoy_api_canonical//envoy/..., tools/dependency/release_dates.py bazel/repository_locations.bzl
Docs Changes: None required
Release Notes: None required

Signed-off-by: Michael Payne michael@sooper.org

…les_python update. Signed-off-by: Michael Payne <michael@sooper.org>

repokitteh-read-only · 2020-12-08T20:12:48Z

CC @envoyproxy/dependency-shepherds: Your approval is needed for changes made to (bazel/.*repos.*\.bzl)|(bazel/dependency_imports\.bzl)|(api/bazel/.*\.bzl)|(.*/requirements\.txt)|(.*\.patch).

🐱

Caused by: #14329 was opened by moderation.

see: more, trace.

Signed-off-by: Michael Payne <michael@sooper.org>

htuch

Thanks! Overall approach makes sense, can you cleanup the Python (re: debug remnants and other nits) and I'll take another pass?
/wait

tools/dependency/release_dates.py

bazel/repository_locations.bzl

… fix release_dates.py. Signed-off-by: Michael Payne <michael@sooper.org>

Signed-off-by: Michael Payne <michael@sooper.org>

moderation · 2020-12-09T21:47:43Z

/retest

repokitteh-read-only · 2020-12-09T21:47:48Z

Retrying Azure Pipelines:
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #14329 (comment) was created by @moderation.

see: more, trace.

htuch

Thanks, just a few logic questions.

tools/dependency/release_dates.py

…ison resilience. Signed-off-by: Michael Payne <michael@sooper.org>

htuch

LGTM modulo wording nit.
/wait

tools/dependency/release_dates.py

htuch · 2020-12-11T21:34:25Z

tools/dependency/release_dates.py

+      for tag in tags.reversed:
+        if tag.name == github_release.version:
+          current_metadata_tag_commit_date = tag.commit.commit.committer.date
+        # Exclude pre-releases like release candidates (eg. apache/kafka)


Something tells me that folks will have ways of creating tag names that are going to break us, but I'm happy to land what you have and iterate.

We already have a bunch of different approaches here with leading v on versions, straight numbers, some with underscores, some with dashes, some with both underscores and dashes and some with alpha numeric strings etc. I think the version import in packaging will help with this and so far the scripts runs OK against our repositories. But definitely a potential source of fragility.

Bigger issue here is a Windows CI failure with the new rules_python 0.1.0 approach - https://github.com/bazelbuild/rules_python/releases/tag/0.1.0. Fails consistently trying to build the Thrift python bits - https://dev.azure.com/cncf/envoy/_build/results?buildId=60601&view=logs&j=b840a642-5ff3-5357-2e4b-e06e40b0cffd&t=67965174-5100-5631-9dc0-68b9f0aacb53. Any ideas @envoyproxy/windows-dev ?

Honest answer is that I dont know why/how this error is coming from. There are multiple mentions across the internet and it seems that this helps other folks.

https://stackoverflow.com/questions/59115861/python-3-6-tensorflow-install-on-windows-failed-with-typeerror-stat-path-shou

I think rules_python might have to update its dependency on setuptools, it is pretty old and looks like the version in use has some incorrect logic around finding the vcruntime*.dll

https://github.com/pypa/setuptools/blob/v51.1.0/setuptools/msvc.py vs. https://github.com/pypa/setuptools/blob/v42.0.2/setuptools/msvc.py

setuptools referenced in rules_python: https://github.com/bazelbuild/rules_python/blob/0.1.0/python/pip_install/repositories.bzl#L19

Specifically in https://github.com/pypa/setuptools/blob/v42.0.2/setuptools/msvc.py#L1540-L1572

I think https://github.com/pypa/setuptools/blob/v42.0.2/setuptools/msvc.py#L1564-L1566 might be wrong

Looking at a build container these are all the paths I see vcruntime*.dll installed:

/c/build/tmp/external/remotejdk11_win/bin/vcruntime140.dll /c/build/tmp/_bazel_ContainerAdministrator/install/7b45c91da117b160cc3b12126324930f/embedded_tools/jdk/bin/vcruntime140.dll /c/Program Files/LLVM/bin/vcruntime140.dll /c/Program Files/LLVM/bin/vcruntime140_1.dll /c/Program Files/Python38/vcruntime140.dll /c/Program Files/Python38/vcruntime140_1.dll /c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Redist/MSVC/14.27.29016/debug_nonredist/x64/Microsoft.VC142.DebugCRT/vcruntime140d.dll /c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Redist/MSVC/14.27.29016/debug_nonredist/x64/Microsoft.VC142.DebugCRT/vcruntime140_1d.dll /c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Redist/MSVC/14.27.29016/onecore/debug_nonredist/x64/Microsoft.VC142.DebugCRT/vcruntime140d.dll /c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Redist/MSVC/14.27.29016/onecore/debug_nonredist/x64/Microsoft.VC142.DebugCRT/vcruntime140_1d.dll /c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Redist/MSVC/14.27.29016/onecore/x64/Microsoft.VC142.CRT/vcruntime140.dll /c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Redist/MSVC/14.27.29016/onecore/x64/Microsoft.VC142.CRT/vcruntime140_1.dll /c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Redist/MSVC/14.27.29016/x64/Microsoft.VC142.CRT/vcruntime140.dll /c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Redist/MSVC/14.27.29016/x64/Microsoft.VC142.CRT/vcruntime140_1.dll /c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Tools/MSVC/14.27.29110/bin/Hostx64/x64/vcruntime140.dll /c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Tools/MSVC/14.27.29110/bin/Hostx64/x64/vcruntime140_1.dll /c/Windows/System32/vcruntime140.dll /c/Windows/System32/vcruntime140d.dll /c/Windows/System32/vcruntime140_1.dll /c/Windows/System32/vcruntime140_1d.dll /c/Windows/SysWOW64/vcruntime140.dll /c/Windows/SysWOW64/vcruntime140d.dll

I think setuptools is looking for /c/Program Files (x86)/Microsoft Visual Studio/2019/BuildTools/VC/Redist/MSVC/14.27.29016/x64/Microsoft.VC142.CRT/vcruntime*.dll but the problematic part is the Microsoft.VC142.CRT rather than Microsoft.VC140.CRT in the path, which is incorrect for the latest versions of MSVC/VCRedist (or inconsistency from Microsoft, idk)

The logic around finding vcruntime*.dll has completely changed so might be worth an update: https://github.com/pypa/setuptools/blob/v51.1.0/setuptools/msvc.py#L217, looks like it handles that path better

This diff seems to have fixed things:

diff --git a/bazel/repositories.bzl b/bazel/repositories.bzl index 92c2b0e84..f4ed59589 100644 --- a/bazel/repositories.bzl +++ b/bazel/repositories.bzl @@ -562,7 +562,11 @@ def _com_google_absl(): ) def _com_google_protobuf(): - external_http_archive("rules_python") + external_http_archive( + name = "rules_python", + patches = ["@envoy//bazel:rules_python.patch"], + patch_args = ["-p1"], + ) external_http_archive( "com_google_protobuf", patches = ["@envoy//bazel:protobuf.patch"], diff --git a/bazel/rules_python.patch b/bazel/rules_python.patch new file mode 100644 index 000000000..81929031d --- /dev/null +++ b/bazel/rules_python.patch @@ -0,0 +1,15 @@ +diff --git a/python/pip_install/repositories.bzl b/python/pip_install/repositories.bzl +index df63674..80824e4 100644 +--- a/python/pip_install/repositories.bzl ++++ b/python/pip_install/repositories.bzl +@@ -16,8 +16,8 @@ _RULE_DEPS = [ + ), + ( + "pypi__setuptools", +- "https://files.pythonhosted.org/packages/54/28/c45d8b54c1339f9644b87663945e54a8503cfef59cf0f65b3ff5dd17cf64/setuptools-42.0.2-py2.py3-none-any.whl", +- "c8abd0f3574bc23afd2f6fd2c415ba7d9e097c8a99b845473b0d957ba1e2dac6", ++ "https://files.pythonhosted.org/packages/3d/f2/1489d3b6c72d68bf79cd0fba6b6c7497df4ebf7d40970e2d7eceb8d0ea9c/setuptools-51.0.0-py3-none-any.whl", ++ "8c177936215945c9a37ef809ada0fab365191952f7a123618432bbfac353c529", + ), + ( + "pypi__wheel",

Signed-off-by: Michael Payne <michael@sooper.org>

htuch · 2020-12-12T23:05:49Z

LGTM @moderation can you back out the problematic rules_python upgrade and do a dedicated PR for that? Seems like something we can hand-off to the Windows folk.

Signed-off-by: Michael Payne <michael@sooper.org>

davinci26 · 2020-12-15T19:14:45Z

Taking a closer look now, trying to repro at my devbox @moderation

sunjayBhatia · 2020-12-23T15:50:19Z

.bazelrc

 build:remote --remote_download_toplevel

 # Windows bazel does not allow sandboxed as a spawn strategy
+build:remote-windows --action_env=WRAPT_INSTALL_EXTENSIONS=false


Probably don't need this and the other change to this file

github-actions · 2021-02-22T20:03:41Z

This pull request has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in 7 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

github-actions · 2021-03-02T00:06:41Z

This pull request has been automatically closed because it has not had activity in the last 37 days. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

Dependencies: refactor to remove bazel/repositories_extra.bzl with ru…

c8166b4

…les_python update. Signed-off-by: Michael Payne <michael@sooper.org>

repokitteh-read-only bot added the deps Approval required for changes to Envoy's external dependencies label Dec 8, 2020

htuch self-assigned this Dec 8, 2020

moderation added 7 commits December 8, 2020 12:54

Fix format and six sha256sum.

88c74c4

Signed-off-by: Michael Payne <michael@sooper.org>

Fix release date for Skywalking API.

d5e81c7

Signed-off-by: Michael Payne <michael@sooper.org>

Correct pip dependency documentation and fix dep validation script.

dc8527d

Signed-off-by: Michael Payne <michael@sooper.org>

Fix todo comment and correct dep validation test.

3a27463

Signed-off-by: Michael Payne <michael@sooper.org>

Add missing pip hashes for new release_dates script.

aaa45b8

Signed-off-by: Michael Payne <michael@sooper.org>

Add missing pip hashes for new release_dates script.

4843dfd

Signed-off-by: Michael Payne <michael@sooper.org>

Sync shadow API bazel/repository_locations.bzl

caa8cff

Signed-off-by: Michael Payne <michael@sooper.org>

htuch suggested changes Dec 9, 2020

View reviewed changes

repokitteh-read-only bot added the waiting label Dec 9, 2020

Address feedback - change order of repository_locations.bzl comments,…

3dd9eae

… fix release_dates.py. Signed-off-by: Michael Payne <michael@sooper.org>

repokitteh-read-only bot removed the waiting label Dec 9, 2020

Merge master.

3324a4c

Signed-off-by: Michael Payne <michael@sooper.org>

moderation requested a review from htuch December 9, 2020 07:48

htuch suggested changes Dec 10, 2020

View reviewed changes

tools/dependency/release_dates.py Outdated Show resolved Hide resolved

tools/dependency/release_dates.py Outdated Show resolved Hide resolved

tools/dependency/release_dates.py Outdated Show resolved Hide resolved

tools/dependency/release_dates.py Outdated Show resolved Hide resolved

Address feedback - remove debug, add comments, improve version compar…

377c07b

…ison resilience. Signed-off-by: Michael Payne <michael@sooper.org>

htuch suggested changes Dec 11, 2020

View reviewed changes

repokitteh-read-only bot added the waiting label Dec 11, 2020

moderation added 2 commits December 11, 2020 15:19

Merge remote-tracking branch 'upstream/master'

3117266

Address wording feedback. Merge master.

b53522d

Signed-off-by: Michael Payne <michael@sooper.org>

repokitteh-read-only bot removed the waiting label Dec 11, 2020

moderation mentioned this pull request Dec 12, 2020

deps: update protobuf to 3.14 #14253

Merged

moderation added 2 commits December 12, 2020 17:33

Merge remote-tracking branch 'upstream/master'

6f82112

Try to get pip installs to work on Windows.

977686c

Signed-off-by: Michael Payne <michael@sooper.org>

Try to get pip installs to work on Windows.

2e03092

Signed-off-by: Michael Payne <michael@sooper.org>

mattklein123 assigned lizan and unassigned htuch Dec 18, 2020

sunjayBhatia reviewed Dec 23, 2020

View reviewed changes

Base automatically changed from master to main January 15, 2021 23:02

github-actions bot added the stale stalebot believes this issue/PR has not been touched recently label Feb 22, 2021

sunjayBhatia mentioned this pull request Mar 1, 2021

bazel: Bump rules_python version to 0.1.0 #15236

Merged

github-actions bot closed this Mar 2, 2021

Conversation

moderation commented Dec 8, 2020

Uh oh!

repokitteh-read-only bot commented Dec 8, 2020

Uh oh!

htuch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

moderation commented Dec 9, 2020

Uh oh!

repokitteh-read-only bot commented Dec 9, 2020

Uh oh!

htuch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

htuch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sunjayBhatia Dec 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

htuch commented Dec 12, 2020

Uh oh!

davinci26 commented Dec 15, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 22, 2021

Uh oh!

github-actions bot commented Mar 2, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sunjayBhatia Dec 21, 2020 •

edited

Loading