[internal] Use Poetry (for now) to generate lockfiles rather than pip…

…-compile (#12549) **Disclaimer**: This is not a formal commitment to Poetry, as we still need a more rigorous assessment it can handle everything we need. Instead, this is an incremental improvement in that Poetry handles things much better than pip-compile. It gets us closer to the final result we want, and makes it much more ergonomic to use the experimental feature—like `generate_all_lockfiles.sh` now not needing any manual editing. But we may decide to switch from Poetry to something like pdb or Pex. -- See #12470 for why we are not using pip-compile. One of the major motivations is that Poetry generates lockfiles compatible with all requested Python interpreter versions, along with Linux, macOS, and Windows. Meaning, you no longer need to generate the lockfile in each requested environment and manually merge like we used to. This solves #12200 and obviates the need for #12463. -- This PR adds only basic initial support. If we do decide to stick with Poetry, some of the remaining TODOs: - Handle PEP 440-style requirements. - Hook up to `[python-setup]` and `[python-repos]` options. - Hook up to caching. - Support `--platform` via post-processing `poetry.lock`: #12557 - Possibly remove un-used deps/hashes to reduce supply chain attack risk: #12458 -- Poetry is more rigorous than pip-compile in ensuring that all interpreter constraints are valid, which prompted needing to tweak a few of our tools' constraints.
pantsbuild · Aug 13, 2021 · df73917 · df73917
1 parent fcec797
commit df73917
Show file tree

Hide file tree

Showing 26 changed files with 874 additions and 1,140 deletions.
diff --git a/3rdparty/python/lockfiles/coverage_py.txt b/3rdparty/python/lockfiles/coverage_py.txt
@@ -6,46 +6,38 @@
 # invalidation digest: 16ff4a564929a4be3d3d447dff1792ed4f6c625afc4cbc71cc8d303fc7f2a4c0
 # --- END PANTS LOCKFILE METADATA ---
 
-#
-# This file is autogenerated by pip-compile with python 3.7
-# To update, run:
-#
-#    pip-compile --allow-unsafe --generate-hashes --output-file=3rdparty/python/lockfiles/coverage_py.txt reqs.txt
-#
-coverage[toml]==5.0.4 \
-    --hash=sha256:03f630aba2b9b0d69871c2e8d23a69b7fe94a1e2f5f10df5049c0df99db639a0 \
-    --hash=sha256:046a1a742e66d065d16fb564a26c2a15867f17695e7f3d358d7b1ad8a61bca30 \
-    --hash=sha256:0a907199566269e1cfa304325cc3b45c72ae341fbb3253ddde19fa820ded7a8b \
-    --hash=sha256:165a48268bfb5a77e2d9dbb80de7ea917332a79c7adb747bd005b3a07ff8caf0 \
-    --hash=sha256:1b60a95fc995649464e0cd48cecc8288bac5f4198f21d04b8229dc4097d76823 \
-    --hash=sha256:1f66cf263ec77af5b8fe14ef14c5e46e2eb4a795ac495ad7c03adc72ae43fafe \
-    --hash=sha256:2e08c32cbede4a29e2a701822291ae2bc9b5220a971bba9d1e7615312efd3037 \
-    --hash=sha256:3844c3dab800ca8536f75ae89f3cf566848a3eb2af4d9f7b1103b4f4f7a5dad6 \
+coverage==5.0.4; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0" and python_version < "4") \
+    --hash=sha256:8a620767b8209f3446197c0e29ba895d75a1e272a36af0786ec70fe7834e4307 \
+    --hash=sha256:73aa6e86034dad9f00f4bbf5a666a889d17d79db73bc5af04abd6c20a014d9c8 \
     --hash=sha256:408ce64078398b2ee2ec08199ea3fcf382828d2f8a19c5a5ba2946fe5ddc6c31 \
-    --hash=sha256:443be7602c790960b9514567917af538cac7807a7c0c0727c4d2bbd4014920fd \
-    --hash=sha256:4482f69e0701139d0f2c44f3c395d1d1d37abd81bfafbf9b6efbe2542679d892 \
+    --hash=sha256:cda33311cb9fb9323958a69499a667bd728a39a7aa4718d7622597a44c4f1441 \
+    --hash=sha256:5f587dfd83cb669933186661a351ad6fc7166273bc3e3a1531ec5c783d997aac \
+    --hash=sha256:9fad78c13e71546a76c2f8789623eec8e499f8d2d799f4b4547162ce0a4df435 \
+    --hash=sha256:2e08c32cbede4a29e2a701822291ae2bc9b5220a971bba9d1e7615312efd3037 \
+    --hash=sha256:922fb9ef2c67c3ab20e22948dcfd783397e4c043a5c5fa5ff5e9df5529074b0a \
+    --hash=sha256:c3fc325ce4cbf902d05a80daa47b645d07e796a80682c1c5800d6ac5045193e5 \
+    --hash=sha256:046a1a742e66d065d16fb564a26c2a15867f17695e7f3d358d7b1ad8a61bca30 \
+    --hash=sha256:6ad6ca45e9e92c05295f638e78cd42bfaaf8ee07878c9ed73e93190b26c125f7 \
+    --hash=sha256:eda55e6e9ea258f5e4add23bcf33dc53b2c319e70806e180aecbff8d90ea24de \
     --hash=sha256:4a8a259bf990044351baf69d3b23e575699dd60b18460c71e81dc565f5819ac1 \
+    --hash=sha256:f372cdbb240e09ee855735b9d85e7f50730dcfb6296b74b95a3e5dea0615c4c1 \
+    --hash=sha256:a37c6233b28e5bc340054cf6170e7090a4e85069513320275a4dc929144dccf0 \
+    --hash=sha256:443be7602c790960b9514567917af538cac7807a7c0c0727c4d2bbd4014920fd \
+    --hash=sha256:165a48268bfb5a77e2d9dbb80de7ea917332a79c7adb747bd005b3a07ff8caf0 \
+    --hash=sha256:0a907199566269e1cfa304325cc3b45c72ae341fbb3253ddde19fa820ded7a8b \
     --hash=sha256:513e6526e0082c59a984448f4104c9bf346c2da9961779ede1fc458e8e8a1f78 \
-    --hash=sha256:5f587dfd83cb669933186661a351ad6fc7166273bc3e3a1531ec5c783d997aac \
-    --hash=sha256:62061e87071497951155cbccee487980524d7abea647a1b2a6eb6b9647df9006 \
+    --hash=sha256:3844c3dab800ca8536f75ae89f3cf566848a3eb2af4d9f7b1103b4f4f7a5dad6 \
     --hash=sha256:641e329e7f2c01531c45c687efcec8aeca2a78a4ff26d49184dce3d53fc35014 \
+    --hash=sha256:db1d4e38c9b15be1521722e946ee24f6db95b189d1447fa9ff18dd16ba89f732 \
+    --hash=sha256:62061e87071497951155cbccee487980524d7abea647a1b2a6eb6b9647df9006 \
     --hash=sha256:65a7e00c00472cd0f59ae09d2fb8a8aaae7f4a0cf54b2b74f3138d9f9ceb9cb2 \
-    --hash=sha256:6ad6ca45e9e92c05295f638e78cd42bfaaf8ee07878c9ed73e93190b26c125f7 \
-    --hash=sha256:73aa6e86034dad9f00f4bbf5a666a889d17d79db73bc5af04abd6c20a014d9c8 \
-    --hash=sha256:7c9762f80a25d8d0e4ab3cb1af5d9dffbddb3ee5d21c43e3474c84bf5ff941f7 \
+    --hash=sha256:1f66cf263ec77af5b8fe14ef14c5e46e2eb4a795ac495ad7c03adc72ae43fafe \
     --hash=sha256:85596aa5d9aac1bf39fe39d9fa1051b0f00823982a1de5766e35d495b4a36ca9 \
     --hash=sha256:86a0ea78fd851b313b2e712266f663e13b6bc78c2fb260b079e8b67d970474b1 \
-    --hash=sha256:8a620767b8209f3446197c0e29ba895d75a1e272a36af0786ec70fe7834e4307 \
-    --hash=sha256:922fb9ef2c67c3ab20e22948dcfd783397e4c043a5c5fa5ff5e9df5529074b0a \
-    --hash=sha256:9fad78c13e71546a76c2f8789623eec8e499f8d2d799f4b4547162ce0a4df435 \
-    --hash=sha256:a37c6233b28e5bc340054cf6170e7090a4e85069513320275a4dc929144dccf0 \
-    --hash=sha256:c3fc325ce4cbf902d05a80daa47b645d07e796a80682c1c5800d6ac5045193e5 \
-    --hash=sha256:cda33311cb9fb9323958a69499a667bd728a39a7aa4718d7622597a44c4f1441 \
-    --hash=sha256:db1d4e38c9b15be1521722e946ee24f6db95b189d1447fa9ff18dd16ba89f732 \
-    --hash=sha256:eda55e6e9ea258f5e4add23bcf33dc53b2c319e70806e180aecbff8d90ea24de \
-    --hash=sha256:f372cdbb240e09ee855735b9d85e7f50730dcfb6296b74b95a3e5dea0615c4c1
-    # via -r reqs.txt
-toml==0.10.2 \
+    --hash=sha256:03f630aba2b9b0d69871c2e8d23a69b7fe94a1e2f5f10df5049c0df99db639a0 \
+    --hash=sha256:7c9762f80a25d8d0e4ab3cb1af5d9dffbddb3ee5d21c43e3474c84bf5ff941f7 \
+    --hash=sha256:4482f69e0701139d0f2c44f3c395d1d1d37abd81bfafbf9b6efbe2542679d892 \
+    --hash=sha256:1b60a95fc995649464e0cd48cecc8288bac5f4198f21d04b8229dc4097d76823
+toml==0.10.2; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version < "4" \
     --hash=sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b \
     --hash=sha256:b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f
-    # via coverage
diff --git a/3rdparty/python/lockfiles/flake8.txt b/3rdparty/python/lockfiles/flake8.txt
@@ -2,63 +2,40 @@
 #
 #    build-support/bin/generate_all_lockfiles.sh
 #
-# You MUST manually add back the `python_version` environment markers. (This will be automated.)
 # --- BEGIN PANTS LOCKFILE METADATA: DO NOT EDIT OR REMOVE ---
 # invalidation digest: 443653bdbc8facef9b521f860e1c11e48d670630150c948dbd5ae444d0796c19
 # --- END PANTS LOCKFILE METADATA ---
 
-#
-# This file is autogenerated by pip-compile with python 3.7
-# To update, run:
-#
-#    pip-compile --allow-unsafe --generate-hashes --output-file=3rdparty/python/lockfiles/flake8.txt reqs.txt
-#
-flake8==3.9.2 \
-    --hash=sha256:07528381786f2a6237b061f6e96610a4167b226cb926e2aa2b6b1d78057c576b \
-    --hash=sha256:bf8fd333346d844f616e8d47905ef3a3384edae6b4e9beb0c5101e25e3110907
-    # via
-    #   -r reqs.txt
-    #   flake8-2020
-    #   flake8-pantsbuild
-flake8-2020==1.6.0 \
-    --hash=sha256:3e438d9d531577fbb9332bbd0bf394eca890f3b7a5311e2278fb3582381a2f13 \
-    --hash=sha256:e464cef7889117e7ae7253e35fcb8272bd8ae68fc10bb3399ef85b036c092528
-    # via -r reqs.txt
-flake8-pantsbuild==2.0.0 \
+flake8-2020==1.6.0; python_full_version >= "3.6.1" \
+    --hash=sha256:e464cef7889117e7ae7253e35fcb8272bd8ae68fc10bb3399ef85b036c092528 \
+    --hash=sha256:3e438d9d531577fbb9332bbd0bf394eca890f3b7a5311e2278fb3582381a2f13
+flake8-pantsbuild==2.0.0; python_version >= "3.6" \
     --hash=sha256:1b72558db6d718c33f4410eff80f7afc2bc0300190a733d92b3cf05b231c2450 \
     --hash=sha256:8d45cf26a55fe7d66de944a6c93b64456852cfd3a7bea81640553081469129e7
-    # via -r reqs.txt
-importlib-metadata==4.6.3 ; python_version == '3.7' \
-    --hash=sha256:0645585859e9a6689c523927a5032f2ba5919f1f7d0e84bd4533312320de1ff9 \
-    --hash=sha256:51c6635429c77cf1ae634c997ff9e53ca3438b495f10a55ba28594dd69764a8b
-    # via
-    #   flake8
-    #   flake8-2020
-    #   flake8-pantsbuild
-mccabe==0.6.1 \
+flake8==3.9.2; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0") \
+    --hash=sha256:bf8fd333346d844f616e8d47905ef3a3384edae6b4e9beb0c5101e25e3110907 \
+    --hash=sha256:07528381786f2a6237b061f6e96610a4167b226cb926e2aa2b6b1d78057c576b
+importlib-metadata==4.6.3; python_full_version >= "3.6.1" and python_version < "3.8" and python_version >= "3.6" and (python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.5.0" and python_version < "3.8" and python_version >= "3.6") \
+    --hash=sha256:51c6635429c77cf1ae634c997ff9e53ca3438b495f10a55ba28594dd69764a8b \
+    --hash=sha256:0645585859e9a6689c523927a5032f2ba5919f1f7d0e84bd4533312320de1ff9
+mccabe==0.6.1; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6" \
     --hash=sha256:ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42 \
     --hash=sha256:dd8d182285a0fe56bace7f45b5e7d1a6ebcbf524e8f3bd87eb0f125271b8831f
-    # via flake8
-pycodestyle==2.7.0 \
+pycodestyle==2.7.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6" \
     --hash=sha256:514f76d918fcc0b55c6680472f0a37970994e07bbb80725808c17089be302068 \
     --hash=sha256:c389c1d06bf7904078ca03399a4816f974a1d590090fecea0c63ec26ebaf1cef
-    # via flake8
-pyflakes==2.3.1 \
+pyflakes==2.3.1; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6" \
     --hash=sha256:7893783d01b8a89811dd72d7dfd4d84ff098e5eed95cfa8905b22bbffe52efc3 \
     --hash=sha256:f5bc8ecabc05bb9d291eb5203d6810b49040f6ff446a756326104746cc00c1db
-    # via flake8
-typing-extensions==3.10.0.0 ; python_version == '3.7' \
+setuptools==44.1.1; python_full_version == "2.7.*" or python_version > "2.7" \
+    --hash=sha256:27a714c09253134e60a6fa68130f78c7037e5562c4f21f8f318f2ae900d152d5 \
+    --hash=sha256:c67aa55db532a0dadc4d2e20ba9961cbd3ccc84d544e9029699822542b5a476b \
+    --hash=sha256:a49230977aa6cfb9d933614d2f7b79036e9945c4cdd7583163f4e920b83418d6 \
+    --hash=sha256:6bac238ffdf24e8806c61440e755192470352850f3419a52f26ffe0a1a64f465
+typing-extensions==3.10.0.0; python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.5.0" and python_version < "3.8" and python_version >= "3.6" \
     --hash=sha256:0ac0f89795dd19de6b97debb0c6af1c70987fd80a2d62d1958f7e56fcc31b497 \
-    --hash=sha256:50b6f157849174217d0656f99dc82fe932884fb250826c18350e159ec6cdf342 \
-    --hash=sha256:779383f6086d90c99ae41cf0ff39aac8a7937a9283ce0a414e5dd782f4c94a84
-    # via importlib-metadata
-zipp==3.5.0 ; python_version == '3.7' \
+    --hash=sha256:779383f6086d90c99ae41cf0ff39aac8a7937a9283ce0a414e5dd782f4c94a84 \
+    --hash=sha256:50b6f157849174217d0656f99dc82fe932884fb250826c18350e159ec6cdf342
+zipp==3.5.0; python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.5.0" and python_version < "3.8" and python_version >= "3.6" \
     --hash=sha256:957cfda87797e389580cb8b9e3870841ca991e2125350677b2ca83a0e99390a3 \
     --hash=sha256:f5812b1e007e48cff63449a5e9f4e7ebea716b4111f9c4f9a645f91d579bf0c4
-    # via importlib-metadata
-
-# The following packages are considered to be unsafe in a requirements file:
-setuptools==57.4.0 ; python_version > "2.7" \
-    --hash=sha256:6bac238ffdf24e8806c61440e755192470352850f3419a52f26ffe0a1a64f465 \
-    --hash=sha256:a49230977aa6cfb9d933614d2f7b79036e9945c4cdd7583163f4e920b83418d6
-    # via -r reqs.txt