Allow subsetting a PEX lockfile in PEX format, not just pip #2411

huonw · 2024-05-20T07:21:22Z

A lockfile pins a potentially-huge universe of dependencies, and there's several use-cases where efficiently cutting that down to only a smaller applicable set would be handy:

debugging/reducing issues (e.g. start with a huge lockfile for a production codebase, and cut it down to only the couple of dependencies required for sharing a bug report to Pex or Pants or ...)
content-based caching of process executions like Pants.
1. currently: any execution has the whole lockfile as an input, meaning potentially-expensive steps like "build a whole .pex file" are spuriously invalidated by changes to dependencies they don't use
2. with this feature: pants could first execute a quick (and deterministic) "subset lockfile" execution, and then feed the resulting subset into the "build a whole .pex package" execution

This might be able to be implemented as a new pex value (or similar) for pex3 lock export-subset --format=...

To make this more concrete, a workflow might be:

A lockfile that contains both cowsay and tensorflow, e.g. PEX_SCRIPT=pex3 pex lock create cowsay tensorflow -o test.lock (contents at the end)
Build a pex that uses only tensorflow: pex tensorflow --lock test.lock -o test.pex, within some system that does process-based caching like Pants or Bazel. (Even with a warm PEX cache, this takes ~30s on my machine. I'm aware of the various settings that can improve this, but that's orthogonal to this feature request, I think.)
Upgrade cowsay without changing tensorflow or any of its dependent libraries.
Rerun step 2 and hope that it's fast...

Currently, naive (aka reliable) entire-file-based caching of the process execution in step 2, will mean step 4's rerun has to execute and cannot be served from cache.

If we had the requested feature, the process runners could instead do two steps to build this PEX:

create a lockfile with only the relevant deps: pex3 lock export-subset --format=pex --lock=test.lock tensorflow -o reduced.lock (the --format=pip version takes about 0.5s on my machine)
build the PEX using that lockfile instead: pex tensorflow --lock reduced.lock -o test.pex

Under this scheme, when cowsay changes, the pex3 lock export-subset --format=pex invocation is invalidated (its input has changed), and has to rerun... but the reduced.lock output will be identical, and thus pex tensorflow --lock reduced.lock -o test.pex can be served from cache. The export-subset invocation is very fast in comparison to the full build, and thus this feature would unlock more efficient use of PEX.

{
  "allow_builds": true,
  "allow_prereleases": false,
  "allow_wheels": true,
  "build_isolation": true,
  "constraints": [],
  "locked_resolves": [
    {
      "locked_requirements": [
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "526a04eadab8b4ee719ce68f204172ead1027549089702d99b9059f129ff1308",
              "url": "https://files.pythonhosted.org/packages/a2/ad/e0d3c824784ff121c03cc031f944bc7e139a8f1870ffd2845cc2dd76f6c4/absl_py-2.1.0-py3-none-any.whl"
            }
          ],
          "project_name": "absl-py",
          "requires_dists": [],
          "requires_python": ">=3.7",
          "version": "2.1.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "c2652417f2c8b5bb325c885ae329bdf3f86424075c4fd1a128674bc6fba4b8e8",
              "url": "https://files.pythonhosted.org/packages/2b/03/13dde6512ad7b4557eb792fbcf0c653af6076b81e5941d36ec61f7ce6028/astunparse-1.6.3-py2.py3-none-any.whl"
            }
          ],
          "project_name": "astunparse",
          "requires_dists": [
            "six<2.0,>=1.6.1",
            "wheel<1.0,>=0.23.0"
          ],
          "requires_python": null,
          "version": "1.6.3"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "dc383c07b76109f368f6106eee2b593b04a011ea4d55f652c6ca24a754d1cdd1",
              "url": "https://files.pythonhosted.org/packages/ba/06/a07f096c664aeb9f01624f858c3add0a4e913d6c96257acb4fce61e7de14/certifi-2024.2.2-py3-none-any.whl"
            }
          ],
          "project_name": "certifi",
          "requires_dists": [],
          "requires_python": ">=3.6",
          "version": "2024.2.2"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "9063e24fdb1e498ab71cb7419e24622516c4a04476b17a2dab57e8baa30d6e03",
              "url": "https://files.pythonhosted.org/packages/46/6a/d5c26c41c49b546860cc1acabdddf48b0b3fb2685f4f5617ac59261b44ae/charset_normalizer-3.3.2-cp310-cp310-macosx_11_0_arm64.whl"
            }
          ],
          "project_name": "charset-normalizer",
          "requires_dists": [],
          "requires_python": ">=3.7.0",
          "version": "3.3.2"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "274b1e6fc1b966d53976333eb90ac94cb07a450a700b455af9fbdf882244b30a",
              "url": "https://files.pythonhosted.org/packages/f1/13/63c0a02c44024ee16f664e0b36eefeb22d54e93531630bd99e237986f534/cowsay-6.1-py3-none-any.whl"
            }
          ],
          "project_name": "cowsay",
          "requires_dists": [],
          "requires_python": ">=3.8",
          "version": "6.1"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "8dbdec58f935f3765e4f7f3cf635ac3a77f83568138d6a2311f524ec96364812",
              "url": "https://files.pythonhosted.org/packages/41/f0/7e988a019bc54b2dbd0ad4182ef2d53488bb02e58694cd79d61369e85900/flatbuffers-24.3.25-py2.py3-none-any.whl"
            }
          ],
          "project_name": "flatbuffers",
          "requires_dists": [],
          "requires_python": null,
          "version": "24.3.25"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "6fc4fa5fa10b72fb8aab4ae58bcb023058386e67b6fa2e3e34cec5c769360316",
              "url": "https://files.pythonhosted.org/packages/fa/39/5aae571e5a5f4de9c3445dae08a530498e5c53b0e74410eeeb0991c79047/gast-0.5.4-py3-none-any.whl"
            }
          ],
          "project_name": "gast",
          "requires_dists": [],
          "requires_python": "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,>=2.7",
          "version": "0.5.4"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "b32482794a366b5366a32c92a9a9201b107821889935a02b3e51f6b432ea84ed",
              "url": "https://files.pythonhosted.org/packages/a3/de/c648ef6835192e6e2cc03f40b19eeda4382c49b5bafb43d88b931c4c74ac/google_pasta-0.2.0-py3-none-any.whl"
            }
          ],
          "project_name": "google-pasta",
          "requires_dists": [
            "six"
          ],
          "requires_python": null,
          "version": "0.2.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "91b73d3f1340fefa1e1716c8c1ec9930c676d6b10a3513ab6c26004cb02d8b3f",
              "url": "https://files.pythonhosted.org/packages/2f/f9/10bae7df60ceab06f71a9000a6c8cfe3c442800560a613ccb17bad07e82b/grpcio-1.63.0-cp310-cp310-macosx_12_0_universal2.whl"
            }
          ],
          "project_name": "grpcio",
          "requires_dists": [
            "grpcio-tools>=1.63.0; extra == \"protobuf\""
          ],
          "requires_python": ">=3.8",
          "version": "1.63.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "c072655ad1d5fe9ef462445d3e77a8166cbfa5e599045f8aa3c19b75315f10e5",
              "url": "https://files.pythonhosted.org/packages/d4/03/bbb9a992fb43d3ce46687b7c14107f0fa56e6c8704c9ca945a9392cbc8ce/h5py-3.11.0-cp310-cp310-macosx_11_0_arm64.whl"
            }
          ],
          "project_name": "h5py",
          "requires_dists": [
            "numpy>=1.17.3"
          ],
          "requires_python": ">=3.8",
          "version": "3.11.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "82fee1fc78add43492d3a1898bfa6d8a904cc97d8427f683ed8e798d07761aa0",
              "url": "https://files.pythonhosted.org/packages/e5/3e/741d8c82801c347547f8a2a06aa57dbb1992be9e948df2ea0eda2c8b79e8/idna-3.7-py3-none-any.whl"
            }
          ],
          "project_name": "idna",
          "requires_dists": [],
          "requires_python": ">=3.5",
          "version": "3.7"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "260df9ef71c6b89eb6816ce1c60f139c38ccdddd16f24e7005d2be127cdef8e4",
              "url": "https://files.pythonhosted.org/packages/8d/44/c604ecc5c9993b6574a681f2f505e980725871a89cfd9e48597b12ccb506/keras-3.3.3-py3-none-any.whl"
            }
          ],
          "project_name": "keras",
          "requires_dists": [
            "absl-py",
            "h5py",
            "ml-dtypes",
            "namex",
            "numpy",
            "optree",
            "rich"
          ],
          "requires_python": ">=3.9",
          "version": "3.3.3"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "83ce5045d101b669ac38e6da8e58765f12da2d3aafb3b9b98d88b286a60964d8",
              "url": "https://files.pythonhosted.org/packages/db/ed/1df62b44db2583375f6a8a5e2ca5432bbdc3edb477942b9b7c848c720055/libclang-18.1.1-py2.py3-none-macosx_11_0_arm64.whl"
            }
          ],
          "project_name": "libclang",
          "requires_dists": [],
          "requires_python": null,
          "version": "18.1.1"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "48f276f4d8cfb8ce6527c8f79e2ee29708508bf4d40aa410fbc3b4ee832c850f",
              "url": "https://files.pythonhosted.org/packages/fc/b3/0c0c994fe49cd661084f8d5dc06562af53818cc0abefaca35bdc894577c3/Markdown-3.6-py3-none-any.whl"
            }
          ],
          "project_name": "markdown",
          "requires_dists": [
            "coverage; extra == \"testing\"",
            "importlib-metadata>=4.4; python_version < \"3.10\"",
            "mdx-gh-links>=0.2; extra == \"docs\"",
            "mkdocs-gen-files; extra == \"docs\"",
            "mkdocs-literate-nav; extra == \"docs\"",
            "mkdocs-nature>=0.6; extra == \"docs\"",
            "mkdocs-section-index; extra == \"docs\"",
            "mkdocs>=1.5; extra == \"docs\"",
            "mkdocstrings[python]; extra == \"docs\"",
            "pyyaml; extra == \"testing\""
          ],
          "requires_python": ">=3.8",
          "version": "3.6"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "355216845c60bd96232cd8d8c40e8f9765cc86f46880e43a8fd22dc1a1a8cab1",
              "url": "https://files.pythonhosted.org/packages/42/d7/1ec15b46af6af88f19b8e5ffea08fa375d433c998b8a7639e76935c14f1f/markdown_it_py-3.0.0-py3-none-any.whl"
            }
          ],
          "project_name": "markdown-it-py",
          "requires_dists": [
            "commonmark~=0.9; extra == \"compare\"",
            "coverage; extra == \"testing\"",
            "gprof2dot; extra == \"profiling\"",
            "jupyter_sphinx; extra == \"rtd\"",
            "linkify-it-py<3,>=1; extra == \"linkify\"",
            "markdown~=3.4; extra == \"compare\"",
            "mdit-py-plugins; extra == \"plugins\"",
            "mdit-py-plugins; extra == \"rtd\"",
            "mdurl~=0.1",
            "mistletoe~=1.0; extra == \"compare\"",
            "mistune~=2.0; extra == \"compare\"",
            "myst-parser; extra == \"rtd\"",
            "panflute~=2.3; extra == \"compare\"",
            "pre-commit~=3.0; extra == \"code-style\"",
            "psutil; extra == \"benchmarking\"",
            "pytest-benchmark; extra == \"benchmarking\"",
            "pytest-cov; extra == \"testing\"",
            "pytest-regressions; extra == \"testing\"",
            "pytest; extra == \"benchmarking\"",
            "pytest; extra == \"testing\"",
            "pyyaml; extra == \"rtd\"",
            "sphinx-copybutton; extra == \"rtd\"",
            "sphinx-design; extra == \"rtd\"",
            "sphinx; extra == \"rtd\"",
            "sphinx_book_theme; extra == \"rtd\""
          ],
          "requires_python": ">=3.8",
          "version": "3.0.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "a17a92de5231666cfbe003f0e4b9b3a7ae3afb1ec2845aadc2bacc93ff85febc",
              "url": "https://files.pythonhosted.org/packages/e4/54/ad5eb37bf9d51800010a74e4665425831a9db4e7c4e0fde4352e391e808e/MarkupSafe-2.1.5-cp310-cp310-macosx_10_9_universal2.whl"
            }
          ],
          "project_name": "markupsafe",
          "requires_dists": [],
          "requires_python": ">=3.7",
          "version": "2.1.5"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8",
              "url": "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl"
            }
          ],
          "project_name": "mdurl",
          "requires_dists": [],
          "requires_python": ">=3.7",
          "version": "0.1.2"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "7afde548890a92b41c0fed3a6c525f1200a5727205f73dc21181a2726571bb53",
              "url": "https://files.pythonhosted.org/packages/62/0a/2b586fd10be7b8311068f4078623a73376fc49c8b3768be9965034062982/ml_dtypes-0.3.2-cp310-cp310-macosx_10_9_universal2.whl"
            }
          ],
          "project_name": "ml-dtypes",
          "requires_dists": [
            "absl-py; extra == \"dev\"",
            "numpy>1.20",
            "numpy>=1.21.2; python_version >= \"3.10\"",
            "numpy>=1.23.3; python_version >= \"3.11\"",
            "numpy>=1.26.0; python_version >= \"3.12\"",
            "pyink; extra == \"dev\"",
            "pylint>=2.6.0; extra == \"dev\"",
            "pytest-xdist; extra == \"dev\"",
            "pytest; extra == \"dev\""
          ],
          "requires_python": ">=3.9",
          "version": "0.3.2"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "7ddb6c2bb0e753a311b7590f84f6da659dd0c05e65cb89d519d54c0a250c0487",
              "url": "https://files.pythonhosted.org/packages/73/59/7854fbfb59f8ae35483ce93493708be5942ebb6328cd85b3a609df629736/namex-0.0.8-py3-none-any.whl"
            }
          ],
          "project_name": "namex",
          "requires_dists": [],
          "requires_python": null,
          "version": "0.0.8"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "2e4ee3380d6de9c9ec04745830fd9e2eccb3e6cf790d39d7b98ffd19b0dd754a",
              "url": "https://files.pythonhosted.org/packages/20/f7/b24208eba89f9d1b58c1668bc6c8c4fd472b20c45573cb767f59d49fb0f6/numpy-1.26.4-cp310-cp310-macosx_11_0_arm64.whl"
            }
          ],
          "project_name": "numpy",
          "requires_dists": [],
          "requires_python": ">=3.9",
          "version": "1.26.4"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "2455e59e3947d3c275477df7f5205b30635e266fe6dc300e3d9f9646bfcea147",
              "url": "https://files.pythonhosted.org/packages/bc/19/404708a7e54ad2798907210462fd950c3442ea51acc8790f3da48d2bee8b/opt_einsum-3.3.0-py3-none-any.whl"
            }
          ],
          "project_name": "opt-einsum",
          "requires_dists": [
            "numpy>=1.7",
            "numpydoc; extra == \"docs\"",
            "pytest-cov; extra == \"tests\"",
            "pytest-pep8; extra == \"tests\"",
            "pytest; extra == \"tests\"",
            "sphinx-rtd-theme; extra == \"docs\"",
            "sphinx==1.2.3; extra == \"docs\"",
            "sphinxcontrib-napoleon; extra == \"docs\""
          ],
          "requires_python": ">=3.5",
          "version": "3.3.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "0df9a3923725aabb112ec7f10c74fa96b6c640da1cd30e7bc62fd4b03ef02875",
              "url": "https://files.pythonhosted.org/packages/3e/4a/fadc346b3c83f0eb3b2cd91cf5ccad2851ac3411c2e6d33c9cfd54b3659a/optree-0.11.0-cp310-cp310-macosx_11_0_arm64.whl"
            }
          ],
          "project_name": "optree",
          "requires_dists": [
            "black>=22.6.0; extra == \"lint\"",
            "cpplint; extra == \"lint\"",
            "dm-tree<0.2.0a0,>=0.1; extra == \"benchmark\"",
            "doc8<1.0.0a0; extra == \"lint\"",
            "docutils; extra == \"docs\"",
            "flake8-bugbear; extra == \"lint\"",
            "flake8-comprehensions; extra == \"lint\"",
            "flake8-docstrings; extra == \"lint\"",
            "flake8-pyi; extra == \"lint\"",
            "flake8-simplify; extra == \"lint\"",
            "flake8; extra == \"lint\"",
            "isort>=5.11.0; extra == \"lint\"",
            "jax; extra == \"jax\"",
            "jax[cpu]; extra == \"docs\"",
            "jax[cpu]<0.5.0a0,>=0.4.6; extra == \"benchmark\"",
            "mypy>=0.990; extra == \"lint\"",
            "numpy; extra == \"docs\"",
            "numpy; extra == \"numpy\"",
            "pandas; extra == \"benchmark\"",
            "pre-commit; extra == \"lint\"",
            "pydocstyle; extra == \"lint\"",
            "pyenchant; extra == \"lint\"",
            "pylint[spelling]>=2.15.0; extra == \"lint\"",
            "pytest-cov; extra == \"test\"",
            "pytest-xdist; extra == \"test\"",
            "pytest; extra == \"test\"",
            "ruff; extra == \"lint\"",
            "sphinx-autoapi; extra == \"docs\"",
            "sphinx-autobuild; extra == \"docs\"",
            "sphinx-autodoc-typehints>=1.19.2; extra == \"docs\"",
            "sphinx-copybutton; extra == \"docs\"",
            "sphinx-rtd-theme; extra == \"docs\"",
            "sphinx>=5.2.1; extra == \"docs\"",
            "sphinxcontrib-bibtex; extra == \"docs\"",
            "tabulate; extra == \"benchmark\"",
            "termcolor; extra == \"benchmark\"",
            "torch; extra == \"docs\"",
            "torch; extra == \"torch\"",
            "torch<2.1.0a0,>=2.0; extra == \"benchmark\"",
            "torchvision; extra == \"benchmark\"",
            "typing-extensions>=4.0.0",
            "xdoctest; extra == \"lint\""
          ],
          "requires_python": ">=3.7",
          "version": "0.11.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "2ddfb553fdf02fb784c234c7ba6ccc288296ceabec964ad2eae3777778130bc5",
              "url": "https://files.pythonhosted.org/packages/49/df/1fceb2f8900f8639e278b056416d49134fb8d84c5942ffaa01ad34782422/packaging-24.0-py3-none-any.whl"
            }
          ],
          "project_name": "packaging",
          "requires_dists": [],
          "requires_python": ">=3.7",
          "version": "24.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "f1279ab38ecbfae7e456a108c5c0681e4956d5b1090027c1de0f934dfdb4b35c",
              "url": "https://files.pythonhosted.org/packages/f3/bf/26deba06a4c910a85f78245cac7698f67cedd7efe00d04f6b3e1b3506a59/protobuf-4.25.3-cp37-abi3-macosx_10_9_universal2.whl"
            }
          ],
          "project_name": "protobuf",
          "requires_dists": [],
          "requires_python": ">=3.8",
          "version": "4.25.3"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "b8e6aca0523f3ab76fee51799c488e38782ac06eafcf95e7ba832985c8e7b13a",
              "url": "https://files.pythonhosted.org/packages/f7/3f/01c8b82017c199075f8f788d0d906b9ffbbc5a47dc9918a945e13d5a2bda/pygments-2.18.0-py3-none-any.whl"
            }
          ],
          "project_name": "pygments",
          "requires_dists": [
            "colorama>=0.4.6; extra == \"windows-terminal\""
          ],
          "requires_python": ">=3.8",
          "version": "2.18.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "58cd2187c01e70e6e26505bca751777aa9f2ee0b7f4300988b709f44e013003f",
              "url": "https://files.pythonhosted.org/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl"
            }
          ],
          "project_name": "requests",
          "requires_dists": [
            "PySocks!=1.5.7,>=1.5.6; extra == \"socks\"",
            "certifi>=2017.4.17",
            "chardet<6,>=3.0.2; extra == \"use-chardet-on-py3\"",
            "charset-normalizer<4,>=2",
            "idna<4,>=2.5",
            "urllib3<3,>=1.21.1"
          ],
          "requires_python": ">=3.7",
          "version": "2.31.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "4edbae314f59eb482f54e9e30bf00d33350aaa94f4bfcd4e9e3110e64d0d7222",
              "url": "https://files.pythonhosted.org/packages/87/67/a37f6214d0e9fe57f6ae54b2956d550ca8365857f42a1ce0392bb21d9410/rich-13.7.1-py3-none-any.whl"
            }
          ],
          "project_name": "rich",
          "requires_dists": [
            "ipywidgets<9,>=7.5.1; extra == \"jupyter\"",
            "markdown-it-py>=2.2.0",
            "pygments<3.0.0,>=2.13.0",
            "typing-extensions<5.0,>=4.0.0; python_version < \"3.9\""
          ],
          "requires_python": ">=3.7.0",
          "version": "13.7.1"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "c636ac361bc47580504644275c9ad802c50415c7522212252c033bd15f301f32",
              "url": "https://files.pythonhosted.org/packages/f7/29/13965af254e3373bceae8fb9a0e6ea0d0e571171b80d6646932131d6439b/setuptools-69.5.1-py3-none-any.whl"
            }
          ],
          "project_name": "setuptools",
          "requires_dists": [
            "build[virtualenv]; extra == \"testing\"",
            "build[virtualenv]>=1.0.3; extra == \"testing-integration\"",
            "filelock>=3.4.0; extra == \"testing\"",
            "filelock>=3.4.0; extra == \"testing-integration\"",
            "furo; extra == \"docs\"",
            "importlib-metadata; extra == \"testing\"",
            "ini2toml[lite]>=0.9; extra == \"testing\"",
            "jaraco.develop>=7.21; (python_version >= \"3.9\" and sys_platform != \"cygwin\") and extra == \"testing\"",
            "jaraco.envs>=2.2; extra == \"testing\"",
            "jaraco.envs>=2.2; extra == \"testing-integration\"",
            "jaraco.packaging>=9.3; extra == \"docs\"",
            "jaraco.path>=3.2.0; extra == \"testing\"",
            "jaraco.path>=3.2.0; extra == \"testing-integration\"",
            "jaraco.tidelift>=1.4; extra == \"docs\"",
            "mypy==1.9; extra == \"testing\"",
            "packaging>=23.2; extra == \"testing\"",
            "packaging>=23.2; extra == \"testing-integration\"",
            "pip>=19.1; extra == \"testing\"",
            "pygments-github-lexers==0.0.5; extra == \"docs\"",
            "pytest!=8.1.1,>=6; extra == \"testing\"",
            "pytest-checkdocs>=2.4; extra == \"testing\"",
            "pytest-cov; platform_python_implementation != \"PyPy\" and extra == \"testing\"",
            "pytest-enabler; extra == \"testing-integration\"",
            "pytest-enabler>=2.2; extra == \"testing\"",
            "pytest-home>=0.5; extra == \"testing\"",
            "pytest-mypy; extra == \"testing\"",
            "pytest-perf; sys_platform != \"cygwin\" and extra == \"testing\"",
            "pytest-ruff>=0.2.1; sys_platform != \"cygwin\" and extra == \"testing\"",
            "pytest-timeout; extra == \"testing\"",
            "pytest-xdist; extra == \"testing-integration\"",
            "pytest-xdist>=3; extra == \"testing\"",
            "pytest; extra == \"testing-integration\"",
            "rst.linker>=1.9; extra == \"docs\"",
            "sphinx-favicon; extra == \"docs\"",
            "sphinx-inline-tabs; extra == \"docs\"",
            "sphinx-lint; extra == \"docs\"",
            "sphinx-notfound-page<2,>=1; extra == \"docs\"",
            "sphinx-reredirects; extra == \"docs\"",
            "sphinx>=3.5; extra == \"docs\"",
            "sphinxcontrib-towncrier; extra == \"docs\"",
            "tomli-w>=1.0.0; extra == \"testing\"",
            "tomli; extra == \"testing\"",
            "tomli; extra == \"testing-integration\"",
            "virtualenv>=13.0.0; extra == \"testing\"",
            "virtualenv>=13.0.0; extra == \"testing-integration\"",
            "wheel; extra == \"testing\"",
            "wheel; extra == \"testing-integration\""
          ],
          "requires_python": ">=3.8",
          "version": "69.5.1"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254",
              "url": "https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl"
            }
          ],
          "project_name": "six",
          "requires_dists": [],
          "requires_python": "!=3.0.*,!=3.1.*,!=3.2.*,>=2.7",
          "version": "1.16.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "9f2b4e7dad86667615c0e5cd072f1ea8403fc032a299f0072d6f74855775cc45",
              "url": "https://files.pythonhosted.org/packages/3a/d0/b97889ffa769e2d1fdebb632084d5e8b53fc299d43a537acee7ec0c021a3/tensorboard-2.16.2-py3-none-any.whl"
            }
          ],
          "project_name": "tensorboard",
          "requires_dists": [
            "absl-py>=0.4",
            "grpcio>=1.48.2",
            "markdown>=2.6.8",
            "numpy>=1.12.0",
            "protobuf!=4.24.0,>=3.19.6",
            "setuptools>=41.0.0",
            "six>1.9",
            "tensorboard-data-server<0.8.0,>=0.7.0",
            "werkzeug>=1.0.1"
          ],
          "requires_python": ">=3.9",
          "version": "2.16.2"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "7e0610d205889588983836ec05dc098e80f97b7e7bbff7e994ebb78f578d0ddb",
              "url": "https://files.pythonhosted.org/packages/7a/13/e503968fefabd4c6b2650af21e110aa8466fe21432cd7c43a84577a89438/tensorboard_data_server-0.7.2-py3-none-any.whl"
            }
          ],
          "project_name": "tensorboard-data-server",
          "requires_dists": [],
          "requires_python": ">=3.7",
          "version": "0.7.2"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "8e376ab46fb1df18a1f927d77011d36ecf7b717a81cbfe4a941c7bf5236939b3",
              "url": "https://files.pythonhosted.org/packages/7d/01/bee34cf4d207cc5ae4f445c0e743691697cd89359a24a5fcdcfa8372f042/tensorflow-2.16.1-cp310-cp310-macosx_12_0_arm64.whl"
            }
          ],
          "project_name": "tensorflow",
          "requires_dists": [
            "absl-py>=1.0.0",
            "astunparse>=1.6.0",
            "flatbuffers>=23.5.26",
            "gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1",
            "google-pasta>=0.1.1",
            "grpcio<2.0,>=1.24.3",
            "h5py>=3.10.0",
            "keras>=3.0.0",
            "libclang>=13.0.0",
            "ml-dtypes~=0.3.1",
            "numpy<2.0.0,>=1.23.5; python_version <= \"3.11\"",
            "numpy<2.0.0,>=1.26.0; python_version >= \"3.12\"",
            "nvidia-cublas-cu12==12.3.4.1; extra == \"and-cuda\"",
            "nvidia-cuda-cupti-cu12==12.3.101; extra == \"and-cuda\"",
            "nvidia-cuda-nvcc-cu12==12.3.107; extra == \"and-cuda\"",
            "nvidia-cuda-nvrtc-cu12==12.3.107; extra == \"and-cuda\"",
            "nvidia-cuda-runtime-cu12==12.3.101; extra == \"and-cuda\"",
            "nvidia-cudnn-cu12==8.9.7.29; extra == \"and-cuda\"",
            "nvidia-cufft-cu12==11.0.12.1; extra == \"and-cuda\"",
            "nvidia-curand-cu12==10.3.4.107; extra == \"and-cuda\"",
            "nvidia-cusolver-cu12==11.5.4.101; extra == \"and-cuda\"",
            "nvidia-cusparse-cu12==12.2.0.103; extra == \"and-cuda\"",
            "nvidia-nccl-cu12==2.19.3; extra == \"and-cuda\"",
            "nvidia-nvjitlink-cu12==12.3.101; extra == \"and-cuda\"",
            "opt-einsum>=2.3.2",
            "packaging",
            "protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3",
            "requests<3,>=2.21.0",
            "setuptools",
            "six>=1.12.0",
            "tensorboard<2.17,>=2.16",
            "tensorflow-io-gcs-filesystem>=0.23.1; python_version < \"3.12\"",
            "termcolor>=1.1.0",
            "typing-extensions>=3.6.6",
            "wrapt>=1.11.0"
          ],
          "requires_python": ">=3.9",
          "version": "2.16.1"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "677d6d7c84a94a3b27ea5d16633ea09adadef09c2630480e8e94209558828b02",
              "url": "https://files.pythonhosted.org/packages/29/85/311c94bbebe8182a52b70100eac483d948f754d5417b6291b9daecd0faeb/tensorflow_io_gcs_filesystem-0.37.0-cp310-cp310-macosx_12_0_arm64.whl"
            }
          ],
          "project_name": "tensorflow-io-gcs-filesystem",
          "requires_dists": [
            "tensorflow-aarch64<2.17.0,>=2.16.0; extra == \"tensorflow-aarch64\"",
            "tensorflow-cpu<2.17.0,>=2.16.0; extra == \"tensorflow-cpu\"",
            "tensorflow-gpu<2.17.0,>=2.16.0; extra == \"tensorflow-gpu\"",
            "tensorflow-rocm<2.17.0,>=2.16.0; extra == \"tensorflow-rocm\"",
            "tensorflow<2.17.0,>=2.16.0; extra == \"tensorflow\""
          ],
          "requires_python": "<3.12,>=3.7",
          "version": "0.37.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "9297c0df9c99445c2412e832e882a7884038a25617c60cea2ad69488d4040d63",
              "url": "https://files.pythonhosted.org/packages/d9/5f/8c716e47b3a50cbd7c146f45881e11d9414def768b7cd9c5e6650ec2a80a/termcolor-2.4.0-py3-none-any.whl"
            }
          ],
          "project_name": "termcolor",
          "requires_dists": [
            "pytest-cov; extra == \"tests\"",
            "pytest; extra == \"tests\""
          ],
          "requires_python": ">=3.8",
          "version": "2.4.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "c1f94d72897edaf4ce775bb7558d5b79d8126906a14ea5ed1635921406c0387a",
              "url": "https://files.pythonhosted.org/packages/01/f3/936e209267d6ef7510322191003885de524fc48d1b43269810cd589ceaf5/typing_extensions-4.11.0-py3-none-any.whl"
            }
          ],
          "project_name": "typing-extensions",
          "requires_dists": [],
          "requires_python": ">=3.8",
          "version": "4.11.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d",
              "url": "https://files.pythonhosted.org/packages/a2/73/a68704750a7679d0b6d3ad7aa8d4da8e14e151ae82e6fee774e6e0d05ec8/urllib3-2.2.1-py3-none-any.whl"
            }
          ],
          "project_name": "urllib3",
          "requires_dists": [
            "brotli>=1.0.9; platform_python_implementation == \"CPython\" and extra == \"brotli\"",
            "brotlicffi>=0.8.0; platform_python_implementation != \"CPython\" and extra == \"brotli\"",
            "h2<5,>=4; extra == \"h2\"",
            "pysocks!=1.5.7,<2.0,>=1.5.6; extra == \"socks\"",
            "zstandard>=0.18.0; extra == \"zstd\""
          ],
          "requires_python": ">=3.8",
          "version": "2.2.1"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "fc9645dc43e03e4d630d23143a04a7f947a9a3b5727cd535fdfe155a17cc48c8",
              "url": "https://files.pythonhosted.org/packages/9d/6e/e792999e816d19d7fcbfa94c730936750036d65656a76a5a688b57a656c4/werkzeug-3.0.3-py3-none-any.whl"
            }
          ],
          "project_name": "werkzeug",
          "requires_dists": [
            "MarkupSafe>=2.1.1",
            "watchdog>=2.3; extra == \"watchdog\""
          ],
          "requires_python": ">=3.8",
          "version": "3.0.3"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "55c570405f142630c6b9f72fe09d9b67cf1477fcf543ae5b8dcb1f5b7377da81",
              "url": "https://files.pythonhosted.org/packages/7d/cd/d7460c9a869b16c3dd4e1e403cce337df165368c71d6af229a74699622ce/wheel-0.43.0-py3-none-any.whl"
            }
          ],
          "project_name": "wheel",
          "requires_dists": [
            "pytest>=6.0.0; extra == \"test\"",
            "setuptools>=65; extra == \"test\""
          ],
          "requires_python": ">=3.8",
          "version": "0.43.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "e4fdb9275308292e880dcbeb12546df7f3e0f96c6b41197e0cf37d2826359020",
              "url": "https://files.pythonhosted.org/packages/32/12/e11adfde33444986135d8881b401e4de6cbb4cced046edc6b464e6ad7547/wrapt-1.16.0-cp310-cp310-macosx_11_0_arm64.whl"
            }
          ],
          "project_name": "wrapt",
          "requires_dists": [],
          "requires_python": ">=3.6",
          "version": "1.16.0"
        }
      ],
      "platform_tag": [
        "cp310",
        "cp310",
        "macosx_14_0_arm64"
      ]
    }
  ],
  "only_builds": [],
  "only_wheels": [],
  "path_mappings": {},
  "pex_version": "2.3.1",
  "pip_version": "20.3.4-patched",
  "prefer_older_binary": false,
  "requirements": [
    "cowsay",
    "tensorflow"
  ],
  "requires_python": [],
  "resolver_version": "pip-legacy-resolver",
  "style": "strict",
  "target_systems": [],
  "transitive": true,
  "use_pep517": null
}

The text was updated successfully, but these errors were encountered:

huonw · 2024-05-20T07:25:35Z

I'm interested to try to implement this, and I imagine the implementation of export-subset is good inspiration. I've found pex.cli.commands.Lock._export_pip (and its callers) and pex.resolve.lockfile.subset but it's not... obvious to me how to turn the Subset/Resolved etc. types back into a pex lockfile. So, a hint there would be very helpful.

jsirois · 2024-05-24T02:03:18Z

Why does the existing export-subset, fed back in to pex -r ... not suffice? The resulting requirements file has hashes, and the underlying pip download call checks these; so is as good as a lock afaict. I, in fact, use this here:

jsirois · 2024-05-24T03:03:09Z

A side note: you often reference commands like PEX_SCRIPT=pex3 pex .... This is extremely non-idiomatic / Pants specific. In almost all cases pex will be the pex console script, for which your command line does not work. It's only when pex is the Pex PEX, that this style of command line makes sense. You are no doubt aware of this, but the issues read strange to be sure. Perhaps I should start publishing the Pex PEX as pex.pex to help avoid this confusing overload.

huonw · 2024-05-24T04:19:54Z

Why does the existing export-subset, fed back in to pex -r ... not suffice? The resulting requirements file has hashes, and the underlying pip download call checks these; so is as good as a lock afaict. I, in fact, use this here:

Ah, if you think it works sufficiently well/captures everything it needs to, I'm happy to use that as official guidance.

I have a confirmation question: I note that export-subset currently has a warning suggesting a lockfile with multiple locks isn't supported. This would suggest to me it doesn't cover all circumstances, but... maybe those circumstances don't matter in practice?

I can see this potentially not mattering for the Pants/caching use case, but I imagine it may matter for the "reduce a lockfile for a bug report" case?

This is extremely non-idiomatic

Ah, sorry. I like the all-in-one pex a lot: I have plopped it into my $PATH and then use it to install random Python tools (unrelated to Pants). Much better than installing pex (or the other tools) into a global venv, and seems a bit silly to have two pexes on disk with different entry-points (and potentially different versions).

I acknowledge that it's a little confusing to have two slightly different meanings of the pex name.

Perhaps I should start publishing the Pex PEX as pex.pex to help avoid this confusing overload.

Just brainstorming an additional option:

make the pex3 entrypoint able to do everything (e.g. add a new subcommand pex3 build $args that does the same thing as pex $args)
publish a pex3 pex with that console script

If this was supported, I'd personally just use pex3 for everything rather than switching between two styles, and I think issues mentioning pex3 would be unambiguous.

jsirois · 2024-05-24T04:36:44Z

Ah, if you think it works sufficiently well/captures everything it needs to, I'm happy to use that as official guidance.

Yup, should work just fine and give all the same guarantees.

I have a confirmation question: I note that export-subset currently has a warning suggesting a lockfile with multiple locks isn't supported. This would suggest to me it doesn't cover all circumstances, but... maybe those circumstances don't matter in practice?

They don't matter in-practice for Pants, unless things have changed. Against my advice, Pants opted to use universal locks which include just 1 lock; so I embarked on a huge Pex code effort to support those. The case export-subset doesn't support is the style of lock that is easy to create and I recommended from the start, which is 1 lock file with multiple single-platform locks within. It's that case, where there are multiple locks inside the lock file to pick from, that the comment refers to.

See pantsbuild/pants#12458 for some background on picking the popular way of doing things (Poetry-style lock files at the time -> Pex --style universal) vs. the right way.

Perhaps a better entry point / discussion: pantsbuild/pants#12200 or pantsbuild/pants#12568

Basically, there was a bit of a war waged around this and I lost and implemented the most complex, least secure thing - --style universal.

jsirois · 2024-05-24T04:42:12Z

@huonw if you find issues with the export-subset -> pex -r ... setup, let me know. The path you were pushing for here suffers from an impedance mismatch that you'd need to grok to add the feature. Namely, export-subset exports a lock for exactly 1 interpreter. What you would seem to want in the Pants case would be subsetting a universal lock to another (smaller) universal lock. That is a much simpler operation than exporting a subset for a specific interpreter which requires selecting which artifact is needed per locked requirement. To subset a lock to another lock, you take Pip's resolve logic as given-good, and so merely need to walk the dep graph of the input top-level subset requirements in the existing lock, ignoring environment markers and requires-python. As such, I think the implementation would be completely disjoint from export-subset to start. To re-use would require breaking export-subset logic into 2 parts - 1st subset the lock, then pick artifacts.

jsirois · 2024-05-24T04:49:26Z

Ah, sorry. I like the all-in-one pex a lot: I have plopped it into my $PATH and then use it to install random Python tools (unrelated to Pants).

I use a user-local venv for this myself. It's a bit nicer than the Pex PEX IMO since I can change its version easily:

python -mvenv ~/bin/pex.venv
~/bin/pex.venv/bin/pip install pex
ln -s ~/bin/pex.venv/bin/pex ~/bin/pex
ln -s ~/bin/pex.venv/bin/pex3 ~/bin/pex3
ln -s ~/bin/pex.venv/bin/pex-tools ~/bin/pex-tools

And later:

~/bin/pex.venv/bin/pip install -U pex

Or:

~/bin/pex.venv/bin/pip install -U pex==<debug some old version>

I think the Pex PEX is only more convenient for Pants itself, which tries not to know anything about Python, and largely succeeds by being able to download a Pex "binary".

jsirois · 2024-05-24T05:21:01Z

make the pex3 entrypoint able to do everything (e.g. add a new subcommand pex3 build $args that does the same thing as pex $args)

Yup. That has been exactly the plan.

huonw · 2024-06-05T01:05:00Z

Okay, I've had a chance to experiment a bit. Some observations:

pip doesn't allow --hash with VCS requirements, and thus a PEX lockfile that happily contains a VCS requirement and PyPI requirements cannot have a subset that contains both of those be safely installed via requirements.txt: need to drop the --hashs on the PyPI requirements.
somewhat relatedly, a bug pex3 lock export can turn VCS requirements into PyPI ones, resulting in hash failures #2416

That first one seems unresolvable with requirements.txt?

jsirois · 2024-06-05T01:38:48Z

Ah, yeah. That's right. I went through some hoops to support locks of both VCS requirements and local project directories, neither of which Pip's --hash supports.

Ok then. My comment above (#2411 (comment)) applies then. Let me know if you need more guidance or cry uncle.

huonw · 2024-06-05T05:00:09Z

Thanks, the prompt to should avoid following export-subset is very helpful. (Not sure when I'll have a chance to look at it, though.)

jsirois · 2024-06-05T14:33:55Z

Ok, I assigned you to help me keep track not to touch this.

huonw added enhancement feature request labels May 20, 2024

huonw mentioned this issue May 20, 2024

Pants triggering a rebuild even if dev-dependency updated pantsbuild/pants#15694

Open

jsirois assigned huonw Jun 5, 2024

huonw mentioned this issue Jun 5, 2024

Subset lockfile as a separately cacheable process before building any PEX pantsbuild/pants#20994

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow subsetting a PEX lockfile in PEX format, not just pip #2411

Allow subsetting a PEX lockfile in PEX format, not just pip #2411

huonw commented May 20, 2024

huonw commented May 20, 2024

jsirois commented May 24, 2024 •

edited

Loading

jsirois commented May 24, 2024 •

edited

Loading

huonw commented May 24, 2024

jsirois commented May 24, 2024 •

edited

Loading

jsirois commented May 24, 2024 •

edited

Loading

jsirois commented May 24, 2024 •

edited

Loading

jsirois commented May 24, 2024

huonw commented Jun 5, 2024

jsirois commented Jun 5, 2024

huonw commented Jun 5, 2024

jsirois commented Jun 5, 2024

Allow subsetting a PEX lockfile in PEX format, not just pip #2411

Allow subsetting a PEX lockfile in PEX format, not just pip #2411

Comments

huonw commented May 20, 2024

huonw commented May 20, 2024

jsirois commented May 24, 2024 • edited Loading

jsirois commented May 24, 2024 • edited Loading

huonw commented May 24, 2024

jsirois commented May 24, 2024 • edited Loading

jsirois commented May 24, 2024 • edited Loading

jsirois commented May 24, 2024 • edited Loading

jsirois commented May 24, 2024

huonw commented Jun 5, 2024

jsirois commented Jun 5, 2024

huonw commented Jun 5, 2024

jsirois commented Jun 5, 2024

jsirois commented May 24, 2024 •

edited

Loading

jsirois commented May 24, 2024 •

edited

Loading

jsirois commented May 24, 2024 •

edited

Loading

jsirois commented May 24, 2024 •

edited

Loading

jsirois commented May 24, 2024 •

edited

Loading