pex3 lock export can turn VCS requirements into PyPI ones, resulting in hash failures

[huonw]

It seems like exporting a lockfile with a VCS requirement can sometimes use a <name>==<version> specifier in the exported requirements.txt, not <name> @ <url> as specified in the lockfile. This results in attempting to install different artifacts, with different hashes, to the ones PEX originally selected & hashed.

pex3 lock create 'pex @ git+https://github.com/pex-tool/[email protected]' -o test.lock --indent=2
pex3 lock export test.lock -o requirements.txt
pip install --no-cache -r requirements.txt

test.lock:

{
  "allow_builds": true,
  "allow_prereleases": false,
  "allow_wheels": true,
  "build_isolation": true,
  "constraints": [],
  "locked_resolves": [
    {
      "locked_requirements": [
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "e6fb2d3eb7893fc5ff61c9e615d7ea8ad2cb9a93ac2b7927b30fe5f1018b2395",
              "url": "git+https://github.com/pex-tool/[email protected]"
            }
          ],
          "project_name": "pex",
          "requires_dists": [
            "subprocess32>=3.2.7; python_version < \"3\" and extra == \"subprocess\""
          ],
          "requires_python": "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,<3.13,>=2.7",
          "version": "2.3.1"
        }
      ],
      "platform_tag": [
        "cp39",
        "cp39",
        "macosx_14_0_arm64"
      ]
    }
  ],
  "only_builds": [],
  "only_wheels": [],
  "path_mappings": {},
  "pex_version": "2.3.1",
  "pip_version": "20.3.4-patched",
  "prefer_older_binary": false,
  "requirements": [
    "pex"
  ],
  "requires_python": [],
  "resolver_version": "pip-legacy-resolver",
  "style": "strict",
  "target_systems": [],
  "transitive": true,
  "use_pep517": null
}

requirements.txt:

pex==2.3.1 \
  --hash=sha256:e6fb2d3eb7893fc5ff61c9e615d7ea8ad2cb9a93ac2b7927b30fe5f1018b2395

Output of pip install:

...
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    pex==2.3.1 from https://files.pythonhosted.org/packages/e7/d0/fbda2a4d41d62d86ce53f5ae4fbaaee8c34070f75bb7ca009090510ae874/pex-2.3.1-py2.py3-none-any.whl (from -r requirements.txt (line 1)):
        Expected sha256 e6fb2d3eb7893fc5ff61c9e615d7ea8ad2cb9a93ac2b7927b30fe5f1018b2395
             Got        64692a5bf6f298403aab930d22f0d836ae4736c5bc820e262e9092fe8c56f830

This example uses the tagged commit of a version, but the same thing applies to using a non-tagged commit, e.g. pex @ git+https://github.com/pex-tool/pex@36dd2374569b5f3fbef54918e6ba3bd1ff852b61 (36dd237)

The same thing applies to pex3 lock export-subset too.

[jsirois]

It's likely the case the only answer here is to fail fast when exporting a lock with either VCS or Local project requirements since the resulting --hashed requirements file will be unusable by any known tool.

[huonw]

I could imagine it's not implausible that someone takes the output of export and munges it to remove the --hashes (opting-in to losing the security advantages of the hashes, of course), and thus making this an error would break them. That might be a use case PEX regards too weird/extreme, though.

[jsirois]

Ok, since I'm confident the current scheme is broken for anyone with VCS or local project requirements (the hash can never match the hash of the pinned artifact currently emitted) I feel safe I don't break anyone using --hash further by switching fro X==Y to the VCS requirement or local project path as appropriate. If there are folks like you hypothezise that munge the file, they'll at least now have the proper list of requirements left. Towards that end, I think I'll add an export format choice to omit hashes. This seems like a real-world reasonablish use case.