Support local requirements with lockfiles

[Eric-Arellano]

From John re how to implement:

I think the same trick used for VCS basically works for local projects, but there is some juggling to do (The original standard Pex resolver does the juggling but both the --lock resolver and --pex-repository resolver refuse to do this.

Two Pants users have weighed in that this would be very useful for their repos, so Pants would love to see this feature.

[jsirois]

Sounds good. IIUC @Eric-Arellano to be able to use this feature Pants will need to get out of the way re requirements parsing, or support local absolute paths in requirements in some ergonomic way. For the former, Pex supports pex . and for the latter pex "pex @ file:///home/jsirois/dev/pantsbuild/pex".

[jsirois]

Ok:

$ python -mpex.cli lock create . --indent 2
{
  "allow_builds": true,
  "allow_prereleases": false,
  "allow_wheels": true,
  "build_isolation": true,
  "constraints": [],
  "locked_resolves": [
    {
      "locked_requirements": [
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "0308b8310a8ac7ecbf4be61e3ac3ad6deac818a522ec76aff551b3be9c634910",
              "url": "file:///home/jsirois/dev/pantsbuild/jsirois-pex"
            }
          ],
          "project_name": "pex",
          "requires_dists": [
            "subprocess32>=3.2.7; extra == \"subprocess\" and python_version < \"3\""
          ],
          "requires_python": "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,<3.11,>=2.7",
          "version": "2.1.77"
        }
      ],
      "platform_tag": [
        "cp310",
        "cp310",
        "manylinux_2_35_x86_64"
      ]
    }
  ],
  "pex_version": "2.1.77",
  "prefer_older_binary": false,
  "requirements": [
    "."
  ],
  "requires_python": [],
  "resolver_version": "pip-legacy-resolver",
  "style": "strict",
  "transitive": true,
  "use_pep517": null
}

This runs into the same issues as #1413 though. Time to think about that.

[Eric-Arellano]

Exciting! If it is amenable to land that, I think it could be useful as a precursor. While no one will be able to use it in production lock files without 1413, it can be helpful when people are debugging locally

[jsirois]

It's not, that's just a lock file, its not consuming it. There is more work to do here to support consumption. The "." in requirements is not a valid PEP requirement, and, on parse it currently needs to be. I generally refuse to half-way things because there is often a very steep cost in self-thrash as fallout. Going to get this right.

[jsirois]

A sticky wicket here turned out to be fingerprinting. It's effectively impossible to know what to exclude when hashing a project directory. I took a stab at just omitting pyc related files and dirs and all conceivable VCS dirs, but that's not enough. There are control dirs from tox and mypy and who knows what else that should not affect the project fingerprint when the project source hasn't changed, but do. Since pip download ... does not create an sdist for us (It just generates the sdist metadata we gather in the form of a project name and version from its logs in the Locker.), it seems I'll have to roll an sdist builder. With that in place the hash of a local project can be the hash of its built sdist (Probably the hash of the sdist cracked back open so we don't need to rely on the build backend creating a reproducible sdist / honoring SOURCE_DATE_EPOCH.). This is not too bad to do since we can fall back on setuptools.build_meta:__legacy__ when there is no pyproject.toml with a build backend defined.