Fetch Git LFS files after clone

[plemarquand]

Motivation:

Currently SPM clones repositories with --mirror, creating a bare repository. If the cloned repository uses git lfs, the files in lfs are never pulled and so the package resolution ultimately fails.

Modifications:

This is a naive implementation of LFS support that simply does a lfs fetch on the mirrored repository, without regard for whether there are LFS files to clone or even if the user has git lfs installed. It does, however, allow packages using LFS to resolve correctly.

Issue: #8233

[dschaefer2]

The main question I have is that we do two clones. One into the cache and then another into the scratch dir (.build). Can we git-lfs through them both?

[plemarquand]

Once the files have been fetched in to the bare repo (the cached folder) they're treated as regular files in the subsequent clone in to the .build folder, so we should only need to work with git lfs the one time.

[sbelbin]

I'd appreciate to learn about the state/progress of this PR.

We've been relying on Git LFS for our development cycle, since we rely on storing large binary files that are generated.

This PR is a step forward to address this limitation within Swift Package Manager & progress in the review of this PR is appreciated by myself and I'd suspect by other developers as well.

[plemarquand]

@sbelbin I'm interested in gathering more example usages for git-lfs in the package ecosystem so we can make sure this solution covers all the usecases. Do you have an example package or usecase where lfs would be necessary?

[sbelbin]

@sbelbin I'm interested in gathering more example usages for git-lfs in the package ecosystem so we can make sure this solution covers all the usecases. Do you have an example package or usecase where lfs would be necessary?

A simple use case that I should be able to spin up (hopefull by Thursday - Eastern time zone) is one that contains a folder with prebuilt libraries of various open source libraries such as SuiteSparse, OpenMP, GMP, & MPFR for different Apple platforms on which our ML libraries depend upon.

P.S. I've seen the commits that were done earlier today and the approach that you've taken of making it configurable opt-in choice seems reasonable to me.

[sbelbin]

@plemarquand
You can experiment with this Git repositories.
Swift Package: https://github.com/sbelbin/sample_lfs
Sampler App: https://github.com/sbelbin/sampler

It's Swift Package contains open source libraries that are built externally as to allow our colleagues to incorporate into their Swift Package Manager.

The Swift Package contains C function called compute having a modulemap as to export that C function.

At this moment the sampler Swift App has a local dependency on the sample_lfs but the goal with your modifications to Swift Package Manager is use Git repository URL.