Add package registry implementation

[mattt]

Related to #2967.

This PR implements the client-side integration necessary for Swift Package Manager to fetch and resolve dependencies using a package registry.

A bunch of unit tests are currently failing because this integration necessitates a semantic change in how packages are identified. Whereas previously, a package was identified by name, this PR introduces a new PackageIdentity type with a more comprehensive notion of what it means for two packages to be the same.

Instructions

Edit: Check out the Swift registry benchmark harness, which automates a lot of this setup.

---

Download the project


Clone the swift-package-manager project and check out the package-registry-implementation branch.

$ git clone https://github.com/apple/swift-package-manager.git
$ cd swift-package-manager
$ git checkout -t origin/package-registry-implementation

# …or with https://github.com/cli/cli
$ gh repo clone apple/swift-package-manager
$ cd swift-package-manager
$ gh checkout 3023

Create spm shim

To try out the local development build of Swift Package Manager in your other project, you can create a shim:

$ cd path/to/swift-package-manager
$ swift build
$ cat <<EOF >>/usr/local/bin/spm
#!/usr/bin/env bash

set -eo pipefail

if [ -z \$1 ]; then
    echo "missing subcommand [build|run]"
    exit 1
fi


BUILD_PATH="$(swift build --show-bin-path)"
case \$1 in
build) \$BUILD_PATH/swift-build "\${@:2}" ;;
test) \$BUILD_PATH/swift-test "\${@:2}" ;;
run) \$BUILD_PATH/swift-run "\${@:2}" ;;
esac
EOF

$ chmod +x /usr/local/bin/spm

The example above supports the build, test, and run commands, but it can be amended to support others.

To uninstall, run $ rm -f /usr/local/bin/spm

You can now invoke spm as you would swift package.

$ cd path/to/LinkedList
$ spm build --enable-package-registry

Create a package registry

Use the reference implementation to serve a package registry locally. Follow the instructions provided in the README under “Command Line Interface” to publish releases of packages that your project uses.

Generate a local HTTPS certificate

Because the package registry requires all connections to use https, you’ll need to serve the package registry with a locally-trusted development certificate. My go-to option for this is mkcert.

Alternatively, you can use ngrok to tunnel local traffic from a public URL over HTTPS.

Mirror your package dependencies to your local

Once your registry is running an accessible from a URL with https, add a mirror for any external dependencies that you’d like to serve over the registry.

For example, if you’re running a registry locally at https://localhost:8080 and have a package with a dependency on https://github.com/apple/swift-argument-parser, you would add a mirror with the following command:

$ swift package config set-mirror \
	--original-url https://github.com/apple/swift-argument-parser \
	--mirror-url https://localhost:8080/github.com/apple/swift-argument-parser

Build your project

Finally, use spm to build your project, cleaning your project beforehand to ensure that dependencies are fetched.

$ swift package clean
$ spm build

[tomerd]

we should probably turn manifest(for version: Version) into an async function, it makes sense in all cases really (does not need to be in this PR)

[tomerd]

the http client allows you to configure "allowed response codes" which does this check for you

[mattt]

I saw that, but I didn't see a way to conveniently configure that on a per-request basis as would be needed here.

[tomerd]

https://github.com/apple/swift-package-manager/blob/main/Tests/BasicsTests/HTTPClientTests.swift#L276

[tomerd]

move to Basics?

[tomerd]

this feels like another method that should become async. not for this PR tho

[mattt]

@tomerd Thanks for reviewing these latest changes. I'm working through your feedback now.

In related news, I'm excited to share some initial benchmarking results for the package registry. I published a harness here that automates (or at least formalizes) many of the steps in described above to get everything set up. Check it out! https://github.com/mattt/swift-registry-benchmark-harness (/cc @neonichu @yim-lee @abertelrud)

Preliminary results show comparable performance to cloning repositories for clean builds:

$ bundle exec rake benchmark
time ./spm run
       57.70 real        87.12 user        10.88 sys
time ./spm run --enable-package-registry
       16.24 real        35.85 user         4.75 sys

(The numbers look a lot better after 74625e4; I should have a PR to TSCBasic for this sometime next week)

[tomerd]

I think we should do this entire operation on the queue (untested code):

queue.async {
  RegistryPackageContainer.create(
                         for: identifier,
                         mirrors: self.mirrors,
                         registryManager: registryManager,
                         manifestLoader: self.manifestLoader,
                         toolsVersionLoader: self.toolsVersionLoader,
                         currentToolsVersion: self.currentToolsVersion
                     ) { result in
                         switch result {
                         case .success(let container):
                             self.containerCache[identifier] = container
                             completion(.success(container))
                         case .failure(let error):
                             completion(.failure(error))
                         }
                     }
}

[tomerd]

alternatively, if RegistryPackageContainer.create is a async, pass the queue to it and then we can simplify further

[tomerd]

we would want to pass in a diagnosticsEngine here, so maybe clientFactory should take one, etc

[mattt]

Added with 15c90e0. I started add diagnostics engine parameters to getContainer, but I quickly got outside my comfort zone for this PR. Something to consider in a follow-up /cc @abertelrud

[tomerd]

this is confusing to me.. are we using the package identity as the URL? what am I missing?

[mattt]

Yes, that's correct. This is discussed in § 4.1.1 Content Negotiation.

The new (non-legacy) package identity is a canonicalized, scheme-agnostic URL. The process of "discovering" registry support involves sending a HEAD to that URL with the right headers to see if the server redirects us to a package registry API endpoint.

[mattt]

As of 5e91c76 (latest force-push), package releases downloaded from a registry are sent to their own, dedicated cache directory. For now, this is called downloads, but I'd love to hear any better suggestions for what this should be called.

[mattt]

Closing in favor of a new PR that implements a future revision to SE-0292. See https://forums.swift.org/t/urls-as-swift-package-identifiers/43404/128