Add purls (Package URLs) to PackageRecord
#63
Conversation
|
Awesome CEP! :) |
|
|
||
| ## Abstract | ||
|
|
||
| This CEP describes a change to the `PackageRecord` format and the corresponding `repodata.json` file to include `purls` (Package Urls) of repackaged packages to identify packages across multiple ecosystems. |
There was a problem hiding this comment.
Can you add a link to the definition of a PackageRecord? I struggle to find an authoritative source for it.
There was a problem hiding this comment.
Unfortunately, I believe that atm there is no actual "authorative" source.
There is this relatively old definition of a RepoDataRecord: https://github.com/conda/schemas/blob/main/repodata-record-1.schema.json
There is this new effort to document the schemas better (conda/schemas#26) where it's also called RepoDataRecord: https://github.com/conda/schemas/blob/b143c82a71833570fbe9be2313368b33c0e84726/conda_models/package_record.py#L23
And we have the definition in rattler: https://docs.rs/rattler_conda_types/latest/rattler_conda_types/struct.PackageRecord.html
In rattler (and I believe in conda as well), there is this distinction:
PackageRecord: contains all the fields for a single entry in therepodata.jsonRepoDataRecord: inherits all fields fromPackageRecordand adds fields to identify the origin of the data (channel, url, etc.)PrefixRecord: inherits all fields fromRepoDataRecordand additionally stores information about how the package was installed.
There was a problem hiding this comment.
Yea, I think the most "official" source for this is https://github.com/conda/conda/blob/e783377439ed1c413c6bffb9b785ae1d79c2392a/conda/models/records.py#L247. That module also offers some sort of definition in the top-level docstring.
Implementation of conda/ceps#63
This PR adds support for checking the satisfiability of the lock-file which includes pypi-dependencies. Purls have been added to the lock-file (conda/rattler#414) (See also: conda/ceps#63). This enables checking which conda packages will install which pypi packages without needing to check the internet. This ensures we can still check if a lock-file is up to date quickly. I did not profile this code but I think there are a lot of places we can improve the performance. Thats for a later PR. I also didn't add tests. I think we should but we can also do that in another PR. Closes #467 --------- Co-authored-by: Ruben Arts <[email protected]>
| } | ||
| ``` | ||
|
|
||
| PURL is already supported by dependency-related tooling like SPDX (see [External Repository Identifiers in the SPDX 2.3 spec](https://spdx.github.io/spdx-spec/v2.3/external-repository-identifiers/#f35-purl)), the [Open Source Vulnerability format](https://ossf.github.io/osv-schema/#affectedpackage-field), and the [Sonatype OSS Index](https://ossindex.sonatype.org/doc/coordinates); not having to wait years before support in such tooling arrives is valuable. |
There was a problem hiding this comment.
I would also mention PEP-725 (WIP).
There was a problem hiding this comment.
The Discourse thread has examples showing how the Spack community wants to use this kind of thing: https://discuss.python.org/t/pep-725-specifying-external-dependencies-in-pyproject-toml/31888/31
| * We can keep this information close to the conda package description. | ||
| * We can incrementally add `purls` through repodata patches. | ||
|
|
||
| The downside is that the (already large) repodata.json file will grow. |
There was a problem hiding this comment.
What if we add a separate-yet-adjacent purls.json like we did with run_exports.json in CEP-12?
There was a problem hiding this comment.
I like the idea and I will be supportive. Havin this metadata readily available would allow us to be listed in repology.org, for example! It would also play nicely with the (draft) PEP-725 for external metadata in PyPI.
However, I think this CEP right now is talking about serving metadata before we have discussed how to source it, define it and store it.
Whatever ends up in the repodata.json comes, in part, from the info/index.json metadata inside the conda artifact. Then this is augmented with things like sha256 and final size by conda-index (because they cannot be known when the package is being archived).
So before we speak about repodata, we should discuss where in the inner artifact metadata we will store the PURL info. To answer that, we must answer where in the conda-build recipe we will include that information :D
IOW, I'd like to know your thoughts about:
- Where in the current
meta.yamlwe should define the PURLs.aboutseems to be the most obvious one, which means this will probably end up ininfo/about.json. - Whether to serve the PURLs separately in a
purls.jsonor not. I honestly don't think putting it inrepodata.jsonis a good idea. I get that it makes sense if you want to have a canonical link between PyPI in conda-forge so Pixi can solve things nicely. It might also be served inchanneldata.json(since most of the time PURLs are tied to the source not the platform-dependent, target artifact).
|
Would this also help us address Repology's needs for supporting Conda packages ( repology/repology-updater#518 )? Edit: Nvm missed Jaime has the same idea |
This CEP describes a change to the
PackageRecordformat and the correspondingrepodata.jsonfile to includepurls(Package URLs of repackaged packages to identify packages across multiple ecosystems.rendered