Design and use a proper "universal" and unique package identifier

[pombredanne]

We need a proper package id that is universal and unique: the difficulty is that each package management technology uses more or less parts in an identifiers beyond basic name+version: Maven GAVs, RPM NEVRA, etc.
A simple solution is to have a single string ID with a prefix that describes what this is about and have a variable number of slash or colon-separated segments URN/URI-style such as used in:

• openshift and fabric8 analytics: Spike around mapping CVEs to individual packages of an ecosystem openshiftio/openshift.io#1052 (comment)
• the new grafeas https://github.com/Grafeas/Grafeas/blob/9d3c7b3a08c74dfa42e91c1ecd428e163f7bfe0a/README.md#resource-urls ( @R2wenD2 is this a convention that is already in use? do you have a longer list of examples? )
• CPEs, (more or less, with several caveats IMHO )
• DejaCode-style URNs

This should probably be defined in ABC Data and with aboutcode-org/aboutcode#6

[R2wenD2]

The naming conventions set forth by Grafeas are at least partially in use by Grafeas partners.
Unfortunately, there isn't currently a longer list.

[pombredanne]

@R2wenD2 Thank you for chiming in! Any pointer to some public or open source reference? or additional details you can share? Any documentation beyond your readme?

[pombredanne]

@mnonnenmacher @sschuberth ping, as this is a topic that is of interest you based on this discussion aboutcode-org/aboutcode#6 (comment)

[R2wenD2]

The best public pointer I can share are jFrog xray component identifiers It has small differences from Grafeas (docker and generic files), but the others match. Does that help?

We're using the identifiers suggested by the package manager where applicable and just prefixing with info about which package manager has specified them. I believe the open question here is about what we use to specify the package manager.

What is currently missing from the Grafeas list that you'd like to see?

[sschuberth]

@pombredanne FYI, the Grafeas spec is at https://github.com/Grafeas/Grafeas

[sschuberth]

@pombredanne Also, right now we use the tuple $packageManager:$namespace:$name:$version as the identifier, where $namespace is "The namespace of the package, for example the group id in Maven or the scope in NPM".

[pombredanne]

@sschuberth this approach works well for Maven and NPMs but may not be work for other package managers/formats/identifiers. Some remarks:

1. namespaces do not exist for Pypi and Rubygems for instance

2. extra identifying elements may be typically needed for Linux distro packages (e.g. rpm and deb) such as distro/os, arch (though release/version/epoch of rpms and similar for debs could be stuffed alright in a version)

3. some extra qualifiers may exists in a few common cases (java vs plain ruby for gems, various combos of os/arch for pypi, type/classifiers/packaging for maven)

So one thing is if we could make everything fit in a schema with a fixed number of segments OR have a variable number of segments depending on the package managers/format/repo technology.

I tend to think the later is more flexible and provide more resilience to changes for the future.

[R2wenD2]

I tend to agree with @pombredanne that the latter is more resilient and preferred.

[sschuberth]

1. namespaces do not exist for Pypi and Rubygems for instance

Correct. We simply use an empty thing in such cases.

2. and 3. could probably be captured by a generic optional extra qualifier field whose meaning depends on the package manager.

[pombredanne]

@R2wenD2 you wrote:

We're using the identifiers suggested by the package manager where applicable and just prefixing with info about which package manager has specified them. I believe the open question here is about what we use to specify the package manager.

Thank you for the xray pointer! Anyone there that you could ping?
I have minor consistencies comments e.g. for instance using gav for things that are from Maven repos and using pip which is the manager for Python pypi packages seem a tad weird as it is either uncommon (gav) or not clear if what is referenced is a packaging format as served by some repo technology (e.g. gav vs. maven) or an installation tool vs. a repo technology (e.g. pip vs. pypi) but this is a great base and it would be easy enough to create bidirectional mappings between similar conventions if needed.

What is currently missing from the Grafeas list that you'd like to see?

At least Rubygems, Composer, Golang and CPAN for a start and many more ;) I can suggest/contribute conventions back to Grafeas FWIW

[pombredanne]

@reiz @andrew out of curiosity, do you use any such thing in versioneye or libraries.io ?
@jpopelka your convention in openshift and fabric8 is "ecosystem:name:version" right? where the ecosystem is more or less the same as the grafeas "package manager" URL?

[jpopelka]

@jpopelka your convention in openshift and fabric8 is "ecosystem:name:version" right? where the ecosystem is more or less the same as the grafeas "package manager" URL?

Yes, our 'ecosystem' is basically a 'package manager'.

[andrew]

@pombredanne we don't have something similar to ecosystem/name/version but that doesn't quite enough to be truly unique it as it ignores architecture (rubygems for example can have different versions available for MRI/JRuby gems) and different registries (Maven central vs jfrog for example, but almost every package manager can point at different registries which may have the same name for a different package.

So really it needs to be something like registry-url/name/version/platform

For our purposes, especially considering Go (which has no registry), we've been treating the fully qualified url to the canonical package page as the unique identifier, although some package managers (like Bower) don't have individual urls for packages, so we just make one up from the registry domain.

[pombredanne]

@andrew Thanks! you wrote:

we've been treating the fully qualified url to the canonical package page as the unique identifier, although some package managers (like Bower) don't have individual urls for packages, so we just make one up from the registry domain

That's a clean and mostly universal approach too! But it does not always convey what a package "format" would be pointed to by a given URL unless this is for well known registries?

@R2wenD2 this brings up a possible ambiguities in the grafeas/xray approach:

• the identifier points unambiguously to a unique file or set of files for your htts://(docker) and file:// schemes
• OR to some package "name/version++" in other cases, and I assumed this would then be from the main public registry/repository for this package "format".

It does not include a notion of which repo/registry this packages lives in and furthermore each package name/version may have more than one "artifact" e.g. an sdist and many wheels for Python or an mri and jruby gem for Ruby, etc.

So in some cases you identify actual exact files or stack of files and in some other cases you identify some pointers to the primary public package repo for this format. This may be OK, but this may be also a source of confusion?

[pombredanne]

@chen-keinan re #805 (comment) I may assume wrongly that you may be involved with Jfrog xrays? or not?

[R2wenD2]

@elad165 Should be able to help here.

[pombredanne]

@elad165 any feedback?

[pombredanne]

So here is my proposal for ScanCode at least and ABC Data in general:
I like the URL/URI approach from Grafeas and xrays a lot. Using plain HTTP URLs as used in libraries.io can work, but it is a tad too generic for my taste.
But using URLs in general is a great thing.

Now about the parts of a package identifier:

1. First is a part to identify what is called ecosystem (openshitf), package_manager (here.com ORT), package_type (ass today in ScanCode) or URL scheme (in the Grafeas or Jfrog/xray URLs). The name used does not matter much, but what this means matters: it captures in a short string a lot of info:

• name and version conventions for id
• package formats, metadata/manifest formats (package.json, a vendor Go dir, a Gemfile, etc)
• build, installation and dependencies resolution convention (e.g. setup.py, RPM specfile, how to document dep constraints, etc)
• protocols to exchange with a remote package registry (registry APIs and protocols, etc)
• package manager(s) that can deal with all these (npm, yarn, pip, mvn, etc)
• a default public registry URL (e.g. implied with maven, npm, gems, etc)

There is no best attribute name to capture all this, but the meaning is clear enough.
I will continue for now to call it package type in ScanCode. 'Kind', 'manager', 'scheme' would all be fair game too. The key point is that we can establish unambiguous mappings between all these 'types'. (e.g. pip and pypi, gav and maven are the same, etc)

Each type has different conventions but all have at least a name and a version (where the version may be a commit) In a very generalized way, we would have something like this from most to least significant:

type namespace name version qualifier

2. After the type, we have some namespace like specified in grafeas/xray for some types or in ORT by @sschuberth and team where the name and version are likely unique (e.g. a scoped npm scope name, the "jessie" debian distro, the /user/ in a github repo, a maven groupid, etc).
What this string means is opaque and specific to a package type. It may be empty. This would be the host in a plain http URL.

3. Next, we have name and version: they are pretty explicit. The exact meaning may vary by type (e.g. sorting conventions to get the last version, etc). Note that for "structured" versions like in RPMs (epoch:version:release) or OSGi/Eclipse (where the version has an optional qualifier) (epoch:version:release) we will treat this whole as a version string. A version may also be a tag or commit'ish or date/time stamp as a last resort.

4. And then we have qualifier(s) (or suffix(es)) that are things like an OS or arch in RPMs or deb, or pypi environment markers, ruby mri vs. jruby packages, maven packaging/qualifier/type, a distro "level" such as el6 or el7 for RPMs or jessie for debian, etc. What this string means is opaque and also specific to a package type.

5. The last important part is the package "location" e.g. the package repository or registry needed to locate this package... The public registry is implied by a type. (e.g. rubygems.org or pypi.python.org or npmjs.org, etc as defaults). I would prefer to keep this entirely separate from the package identifier: we should use instead an extra separate URL attribute to point to an alternate registry, be it public or private beyond the implicit default for a package type. I suggest a registry_url attribute for this.

6. I left aside anything about content-based identifiers (e.g. checksums): this is a separate and solved topic that does not need much discussion IMHO... though I like to be able to identify a plain file that is effectively a package using only a checksum like in xray (with a "generic" type) as a last resort. This would be for a few odd cases anyway.

So in recap the package id is either a string, or discrete fields using this convention: type namespace name version qualifier

• Attributes can be parsed from a string unambiguously given a type.
• Most parts are optional, but you need something to form some id of course.
• These form a natural hierarchy from most to least significant.

When used as a single string, they can form a URL of sorts using '/' , ':' or '@' -separators such as:

• type://namespace/name:version:qualifier
• or this: type:namespace/name:version:qualifier
• or this: type://namespace/name:version@qualifier
• or this: type://namespace/name@version#qualifier
• or this: type:namespace:name:version:qualifier (which is a tad ambiguous)

As long as each segment can be unambiguously parsed back for a given type.
I prefer using the plain colon to using // or /// as a separator but it does not make a big difference so long as it parses. Using an HTTP-like URL structure could have a lot of benefits as there are plenty of URLs parsers available.

Which one to pick?

Some examples:

pypi:///scancode-toolkit would mean the scancode-toolkit package on the public pypi regsitry.
No version is specified, which is OK: this forms a natural hierarchy where data can be stored at higher or lower level nicely, the higher levels providing a default and lower levels can override this with specific data.

And pypi:///scancode-toolkit:2.0.0 would identify the scancode-toolkit package v 2.0.0. It may be a wheel for a certain or a source distro. A qualifier may be needed if you want extra precision about an exact wheel or source distribution.

Some other notes:

• What are the main differences with the Grafeas/Xray approach? Not many, only small details.

Both seem to use things like OS/arch as "classifiers"/prefixes rather than qualifiers/suffixes for distro packages. IMHO this is not something that is higher in the identification hierarchy than the name/version (even though this is a major in locating arch-specific packages for Yum repos), so I prefer to stick with using a qualifier for these, especially since this is an indication of a build of a given version.
It otherwise does not feel right to force to have an architecture identifier. But in any case the conversion from one to the other is straightforward.

• For Docker, Grafeas use a plain https URL which squats the namespace segment (e.g. gcr.io): squatting the super common https scheme as a package type is not great IMHO. I prefer the xray clear docker type.

Also using @ as a separator for version/image/id is a variant to consider everywhere rather than making this an exception? e.g use @ as a version separator all the way like here in SPDX I specified using @ to separate a tag/commit (using the same convention as in pip VCS urls) https://github.com/spdx/spdx-spec/blob/cfa1b9d08903befdf03e669da6472707b7b60cb9/chapters/3-package-information.md#37-package-download-location- ?

• in Grafeas/xray I find using a gav type for maven a tad cryptic, but not a big deal as long as they map to each other OK.

[pombredanne]

And here is a short summary: I propose we use five parts/fields to identify a package:

• type: such as maven, npm, gem, pypi
• namespace: such as a maven groupid, a debian distro
• name: the name
• version: the version
• qualifier: extra things beyond the version such as OS, architecture, src or doc, eg. that qualifies specific details for a given version

Most are optional and can also be composed in some URL as in:
maven://org.apache.commons/io:1.2.3 e.g. type://namespace/name:version

The exact format to use for his URL is not fully specified yet.

And we add an extra field to point to an alternative package registry_url such as a private NPM repo.

[pombredanne]

Here is a revised design:

A package identifier is defined by six parts or fields that form a hierarchy, from the least specific to the most specific identifying information:

• type: such as maven, npm, gem, pypi. required
• namespace: such as a maven groupid, a debian distro. optional
• name: the name. required
• version: the version. optional
• qualifiers: extra qualifying data for a package such as OS, architecture, src or doc. optional
• path: optional path within a package. optional

At the minimum a type and a name are required.
Other parts are optional.

A package identifier is either discrete fields or a URL string using these conventions:

type:namespace/name@version?qualifiers#path

For instance:

• maven:org.apache.xmlgraphics/[email protected]?packaging=sources --> the source jar
• go:google.golang.org/genproto#googleapis/api/annotations --> a path inside a Go package repo
• rpm:fedora-25/[email protected]?arch=src --> the src RPM
• rpm:fedora-25/[email protected]?arch=i386 --> a i386 build
• docker:scanning-customer/dockerimage@sha256:244fd47e07d1004f0aed9c156aa09083c82bf8944eceb67c946ff7430510a77b --> a Docker image with a specific id as version
• docker:cassandra@cassandra --> a Docker image with a specific tag as version
• pypi:[email protected]
• gem:[email protected]
• gem:jruby-launcher/versions/1.1.2?platform=java --> a gem for JRuby
• npm:%40angular/[email protected] --> a scoped node package

The string would UTF-8 encoded, with percent-encoding were needed https://en.wikipedia.org/wiki/Percent-encoding with these rules for each part:

• type: composed only of ASCII letters and numbers, + and - and _.
• namespace: contains zero or more segments, separated by a slash. Each segment must be a percent-encoded string.
• name: must be a percent-encoded string.
• version: must be a percent-encoded string.
• qualifiers: must be a percent-encoded string. The content structure is defined by the type. IMHO the best would be a HTTP URL query string with name/value pairs.
• path: contains zero or more segments, separated by a slash. Each segment must be a percent-encoded string.

The parsing approach would be:

1. split package identifier string once from right on #, right side is the path after percent-decoding.
2. split left side from 1. once from right on ?, right side is the qualifiers after percent-decoding (and eventual parsing of the query string name/value pairs.)
3. split left side from 2. once from right on /, right side is the name after percent-decoding.
4. split left side from 3. once from right on :, left side is the type, right side is the namespace after percent-decoding.
   .... which (in Python) would mean using the str.rpartition() function for the splits.

And to get an exact download, we either provide an optional registry base url if this is not on the standard public registry and/or an optional full direct download URL.

I cannot think of any case I know of that would not work with this approach.
After all this is a URL which is a tried and true way to create identifiers and locators ;)

[pombredanne]

ok, I did not receive much pushback on the details, so I am assuming this is a good thing!
@JonoYang suggested a great name for these: puurl standing for Package "mostly" Universal URL.
The only cosmetic thing is whether to use type:// or type: for a "canonical" form. The // is not significant in any case here.
e.g. this:
maven:org.apache.commons/[email protected]
or this:
maven://org.apache.commons/[email protected]

Aesthetically, the // looks much better I guess!

I have a Python implementation in the #275 branch here:
https://github.com/nexB/scancode-toolkit/blob/275-streamline-package-manifests-models/src/packagedcode/models.py#L169
In particular the creation and parsing is straightforward.

It should be trivial to have a Go or Ruby or JS implementation

@R2wenD2 I would like to also contribute this spec to Grafeas FWIW.

[R2wenD2]

Can you add a proposal issue to Grafeas? Feel free to copy your design proposal above - I just want to make sure folks interested in Grafeas have a chance to review.
One small note - Grafeas can't support docker images by tag reference because tags are mutable.

[pombredanne]

Package URL is now implemented alright in develop and works well. Next step is to call the prul spec as a 1.0

[pombredanne]

I am closing this now. The Package URL lives its own life now at https://github.com/package-url ... and is heavily used in ScanCode and other places.
Thanks you all for the contributions and feeback!