Allow use of an OCI Registry instead/alongside of buffrs registry

[DerTiedemann]

Description

The problem the buffrs registry is trying to solve is the structured storage of versioned binary blobs containing .proto files (if i have understood correctly). It seems that the could be used to not have to implement the details of this problem in this project.

Here are some open source registries:

• https://github.com/distribution/distribution
• https://github.com/goharbor/harbor
• https://github.com/mcronce/oci-registry
• https://github.com/project-zot/zot

Reasoning

Currently the registry is being developed as a gateway to publish and allow the consumption versioned .proto files, which are tarballed and gzipped to some form of external file storage. These exact kinds of operations have been standardized in the aforementioned OCI distribution spec and have implementations that scale to a reasonable degree already publicly available. These registries are also offered by 3rd party cloud providers. From my point of view it would be way more attractive to a company adopting buffrs if it was possible to reuse already existing infrastructure + authentication config instead of having to manage another component. The concern of data locality can be addressed by the before mentioned open source registries most of which offer a broad range of storage drivers (s3, gcs, azure, fs, in-memory).
Using an open standard is also beneficial when it comes to 3rd party tooling e.g. which can be used to verify artifact integrity.

TLDR: Rather to build 'Yet another package manger' + another storage solution, use existing storage solutions instead and build the spec on how to interact with it on top of that

Caveats

• Backwards compatibility with the already existing buffrs registry
• OCI is realized via HTTP
• external dependency in a spec and implementation
• too complex for 1.0

Links

• Helm OCI MediaTypes: https://helm.sh/blog/helm-oci-mediatypes/
• OCI Registry as Storage: https://oras.land/docs/ (rust lib started but abandoned)
• Rust lib to interact with OIC: https://github.com/krustlet/oci-distribution

This is more of an Idea than an issue, please let me know what you think and give feedback.

[mara-schulke]

Hi @DerTiedemann, I think this is a very good idea and probably even more interesting to build than continuing the development of the buffrs registry itself in the short/mid-term!

For context: The initial idea behind the buffrs registry was the automated documentation + observability benefits you get from it (which you don't get from generic registries usually) but I do heavily agree with the fact that attractiveness of adoption is more important than any niche features as of right now.

I don't think that I will have the capacity to build this in the short term, would you be open to draft something?

[DerTiedemann]

TBH I'd love to but due to an upcoming exam period, I am quite at capacity atm. I can diagram out a rough idea on what to change where, and what MediaTypes could look like (sth along the lines of metadata, schema, blob, etc). But no ETA on that earliest in 2-3 weeks.

[mara-schulke]

#235 should be implemented prior to this