wip: repokeys via embedded FS

[lzap]

Allows defining GPG repo keys via URI, I do not like using file:// URL because these are in fact read from embedded virtual FS. I do not think it is worth distributing these files together with repo configs, these are pretty much constants unlikely to change too often. Therefore, I came up with mem:// URI prefix. I wanted the prefix to be easy to understand when someone randomly takes a file from disk and want to edit it.

The patch is very small as we already do gpgkey/gpgkeys JSON processing so the necessary code is really tine. It is also full backward compatible - this is important we do not want to break this contract.

For the initial draft, I included CentOS and Red Hat GPG keys and refactored one repo file for each. If you like this approach (let me know) I will work on the rest.

[schutzbot]

This PR changes the images API or behaviour causing integration failures with osbuild-composer. The next update of the images dependency in osbuild-composer will need work to adapt to these changes.

This is simply a notice. It will not block this PR from being merged.

[supakeen]

If we were to do this I'd prefer if the keys are in the same filesystem as where we store the repository configs themselves instead of a separate one, with any path (you can use file:// at that point) relative to that root.

This is because we have --force-repo-dir in image-builder to override the root at which we look for repository configurations and it'd allow users to use this in their own configurations as well :)

---

It would also potentially allow users to refer to GPG keys stored on their system when using --force-repo-dir (e.g: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-rawhide-primary) though perhaps that should be a separate option once we start considering it. Especially if we potentially start using bootstraps by default.

[achilleas-k]

I don't understand what benefit this brings us. Also, the repos.go file which embeds the repository configs is deleted? How do the repository configs get loaded/embedded now?

[achilleas-k]

Also, let's please take a step back and think about things a bit more globally.

• How will this affect overrides?
• Would we rather go back to distributing repo configs downstream so they're easier to manage outside of the compiled binaries?

I think we've had enough experience with both distributing repo configs separately and embedding to be able to make an informed decision about what works better for all the projects that use them. I'd also like to see a better-thought-out approach to unifying our repo config format with the .repo format, or adding them into the distro configs themselves.

But the comments of this PR aren't the place to discuss all that. My point is I don't think the change in this PR provides much (or any) value and only distracts from the bigger picture.

[supakeen]

Also, let's please take a step back and think about things a bit more globally.

* How will this affect overrides?

* Would we rather go back to distributing repo configs downstream so they're easier to manage outside of the compiled binaries?

I think we've had enough experience with both distributing repo configs separately and embedding to be able to make an informed decision about what works better for all the projects that use them. I'd also like to see a better-thought-out approach to unifying our repo config format with the .repo format, or adding them into the distro configs themselves.

But the comments of this PR aren't the place to discuss all that. My point is I don't think the change in this PR provides much (or any) value and only distracts from the bigger picture.

Separately, I think if the notation of repository keys annoys us then that (partially) gets addressed by #2095 :)

[lzap]

I'd also like to see a better-thought-out approach to unifying our repo config format with the .repo format

How does changing the format from JSON to .repo solves the GPG problem, images library need the keys anyway so we would need to have, what is worse than JSON, a slightly different .repo format, or emulating an environment that does not exist (replacing file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-rawhide-primary with path to a different file perhaps).

Is the idea to take repo file from distro and copy it over to ease maintenance? Are we sure this will work? Fedora typically contains mirrorlist stuff that might not work and we would end up editing them anyways. Having a ton of extra (unused) configuration might be also confusing too.

Separately, I think if the notation of repository keys annoys us then that (partially) gets addressed by #2095 :)

I do not know the full history of this and I have not yet added or bumped OS repository config myself, but let me ask this: if you could do this feature from scratch, what format or workflow would you like to be used? I find YAML very appealing and if we had a generic YAML "include" implemented that could be a nice way of doing things, we do similar stuff after all with distrodefs. Heck, maybe we could take similar approach and build a loader.

[achilleas-k]

I'd also like to see a better-thought-out approach to unifying our repo config format with the .repo format

How does changing the format from JSON to .repo solves the GPG problem, images library need the keys anyway so we would need to have, what is worse than JSON, a slightly different .repo format, or emulating an environment that does not exist (replacing file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-rawhide-primary with path to a different file perhaps).

Doesn't need to be actual .repo format, but we could replicate the structure, in json, yaml, or whatever, so that it's simpler to convert from one to the to the other.
The .repo format doesn't solve the GPG problem, but it's a bigger QoL improvement for users who know how to write repo files and might expect their tooling to use the same. It's not a decision I'm making, but it's something to consider. It's a bigger change, but it would be something to coordinate across all the projects and our testing infrastructure which would have an obvious benefit.

But also, more to the point: What GPG problem?
Ok, long multi-line strings in JSON are ugly, but what's the actual problem here? What is breaking? What issues does it cause?

Is the idea to take repo file from distro and copy it over to ease maintenance? Are we sure this will work?

Why wouldn't it? I'm not saying plop the .repo file and see if it works, I'm saying we can actually engineer a solution to this rather than making changes without considering all the implications.

Fedora typically contains mirrorlist stuff that might not work and we would end up editing them anyways. Having a ton of extra (unused) configuration might be also confusing too.

Our repo configs already support mirrorlist. And metalinks. In fact, using .repo would be simpler than what we use now because at the end of the day these configs end up going straight into libdnf through osbuild-depsolve-dnf.

Separately, I think if the notation of repository keys annoys us then that (partially) gets addressed by #2095 :)

I do not know the full history of this and I have not yet added or bumped OS repository config myself, but let me ask this: if you could do this feature from scratch, what format or workflow would you like to be used? I find YAML very appealing and if we had a generic YAML "include" implemented that could be a nice way of doing things, we do similar stuff after all with distrodefs. Heck, maybe we could take similar approach and build a loader.

JSON has the benefit that it doesn't require third party libraries. Our repo configs were originally never really meant to be human-writable. We have tooling that updates them automatically, for example. There is documentation (both upstream and in RHEL) for configuring custom repos, but I'd say that's considered an advanced feature. Making repo configs human-facing is a rather recent change for us, so I get the desire to make it easier to write and edit by hand. But going back to my earlier point about .repo, I think if I was doing it from scratch I would use the .repo format and maybe write my own parser if one doesn't already exist. Because I do find it weird that I have system repository configurations (in /etc/yum.repos.d) and system tools that use repository configurations (osbuild-composer) and the structure of those files is just slightly different (ignoring the file format difference).

[bcl]

I think this would make it more difficult to maintain and override, now you have to change more things when overriding or updating a repo instead of just the repository json file.

[lzap]

I think this would make it more difficult to maintain and override, now you have to change more things when overriding or updating a repo instead of just the repository json file.

I haven't removed the original behavior it was ment for the pre-defined repos. We could even hide this from the user.

Anyways, okay. We are shipping extra 2.2 MB of strings in the binary while it could have been just 73 kB which is driving me crazy but I do not have any better ideas how to solve this and I do not want to start a big refactoring.