add lint.sh and check in CI

[imjasonh]

This adds a simple lint.sh script that processes files with yq to:

• remove any .environment.contents.repositories and .environment.contents.keyring from Melange configs -- these packages should be explicitly pulled into the local repository and --repository-appended when Melange is invoked
• consistently format YAML files, especially whitespace

Unfortunately this strips empty lines, which might be less than optimal for readability (and the size of this diff). If we want to handle this better, we can.

This adds a GitHub CI check that runs ./lint.sh and fails if it resulted in any diffs.

We can extend this to perform more checks if we want them. I'd like to make sure the Makefile contains a line for every package-version-epoch.apk, for example. More ideas welcome.

Signed-off-by: Jason Hall [email protected]

[imjasonh]

This is sort of unfortunate 😒

[kaniini]

Can we not? :)

[imjasonh]

Yeah, let's not.

The easiest way will be to just have it be a single line, which is unambiguously formatted. Let me know if you'd like me to figure out how to get the newlines preserved instead.

[kaniini]

I would prefer to keep the newlines as they are.

[imjasonh]

I would also prefer that. Unfortunately I can't figure out how to get yq to preserve those. Short of writing our own yamlfmt or adding support for this to yq, I think it's a worthy sacrifice if it means we get linting and bug catching in place.

[kaniini]

remove any .environment.contents.repositories and .environment.contents.keyring from Melange configs -- these packages should be explicitly pulled into the local repository and --repository-appended when Melange is invoked

This should only be removing the https://packages.wolfi.dev/os repository, right? We do need to specify the stage3 repository for the early packages where we are still lifting ourselves out of bootstrap world.

[imjasonh]

remove any .environment.contents.repositories and .environment.contents.keyring from Melange configs -- these packages should be explicitly pulled into the local repository and --repository-appended when Melange is invoked

This should only be removing the https://packages.wolfi.dev/os repository, right? We do need to specify the stage3 repository for the early packages where we are still lifting ourselves out of bootstrap world.

Yeah that makes sense. I had thought we were past the bootstrapping phase enough that we didn't need these anymore, but if there are things we still need from there then we should keep them. I've updated the script to only remove packages.wolfi.dev/os.

Is there a list of things we need to do to fully exit bootstrapping? It will be easier to tell our build is hermetic when we can show we don't have any external repos, including bootstrapping.

[kaniini]

Is there a list of things we need to do to fully exit bootstrapping? It will be easier to tell our build is hermetic when we can show we don't have any external repos, including bootstrapping.

In practice, for x86_64, we have exited bootstrapping, but we want to keep the distribution bootstrappable. This is helpful for adding additional archs, and also allowing others to reproduce the entire distribution from scratch.

We can probably have a setting via the melange templating system to ignore the repos when not in bootstrap mode?

[imjasonh]

In practice, for x86_64, we have exited bootstrapping, but we want to keep the distribution bootstrappable. This is helpful for adding additional archs, and also allowing others to reproduce the entire distribution from scratch.

We can probably have a setting via the melange templating system to ignore the repos when not in bootstrap mode?

Could we make the configs bootstrappable again by adding MELANGE_EXTRA_OPTS=--repository-append=https://packages.wolfi.dev/bootstrap/stage3?

If these are left as markers for where bootstrapping comes into play then that's great and we can leave it.

[kaniini]

My intent is to leave it as a marker, but also I don’t want to append the repository to every build, just the bootstrapped ones.

[imjasonh]

lol #80 means glibc.yaml isn't YAML anymore, so yq can't parse or format it.

Going back to the drawing board folks.