System Ready Minimal Images fixes

[nullr0ute]

These are some fixes for the minimal image that are needed to fix up a few minor issues.

Signed-off-by: Peter Robinson pbrobinson@gmail.com

[nullr0ute]

@sarmahaj FYI

[runcom]

Lgtm

[thozza]

I think that the PR is missing changes to https://github.com/osbuild/images/blob/81ab54167675ff8f7f319be23c1a4e386c80a7f7/pkg/distro/rhel8/package_sets.go#L77-L83 and nothing in the commits suggests otherwise

Also the Fedora minimal-raw does not reflect the changes to kernel command line arguments done in RHEL -> https://github.com/osbuild/images/blob/81ab54167675ff8f7f319be23c1a4e386c80a7f7/pkg/distro/fedora/distro.go#L370-L388
https://github.com/osbuild/images/blob/81ab54167675ff8f7f319be23c1a4e386c80a7f7/pkg/distro/fedora/distro.go#L35

[miabbott]

Do these changes prevent the minimal-raw artifacts from being used on devices or are they nice to have?

Given that we are past dev-freeze for RHEL 9.3, I don't think we should be making changes to the minimal-raw type unless they are absolutely necessary.

[blomquisg]

I've asked around to figure out what is controversial about this change, and the main thing I've heard is the inclusion of initial-setup.

The workflow as I understand it is to use Image Builder to produce a bootable image from these minimal-raw images.

It sounds like the compromise here is to remove initial-setup from this PR, then document two available paths for users:

1. Use the minimal-raw image and a blueprint with Image Builder to create an initial user account, or
2. Add initial-setup back using Image Builder.

If users are looking for an experience that's similar to building other images with Image Builder, they would choose option 1. If users are looking to test out a simple bootable image and don't mind manually configuring the installation, they could choose option 2.

Thoughts?

/cc @nullr0ute @thozza @runcom @miabbott @mrguitar

[nullr0ute]

I've asked around to figure out what is controversial about this change, and the main thing I've heard is the inclusion of initial-setup.

The workflow as I understand it is to use Image Builder to produce a bootable image from these minimal-raw images.

That's the first step of the whole process, the next step is to provision it onto the device.

It sounds like the compromise here is to remove initial-setup from this PR, then document two available paths for users:

I did not take that conclusion away from yesterday's meeting. The decision from what I understand it was "what ever we do we can't change it once it's shipped". Also they're not mutually exclusive.

1. Use the minimal-raw image and a blueprint with Image Builder to create an initial user account, or

This will cause potential security implications. We don't ship default credentials because of that.

2. Add `initial-setup` back using Image Builder.

If users are looking for an experience that's similar to building other images with Image Builder, they would choose option 1. If users are looking to test out a simple bootable image and don't mind manually configuring the installation, they could choose option 2.

That depends on the usecase. In the case of 1) that may be useful for a single user, but it falls over if it's being provided to end users, partners or other cases.

[nullr0ute]

I have found more fixes needed so I'm going to update this PR with more commits soon once I've worked it all out. New update soon.

[nullr0ute]

I think that the PR is missing changes to
and nothing in the commits suggests otherwise

You are correct, but TBH this was never meant to land for el8 as these devices will only ever be supported on el9, but I will fix now it's there, I thought the package set was the same across both, I missed the 9 in the name.

Also the Fedora minimal-raw does not reflect the changes to kernel command line arguments done in RHEL ->

And I missed that variable, I have other fixes coming in too, will update.

[miabbott]

I did not take that conclusion away from yesterday's meeting. The decision from what I understand it was "what ever we do we can't change it once it's shipped". Also they're not mutually exclusive.

Correct; we want to understand what (if any) changes are required in order for users/customers to be able to boot devices using this artifact type.

That depends on the usecase. In the case of 1) that may be useful for a single user, but it falls over if it's being provided to end users, partners or other cases.

I'd like to understand what would be the more common use case for this artifact.

Even if it is the latter use case, including initial-setup by default would cause this artifact to be the unicorn among all the other artifacts produced by osbuild. Maybe that is the decision we come to, but it does feel anti-pattern compared to how most users will handle user provisioning/configuration when using osbuild.

The other consideration when adding packages to the default set is that end-users have no means of removing those default packages as part of the osbuild compose process, so we need to be very certain that any defaults are going to be applicable to the majority of users.

Regardless of the decisions made about this PR, we will need strong documentation explaining the outcome and how user provisioning/configuration will be done with this artifact.

[nullr0ute]

Even if it is the latter use case, including initial-setup by default would cause this artifact to be the unicorn among all the other artifacts produced by osbuild. Maybe that is the decision we come to, but it does feel anti-pattern compared to how most users will handle user provisioning/configuration when using osbuild.

For osbuild maybe, but from an install experience on devices (AKA phase two in this process) the user generally does have to deal with users and root passwords in the traditional installer (anaconda) use case, whether that be specifying it in the UX doing a "CD style installer (via traditional media or USB stick)" or as an option in the kickstart so for the "install on device" part it is not differing from the standard user experience. If say Partner Management provided a image to a partner to put on a device the partner would never deal with osbuild so from their perspective it would not be different because with anaconda you have to specify those pieces. In fact if you do that install over a serial or text console it's that exact same UX that is shown to the user.

[nullr0ute]

Documentation for RHEL-8 here: https://access.redhat.com/articles/4323781

[miabbott]

For osbuild maybe, but from an install experience on devices (AKA phase two in this process) the user generally does have to deal with users and root passwords in the traditional installer (anaconda) use case, whether that be specifying it in the UX doing a "CD style installer (via traditional media or USB stick)" or as an option in the kickstart so for the "install on device" part it is not differing from the standard user experience.

If the user composed an image using osbuild and the blueprint was missing any user customizations, then yes they would have to handle it during the install phase. But that can be avoided if the blueprint customization is used.

If say Partner Management provided a image to a partner to put on a device the partner would never deal with osbuild so from their perspective it would not be different because with anaconda you have to specify those pieces. In fact if you do that install over a serial or text console it's that exact same UX that is shown to the user.

I think this use case is where I would dig into...is the most common use case for minimal-raw going to be something like this? What happens if someone provides user customization in a blueprint for a minimal-raw image? Will initial-setup still run and have to be dealt with?

[mrguitar]

Documentation for RHEL-8 here: https://access.redhat.com/articles/4323781

That kbase describes how to answer the questions in the TUI. That part is fine and self explanatory. It's more about making the call to force this on users by default, when blueprints make it crazy trivial to add it when you need it. We don't have any product plans to distribute these images so I don't know why we care if customers embed users/credentials. I don't particularly like it, but it's super common (like a million other bad security practices that user do and we don't prohibit). Any images we make for hardware partners can easily include this in a blueprint (which we have to make anyway to include the HTS packages).

I suspect there's a unit file we can disable/mask to stop this from running but I haven't cracked open the rpm to look at it yet. .....I still think it's a bad experience to surprise users with a bespoke experience here and expect them to figure out how to skip initial-setup when they want something more streamlined. I think we'd all agree that a manual process like this doesn't scale for any real deployments. I can see this being a useful image type for embedded style systems ....but not if they need hands-on-keys to complete a first boot. Why push for that? That feels useless and very limiting to me.

[nullr0ute]

For osbuild maybe, but from an install experience on devices (AKA phase two in this process) the user generally does have to deal with users and root passwords in the traditional installer (anaconda) use case, whether that be specifying it in the UX doing a "CD style installer (via traditional media or USB stick)" or as an option in the kickstart so for the "install on device" part it is not differing from the standard user experience.

If the user composed an image using osbuild and the blueprint was missing any user customizations, then yes they would have to handle it during the install phase. But that can be avoided if the blueprint customization is used.

The question then becomes how many people will take the defaults vs customising it? I suspect the answer to that is impossible to predict BUT we do need to provide SANE defaults, if a user does nothing in the blueprint as it stands now they will have an image that they have no way to login to, that is confusing and I am absolutely against providing a default username and password (as I'm sure product security will be as well), on the other side of that in Fedora we have had no issues with users understanding how to use initial-setup.

If say Partner Management provided a image to a partner to put on a device the partner would never deal with osbuild so from their perspective it would not be different because with anaconda you have to specify those pieces. In fact if you do that install over a serial or text console it's that exact same UX that is shown to the user.

I think this use case is where I would dig into...is the most common use case for minimal-raw going to be something like this? What happens if someone provides user customization in a blueprint for a minimal-raw image? Will initial-setup still run and have to be dealt with?

Well the intention of the minimal-raw was intended for basic development, board bring up etc, which would generally be a single user..... BUT like all these things predicting what is intended vs how customers end up using things is impossible to predict.

[nullr0ute]

Documentation for RHEL-8 here: https://access.redhat.com/articles/4323781

That kbase describes how to answer the questions in the TUI. That part is fine and self explanatory. It's more about making the call to force this on users by default, when blueprints make it crazy trivial to add it when you need it. We don't have any product plans to distribute these images so I don't know why we care if customers embed users/credentials. I

What happens if they don't specify a username/password in the blueprint? They can't login. In the case with initial-setup they should be able to just exit and login, or just ssh in straight up if they're doing it remotely. Also initial-setup may just exit if a user or root password is set (I need to verify this, because it's nuanced and I don't remember exact details, it may just not make them create it and they can just continue).

[mrguitar]

What happens if they don't specify a username/password in the blueprint? They can't login. In the case with initial-setup they should be able to just exit and login, or just ssh in straight up if they're doing it remotely. Also initial-setup may just exit if a user or root password is set (I need to verify this, because it's nuanced and I don't remember exact details, it may just not make them create it and they can just continue).

Honestly, neither option sounds great. :(

I cracked the package open and I believe this will skip the "wizard" in a blueprint (or something similar).

[customizations]

[[customizations.user]]
name = "core"
description = "core"
password = "hash"
key = "ssh-rsa AAAAB3NzaC"
groups = ["wheel"]

[customizations.services]
disabled = ["initial-setup", "initial-setup-reconfigure"]

That's probably overkill because there's a ConditionPathExists=/.unconfigured in the initial-setup-reconfigure unit file. I don't see that file as part of the rpm, so I'm not sure if a post script creates it. My guess is the firstboot --reconfigure (from memory so could be wrong) kickstart options tickles this. I know we can create files in the blueprint, but I don't know if we can remove or "untouch" a file like this. A drop-in feels more complicated and a worse customer experience to override. I think knowing the easiest/best way to bypass initial-setup would really help us weight the pros & cons here.

If I can dream a bit, an even better experience would be keeping the defaults @nullr0ute provided and empowering users to override/remove the base packages via a blueprint (or other method). That way we get away from hard coding things and backing ourselves (and customers) into corners. This causes so much pain right now particularly with packages like podman, when users don't need/want containers. We can still provide support because we can point to an image override if something doesn't work as expected.

...Just thinking out loud here, and unfortunately we don't live in this world w/ 9.3 around the corner.

[blomquisg]

I'm mostly stuck on various types of users for the minimal-raw image. Looking back over the comments, it seems to be a mostly open question, honestly. We have some ideas of some users, but I don't see anything concrete.

That said, if we have a definite way to bypass the initial-setup tui, and we can document that somewhere, then I guess it's less of a big deal. But, that needs to be included in testing this before we release it. If including initial-setup necessarily drops the user into an interactive tui, that feels like a miss.

Honestly, since we'll have to document the user interaction either way (whether we have to show them how to skip initial-setup with a blueprint, or how the user would add initial-setup back in via Image Builder), I would lean towards including fewer packages, and document how to build an image that includes what they need. But, this goes back to the understanding who we think will use the minimal-raw, and how.

[nullr0ute]

I think that the PR is missing changes to

I think these are now all fixed, along with a bunch of other changes :)

[thozza]

That's probably overkill because there's a ConditionPathExists=/.unconfigured in the initial-setup-reconfigure unit file. I don't see that file as part of the rpm, so I'm not sure if a post script creates it. My guess is the firstboot --reconfigure (from memory so could be wrong) kickstart options tickles this. I know we can create files in the blueprint, but I don't know if we can remove or "untouch" a file like this. A drop-in feels more complicated and a worse customer experience to override. I think knowing the easiest/best way to bypass initial-setup would really help us weight the pros & cons here.

No, there is no option to delete files from the image.

From what I can tell, disabling the initial-setup.service unit looks like the most reasonable way to disable initial-setup from running in the image on the first boot.

If I can dream a bit, an even better experience would be keeping the defaults @nullr0ute provided and empowering users to override/remove the base packages via a blueprint (or other method). That way we get away from hard coding things and backing ourselves (and customers) into corners. This causes so much pain right now particularly with packages like podman, when users don't need/want containers. We can still provide support because we can point to an image override if something doesn't work as expected.

I don't see this happening any time soon (or at all). We intentionally don't want users to be able to exclude packages from the image base package set. @supakeen is experimenting with an "empty" image in Fedora and an option to exclude packages in BP, but that would still be possible only with user-specified packages, not the base package set.

On a general note, all image types produced by image builder come without any user by default. Things are different for various image types, but in general there is always some form of mechanism for creating users on the first boot (cloud-init, ignition, cloud-specific guest tools, etc.).

So having an image type which does not have any default user (which is desired), does not have any way to create it on the first-boot and needs a user in order to use the OS (e.g. container images don't need one) is an anti-pattern from my PoV and something that would not make sense...

[supakeen]

Looks OK in my book; I agree with @thozza on the users/initial-setup.

@nullr0ute Are these images to be used on Raspberry Pi's as well (at least for Fedora)? If so you might want to do something similar as done for Aarch64_IoT in distro/fedora/distro.go.

[nullr0ute]

OK I think this is ready for review and merge, just generating test images.

[achilleas-k]

LGTM but I wont approve since I wrote some of the changes myself. I'd like to get an extra pair of eyes on this before merging.

[nullr0ute]

OK I think this is ready for review and merge, just generating test images.

I've tested both RHEL and Fedora images on an imx8m device and works as expected 🥳

[thozza]

I added a few minor comments, but in general this looks OK.

@nullr0ute I don't see any partition table start offset being set for RHEL-8 image. Is this intentional?

Commented "LGTM" later in the PR...

[mrguitar]

@thozza RHEL 8 doesn't have the necessary aarch64 kernel enablement for minimal-raw to be viable. I don't believe that's a target. I suspect that's why it's missing here.

[nullr0ute]

@nullr0ute I don't see any partition table start offset being set for RHEL-8 image. Is this intentional?

Yes, RHEL-8 only supports ServerReady and that doesn't need the offsets if people wish to use it on that release.

[nullr0ute]

I had planned, had a brief discussion with @achilleas-k, on doing some more cleanups around the Fedora pieces on a some follow up clean ups (I have some other ideas for cleanups too), the el8 offset one has now been answered, what's left?

[thozza]

I had planned, had a brief discussion with @achilleas-k, on doing some more cleanups around the Fedora pieces on a some follow up clean ups (I have some other ideas for cleanups too), the el8 offset one has now been answered, what's left?

Nothing, I got all the answers I needed... Approved...

[miabbott]

on doing some more cleanups around the Fedora pieces on a some follow up clean ups (I have some other ideas for cleanups too),

@nullr0ute could you file issues for these changes? It would enable us to get others working on the changes while you are on PTO (or even when you are not on PTO).