firstboot: initial support by lzap · Pull Request #1705 · osbuild/images

lzap · 2025-07-29T18:04:29Z

This is a very ugly draft of the firstboot support, there are no BP changes yet only the internal BP struct because I think this will change as the review goes on. Also, I stuffed everything into one commit, will break it up later. I just want to get some feedback because I am not sure about this yet.

There are three ways how to submit firstboot:

custom script (or executable of any kind with shebang, only text-based are supported no base64 encoding)
aap registration info
satellite registration command

Scripts are executed in-order and they do not fail the whole systemd unit, unless flag is set. Optional name can be provided.

For AAP, host config key and URL must be provided, the fistboot script is rendered to a shell script which looks like:

#/usr/bin/bash
curl -s -i --data "host_config_key=XXXXXX" https://aap-server/api/v2/job_templates/XXXX/callback/

Satellite is actually a shell one-liner generated by Satellite UI/CLI, looks like:

set -o pipefail && curl -sS 'https://ec2-107-23-89.compute-1.amazonaws.com/register?activation_keys=ak&location_id=2&organization_id=1&update_packages=false' -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjo1LCJpYXQiOjE3MjM4Mzc3MjIsImp0aSI6IjBhNTU3MDM3ZDIyNzUyMDYwM2M3MWIzZDI4NGYwZjQ1NmFjYjE5NzEyNmFmNTk5NzU0NWJmODcwZDczM2RhY2YiLCJleHAiOjE3MjM4NTIxMjIsInNjb3BlIjoicmVnaXN0cmF0aW9uI2dsb2JhbCByZWdpc3RyYXRpb24jaG9zdCJ9.HsSnZEqq--MIJfP3_awn6SflEruoEm77iSWh0Pi6EW4' | bash

In both cases, additional CA certificates (one or more) can be provided which are then added to the list of CA certs enrolled into the system bundle. This is done by a separate code (and stage) which I contributed last year.

achilleas-k

This is a very ugly draft of the firstboot support,

I'll skip any comments on code style then.

there are no BP changes yet only the internal BP struct because I think this will change as the review goes on.

I think in this case, the blueprint change is the most important thing we need to figure out. I think if we all had to separately figure out how this would work internally we'd all have roughly the same idea. We already have a few firstboot-type things (the subscription handling, the frontend feature, etc), so in terms of designing what these things will look like in the manifest, we have that pretty much solved.

achilleas-k · 2025-07-30T10:37:09Z

pkg/manifest/firstboot.go

+		}
+
+		// create the executable
+		exec := fmt.Sprintf("/usr/local/sbin/%s", name)


Why sbin and not bin?

Ah is a symlink now, old habit I guess, fixing.

As followup to that, should this really life in /usr/local/bin/ ? it will be available via PATH for the entire life of the system, shouldn't it go to a different place outside PATH? I mean, its not meant to be run by the user and not meant to be run again

/usr/libexec might make more sense for one-off scripts not expected to be ran by users.

/usr/libexec might make more sense for one-off scripts not expected to be ran by users.

We shouldn't ideally put there random scripts that are not owned by any package on the system. But in general, I agree. So /usr/local/libexec/ would probably be the ideal place.

achilleas-k · 2025-07-30T10:44:07Z

pkg/blueprint/firstboot_customizations.go

+	// Optional Satellite firstboot customization which is executed before all
+	// custom firstboot scripts.
+	Satellite *SatelliteFirstbootCustomization `json:"satellite,omitempty" toml:"satellite,omitempty"`
+
+	// Optional Ansible Automation Platform (AAP) firstboot customization which
+	// is executed before all custom firstboot scripts.
+	AAP *AAPFirstbootCustomization `json:"aap,omitempty" toml:"aap,omitempty"`


Is there any case where one would want multiple Satellite or AAP things to run? Should those also be arrays? Also, can these all be used in combination?

Or maybe this could be done in the same way as we did partitioning, where we would have an array of FirstBootCustomization at the top level and each customization has a type = custom|satellite|aap? That does get tricky to work with and validate though, so maybe we can avoid it if we can.

Multiple Satellites no. Multiple AAP, I do believe that is very unlikely case but can't say for sure. Run both: yes, although Satellite does provide some Ansible capabilities the AAP is very often used in combination with Satellite IIRC.

That does get tricky to work with and validate though, so maybe we can avoid it if we can.

I thought about it, yeah. Well, what needs to be done must be done. Unless there are objections, I am gonna do that.

One idea tho, this pattern will allow to repeat AAP/Sat. I guess in that case it is an error.

The big advantage of this approach is the ordering is always guaranteed.

thozza · 2025-07-30T10:23:17Z

pkg/blueprint/firstboot_customizations.go

+	// Required firstboot script contents. Strings without shebang will be
+	// interpreted as shell scripts, otherwise the script will be executed
+	// using the shebang interpreter.
+	Contents string `json:"contents,omitempty" toml:"contents,omitempty"`


I suggest providing just a path to a script in the image, or as an alternative to providing the script content. We already have convenient ways to create custom files (scripts) in the image by specifying them inline or referencing them using a URI from the host filesystem.

The same mechanism also allows you to create a systemd unit file itself. I can take your idea to the extreme by saying the feature I am working on can be replaced just with files customization. I want to keep this simple and convenient to the user, which was the motivation for this change.

I'm sorry, but I disagree with your statement because you are trying to make it something that wasn't the core of my comment.

My main point is that file customizations already provide a much more convenient way to incorporate any custom script into the image and will continue to evolve in the future. With your approach, you'd always have to catch up to provide the same level of convenience with custom first boot customization.

EDIT: i.e, for on-prem, the ability to reference an actual local script by its path, instead of adding it inline to the BP, would be IMHO a very convenient thing.

Lastly, I suggest making this an option at least. That way, you can keep your current approach as well.

thozza · 2025-07-30T12:11:41Z

pkg/manifest/firstboot.go

+ConditionPathExists=!/var/local/.osbuild-custom-first-boot-done
+Wants=network-online.target
+After=network-online.target
+After=osbuild-first-boot.service
+
+[Service]
+Type=oneshot
+{{ range .Executables }}
+ExecStart={{ . -}}
+{{ end }}
+ExecStartPost=/usr/bin/touch /var/local/.osbuild-custom-first-boot-done
+RemainAfterExit=yes
+
+[Install]
+WantedBy=basic.target


The customization allows one to define multiple first-boot custom scripts, which is a very good approach.

However, AFAICT, all of them would conditionally (not) run based on the existence of the same file, so in practice, only one would be run.

We could also order them based on the position in the list and allow specifying additional ordering for other unit files.

I will be implementing @achilleas-k suggestion to have one array with multiple "types" of firstboot scripts similarly as in disk customization. Therefore, order will always be guaranteed.

I meant order when systemd runs the unit. That needs to be expressed in the unit file, which was not handled at all in the original PR. Lastly, the ConditionPathExists needs to be addressed in any case.

lzap · 2025-07-30T13:21:28Z

In the meantime, I was able to test the PR and it fails the build :-D

copying '/run/osbuild/inputs/file-920538f90222d1e89332eb886496c2e09b3770c6f9c67cfddb5207eab6c1039f/sha256:920538f90222d1e89332eb886496c2e09b3770c6f9c67cfddb5207eab6c1039f' -> '/run/osbuild/tree/usr/local/sbin/osbuild-first-satellite'
cp: cannot create regular file '/run/osbuild/tree/usr/local/sbin/osbuild-first-satellite': No such file or directory
Traceback (most recent call last):
  File "/run/osbuild/bin/org.osbuild.copy", line 29, in <module>
    r = main(_args, _args["options"])
  File "/run/osbuild/bin/org.osbuild.copy", line 22, in main
    subprocess.run(cmd + [src, dst], check=True)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['cp', '-a', '--reflink=auto', '--remove-destination', '/run/osbuild/inputs/file-920538f90222d1e89332eb886496c2e09b3770c6f9c67cfddb5207eab6c1039f/sha256:920538f90222d1e89332eb886496c2e09b3770c6f9c67cfddb5207eab6c1039f', '/run/osbuild/tree/usr/local/sbin/osbuild-first-satellite']' returned non-zero exit status 1.

Not sure I understand. Could it be that /usr/local/sbin does not exist yet in the buildroot?

supakeen · 2025-07-30T14:21:35Z

In the meantime, I was able to test the PR and it fails the build :-D

copying '/run/osbuild/inputs/file-920538f90222d1e89332eb886496c2e09b3770c6f9c67cfddb5207eab6c1039f/sha256:920538f90222d1e89332eb886496c2e09b3770c6f9c67cfddb5207eab6c1039f' -> '/run/osbuild/tree/usr/local/sbin/osbuild-first-satellite'
cp: cannot create regular file '/run/osbuild/tree/usr/local/sbin/osbuild-first-satellite': No such file or directory
Traceback (most recent call last):
  File "/run/osbuild/bin/org.osbuild.copy", line 29, in <module>
    r = main(_args, _args["options"])
  File "/run/osbuild/bin/org.osbuild.copy", line 22, in main
    subprocess.run(cmd + [src, dst], check=True)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['cp', '-a', '--reflink=auto', '--remove-destination', '/run/osbuild/inputs/file-920538f90222d1e89332eb886496c2e09b3770c6f9c67cfddb5207eab6c1039f/sha256:920538f90222d1e89332eb886496c2e09b3770c6f9c67cfddb5207eab6c1039f', '/run/osbuild/tree/usr/local/sbin/osbuild-first-satellite']' returned non-zero exit status 1.

Not sure I understand. Could it be that /usr/local/sbin does not exist yet in the buildroot?

Correct, however it's not about the buildroot but the os pipeline itself.

achilleas-k · 2025-07-30T15:17:54Z

Could it be that /usr/local/sbin does not exist yet in the buildroot?

This means the path doesn't exist in the OS tree itself. Which is weird because /usr/local/sbin should be created by the filesystem package.

supakeen · 2025-07-30T16:10:36Z

Could it be that /usr/local/sbin does not exist yet in the buildroot?

This means the path doesn't exist in the OS tree itself. Which is weird because /usr/local/sbin should be created by the filesystem package.

Right, but as a symlink to /usr/local/bin (since usrmerge happened in Fedora, at least). Perhaps something is going on with the symlink.

lzap · 2025-07-30T17:58:15Z

Just for the record, I created an empty build config with just firstboot customizations. It is a Fedora 42 server qcow2 config.

mvo5 · 2025-07-31T06:46:29Z

pkg/manifest/firstboot.go

+	"github.com/osbuild/images/pkg/customizations/fsnode"
+)
+
+// checkName prevents path traversal


I think the comment is a bit misleading, the checkName function is much stricter, as a side effect it prevents path traversal but if that was the goal the name could be relaxed.

mvo5 · 2025-07-31T06:47:37Z

pkg/manifest/firstboot.go

+		return fmt.Errorf("name cannot be empty")
+	}
+
+	for _, r := range str {


Could we just use: ^[a-zA-Z0-9_-]+$ here?

mvo5 · 2025-07-31T06:50:48Z

pkg/manifest/firstboot.go

+		}
+
+		// create the executable
+		exec := fmt.Sprintf("/usr/local/sbin/%s", name)


As followup to that, should this really life in /usr/local/bin/ ? it will be available via PATH for the entire life of the system, shouldn't it go to a different place outside PATH? I mean, its not meant to be run by the user and not meant to be run again

lzap · 2025-07-31T08:50:48Z

Can we agree on the customization before I commit to the unmarshaling implementation?

package blueprint

type FirstbootCustomization struct {
	// Type of the firstboot customization. Supported values are:
	// "custom", "satellite", and "aap".
	Type string `json:"type,omitempty" toml:"type,omitempty"`

	// Optional firstboot name. Must be unique within the blueprint and only
	// alphanumeric characters with dashes and underscores are allowed.
	Name string `json:"name,omitempty" toml:"name,omitempty"`

	// Ignore errors when executing the firstboot script and continue with
	// execution of the following firstboot scripts, if any. By default,
	// firstboot scripts are executed in order and if one of them fails, the
	// execution stops immediately.
	IgnoreFailure bool `json:"ignore_failure,omitempty" toml:"ignore_failure,omitempty"`

	// Optional CA certificate to enroll into the system before executing the
	// firstboot script.
	CACerts []string `json:"cacerts,omitempty" toml:"cacerts,omitempty"`

	CustomFirstbootCustomization
	SatelliteFirstbootCustomization
	AAPFirstbootCustomization
}

type CustomFirstbootCustomization struct {
	// Strings without shebang will be interpreted as shell scripts, otherwise
	// the script will be executed using the shebang interpreter. Required if
	// type is set to "custom".
	Contents string `json:"contents,omitempty" toml:"contents,omitempty"`
}

type SatelliteFirstbootCustomization struct {
	// Registration command as generated by the Satellite server. Required, if
	// type is set to "satellite".
	Command string `json:"command,omitempty" toml:"command,omitempty"`
}

type AAPFirstbootCustomization struct {
	// Job template URL as generated by the AAP server. Required if type is set
	// to "aap". Example URLs are
	// https://aap.example.com/api/controller/v2/job_templates/9/callback/ or
	// https://aap.example.com/api/v2/job_templates/9/callback/ depending on the
	// AAP version.
	JobTemplateURL string `json:"job_template_url,omitempty" toml:"job_template_url,omitempty"`

	// The host config key. Required if type is set to "aap".
	HostConfigKey string `json:"host_config_key,omitempty" toml:"host_config_key,omitempty"`
}

I am thinking prefixing all AAP fields with AAP and Satellite as well because possible future clash when we add a new type.

lzap · 2025-07-31T09:41:35Z

Please don't just downvote, explain what is wrong with this.

thozza · 2025-07-31T10:24:02Z

Please don't just downvote, explain what is wrong with this.

Sorry. Basically, the comment that @mvo5 had on Slack. I'd suggest doing it the other way around. Instead of FirstbootCustomization being the superset of all customizations and embedding them, have a genericFirstbootCustomization that is embedded by all available types of customizations.

lzap · 2025-07-31T11:29:02Z

I'd suggest doing it the other way around.

I think I am not following this, fill me in please @mvo5 or @thozza with an example.

Edit: I tried this but I am coming up with some wild things, I guess I just do not understand what you mean.

thozza · 2025-08-13T08:50:17Z

I'd suggest doing it the other way around.

I think I am not following this, fill me in please @mvo5 or @thozza with an example.

Edit: I tried this but I am coming up with some wild things, I guess I just do not understand what you mean.

It is weird from my PoV for the FirstbootCustomization to be a superset of all types of firstboot customizations. This means that fields that are not relevant for i.e. an AAP firstboot customization, should not be available in struct representing i.e. a custom firstboot customization. If there are common parts for all customizations, then these should be extracted into a common structure that is embedded in each firstboot customization type. Not the other way around - all types being embedded in the single fistboot customization struct.

Hope this helps.

kingsleyzissou · 2025-08-13T09:09:40Z

pkg/customizations/firstboot/firstboot.go

+type AAPFirstbootOptions struct {
+	JobTemplateURL string
+	HostConfigKey  string
+	CACerts        []string


I think CRC also accepts a skip tls verification option too:

https://github.com/osbuild/image-builder-crc/blob/cbfac18e1a143fc761a396a4b2a0b3538dba2fb1/internal/v1/api.yaml#L1480-L1492

I was thinking not having any certs would skip but being explicit is probably safer.

Ya that's valid, just making a note of it :)

lzap · 2025-08-14T14:34:04Z

Since blueprints were extracted now, I am moving the conversation about the customization struct back to osbuild/blueprint#23 then I will continue this PR once that one is merged.

lzap · 2025-08-18T15:46:00Z

Note for myself: Once blueprint is merged and upgraded, resolve comments above.

As followup to that, should this really life in /usr/local/bin/ ? it will be available via PATH for the entire life of the system, shouldn't it go to a different place outside PATH? I mean, its not meant to be run by the user and not meant to be run again

Good point, libexec sounds nicer. The thread is closed so making a note here:

Rename script path

github-actions · 2025-09-18T04:00:39Z

This PR is stale because it had no activity for the past 30 days. Remove the "Stale" label or add a comment, otherwise this PR will be closed in 7 days.

github-actions · 2025-09-26T04:00:27Z

This PR was closed because it has been stalled for 30+7 days with no activity.

achilleas-k reviewed Jul 30, 2025

View reviewed changes

thozza reviewed Jul 30, 2025

View reviewed changes

mvo5 reviewed Jul 31, 2025

View reviewed changes

firstboot: initial support

36365ef

lzap force-pushed the firstboot1 branch from 0ffbc38 to 36365ef Compare August 4, 2025 08:05

kingsleyzissou reviewed Aug 13, 2025

View reviewed changes

achilleas-k mentioned this pull request Aug 18, 2025

customization: add firstboot osbuild/blueprint#23

Merged

github-actions bot added the Stale label Sep 18, 2025

github-actions bot closed this Sep 26, 2025

lzap mentioned this pull request Sep 30, 2025

manifest: add firstboot support #1913

Open

Conversation

lzap commented Jul 29, 2025

Uh oh!

achilleas-k left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lzap Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lzap Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thozza Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lzap commented Jul 30, 2025

Uh oh!

supakeen commented Jul 30, 2025

Uh oh!

achilleas-k commented Jul 30, 2025

Uh oh!

supakeen commented Jul 30, 2025

Uh oh!

lzap commented Jul 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lzap commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lzap commented Jul 31, 2025

Uh oh!

thozza commented Jul 31, 2025

Uh oh!

lzap commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thozza commented Aug 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lzap commented Aug 14, 2025

Uh oh!

lzap commented Aug 18, 2025

Uh oh!

github-actions bot commented Sep 18, 2025

Uh oh!

lzap Jul 30, 2025 •

edited

Loading

lzap Jul 31, 2025 •

edited

Loading

thozza Jul 31, 2025 •

edited

Loading

lzap commented Jul 31, 2025 •

edited

Loading

lzap commented Jul 31, 2025 •

edited

Loading