test/manifests/all-customizations: don't embed local file via URI #1684

thozza · 2025-07-23T13:48:51Z

The all-customizations (and empty) config is used for manifest tests in osbuild. Previously, the all-customizations config was specifying a file customization referencing a local file from the host using URI. This turned out to be problematic, because all manifests in images CI are generated and built on Fedora host, thus embedding /etc/os-release from Fedora. However, in osbuild, manifests are generated and built on the same distribution as the manifest's distribution. As a result, the manifest checksums differed for the all-customizations manifests on all distros, except for the Fedora version used by images CI.

Initially, the plan was to workaround the issue by mocking the local file. However, that would not work, because the manifest is actually being built as part of the CI pipeline. The local file is turned into CURL source in the manifest, with its URI pointing to a temporary path that would not exist on the host at the time of image build.

Therefore, to solve the issue with manifest tests in osbuild, do not use the file customization with URI, pointing to a local file, in the all-customizations config.

The URI scenario is moved to a dedicated config, so it is still boot-tested in images CI, but not used in osbuild for manifest tests.

I also kept the enhancements that were part of the initial version of this PR, as I believe they are still useful.

This will make debugging much easier. Signed-off-by: Tomáš Hozza <[email protected]>

This will help with debugging potential issues in CI. Signed-off-by: Tomáš Hozza <[email protected]>

mvo5

Thank you - out of curiosity, why is this draft? What is missing?

mvo5 · 2025-07-23T15:26:01Z

cmd/gen-manifests/main.go

+	// we might need to write temporarily)
+	tmpdirRoot, err := os.MkdirTemp("", "gen-manifests-tmpdir-*")
+	if err != nil {
+		panic(fmt.Sprintf("failed to create temporary directory: %s", err.Error()))


(super nitpick) if we use fmt.Errorf("failed to create temporary directory: %w", err)) here we are ready for a time when we maybe panic less

I changed the code a bit to make the generation of manifests stable across tool executions and there is no longer any check for an error.

bcl

Looks good to me. Thanks!

The source URI for the file to be embedded doesn't matter, so we can use a temporary directory for it. This solves an issue where the manifest generation would fail when the cacheRoot was not an absolute path, since the blueprint customization requires absolute paths for content URIs. Create a tmpdirRoot in main() and delete it at the end so we can reuse it for any other temporary file creation we might need.

The all-customizations (and empty) config is used for manifest tests in osbuild. Previously, the all-customizations config was specifying a file customization referencing a local file from the host using URI. This turned out to be problematic, because all manifests in images CI are generated and built on Fedora host, thus embedding /etc/os-release from Fedora. However, in osbuild, manifests are generated and built on the same distribution as the manifest's distribution. As a result, the manifest checksums differed for the all-customizations manifests on all distros, except for the Fedora version used by images CI. Initially, the plan was to workaround the issue by mocking the local file. However, that would not work, because the manifest is actually being built as part of the CI pipeline. The local file is turned into CURL source in the manifest, with its URI pointing to a temporary path that would not exist on the host at the time of image build. Therefore, to solve the issue with manifest tests in osbuild, do not use the file customization with URI, pointing to a local file, in the all-customizations config. The URI scenario will be moved to a dedicated config in the next commit, so it will be still boot-tested in images CI, but not used in osbuild for manifest tests. Signed-off-by: Tomáš Hozza <[email protected]>

Add a dedicated config for file customizations, including embedding a local file using an URI. Use the config for AMIs, so that it will get boot-tested in images CI. Signed-off-by: Tomáš Hozza <[email protected]>

achilleas-k

Thank you!!

bcl

Ack

tag v0.165.0 Tagger: imagebuilder-bot <[email protected]> Changes with 0.165.0 ---------------- * distro: move rhel9 into a generic distro (osbuild/images#1645) * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Simon de Vlieger * Revert "distro: drop `ImageType.BasePartitionTable()`" (osbuild/images#1691) * Author: Michael Vogt, Reviewers: Simon de Vlieger, Tomáš Hozza * Update dependencies 2025-07-20 (osbuild/images#1675) * Author: SchutzBot, Reviewers: Achilleas Koutsou, Simon de Vlieger * defs: add missing `bootstrap_containers` (osbuild/images#1679) * Author: Michael Vogt, Reviewers: Simon de Vlieger, Tomáš Hozza * disk: handle adding `PReP` partition on PPC64/s390x (HMS-8884) (osbuild/images#1681) * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Simon de Vlieger * distro: bring per-distro checkOptions back (osbuild/images#1678) * Author: Michael Vogt, Reviewers: Simon de Vlieger, Tomáš Hozza * distro: cleanups in the pkg/distro/generic area (osbuild/images#1686) * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Simon de Vlieger * distro: move rhel8 into a generic distro (osbuild/images#1643) * Author: Michael Vogt, Reviewers: Nobody * distro: small followups for PR#1682 (osbuild/images#1689) * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Simon de Vlieger, Tomáš Hozza * distro: unify transform/match into a single concept (osbuild/images#1682) * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Tomáš Hozza * distros: de-duplicate runner build packages for centos10 (osbuild/images#1680) * Author: Michael Vogt, Reviewers: Simon de Vlieger, Tomáš Hozza * github: disable Go dep updates through dependabot (osbuild/images#1683) * Author: Achilleas Koutsou, Reviewers: Simon de Vlieger, Tomáš Hozza * repos: include almalinux 9.6 (osbuild/images#1677) * Author: Simon de Vlieger, Reviewers: Lukáš Zapletal, Tomáš Hozza * rhel9: wsl distribution config (osbuild/images#1694) * Author: Simon de Vlieger, Reviewers: Michael Vogt, Sanne Raymaekers * test/manifests/all-customizations: don't embed local file via URI (osbuild/images#1684) * Author: Tomáš Hozza, Reviewers: Achilleas Koutsou, Brian C. Lane — Somewhere on the Internet, 2025-07-28 --- tag v0.166.0 Tagger: imagebuilder-bot <[email protected]> Changes with 0.166.0 ---------------- * customizations/subscription: conditionally enable semanage call (HMS-8866) (osbuild/images#1673) * Author: Sanne Raymaekers, Reviewers: Achilleas Koutsou, Michael Vogt * distro/rhel-10: versionlock shim-x64 in the azure-cvm image (osbuild/images#1697) * Author: Achilleas Koutsou, Reviewers: Michael Vogt, Simon de Vlieger * manifestmock: move container/pkg/commit mocks into helper (osbuild/images#1700) * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Simon de Vlieger * rhel9: `vagrant-libvirt`, `vagrant-virtualbox` (osbuild/images#1693) * Author: Simon de Vlieger, Reviewers: Michael Vogt, Sanne Raymaekers * rhel{9,10}: centos WSL refinement (HMS-8922) (osbuild/images#1690) * Author: Simon de Vlieger, Reviewers: Ondřej Budai, Sanne Raymaekers, Tomáš Hozza — Somewhere on the Internet, 2025-07-29 --- tag v0.167.0 Tagger: imagebuilder-bot <[email protected]> Changes with 0.167.0 ---------------- * RHEL/Azure: drop obsolete WAAgentConfig keys [RHEL-93894] and remove loglevel kernel option [RHEL-102372] (osbuild/images#1611) * Author: Achilleas Koutsou, Reviewers: Michael Vogt, Ondřej Budai, Sanne Raymaekers * Update dependencies 2025-07-27 (osbuild/images#1699) * Author: SchutzBot, Reviewers: Achilleas Koutsou, Simon de Vlieger * distro/rhel9: set default_kernel to kernel-uki-virt (osbuild/images#1704) * Author: Achilleas Koutsou, Reviewers: Ondřej Budai, Simon de Vlieger * distro: drop legacy loaders and update tests (osbuild/images#1687) * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Tomáš Hozza * distro: fix issues with yaml distro definitions and enable yaml checks (osbuild/images#1702) * Author: Achilleas Koutsou, Reviewers: Michael Vogt, Ondřej Budai, Simon de Vlieger — Somewhere on the Internet, 2025-07-30 ---

tag v0.165.0 Tagger: imagebuilder-bot <[email protected]> Changes with 0.165.0 ---------------- * distro: move rhel9 into a generic distro (osbuild/images#1645) * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Simon de Vlieger * Revert "distro: drop `ImageType.BasePartitionTable()`" (osbuild/images#1691) * Author: Michael Vogt, Reviewers: Simon de Vlieger, Tomáš Hozza * Update dependencies 2025-07-20 (osbuild/images#1675) * Author: SchutzBot, Reviewers: Achilleas Koutsou, Simon de Vlieger * defs: add missing `bootstrap_containers` (osbuild/images#1679) * Author: Michael Vogt, Reviewers: Simon de Vlieger, Tomáš Hozza * disk: handle adding `PReP` partition on PPC64/s390x (HMS-8884) (osbuild/images#1681) * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Simon de Vlieger * distro: bring per-distro checkOptions back (osbuild/images#1678) * Author: Michael Vogt, Reviewers: Simon de Vlieger, Tomáš Hozza * distro: cleanups in the pkg/distro/generic area (osbuild/images#1686) * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Simon de Vlieger * distro: move rhel8 into a generic distro (osbuild/images#1643) * Author: Michael Vogt, Reviewers: Nobody * distro: small followups for PR#1682 (osbuild/images#1689) * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Simon de Vlieger, Tomáš Hozza * distro: unify transform/match into a single concept (osbuild/images#1682) * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Tomáš Hozza * distros: de-duplicate runner build packages for centos10 (osbuild/images#1680) * Author: Michael Vogt, Reviewers: Simon de Vlieger, Tomáš Hozza * github: disable Go dep updates through dependabot (osbuild/images#1683) * Author: Achilleas Koutsou, Reviewers: Simon de Vlieger, Tomáš Hozza * repos: include almalinux 9.6 (osbuild/images#1677) * Author: Simon de Vlieger, Reviewers: Lukáš Zapletal, Tomáš Hozza * rhel9: wsl distribution config (osbuild/images#1694) * Author: Simon de Vlieger, Reviewers: Michael Vogt, Sanne Raymaekers * test/manifests/all-customizations: don't embed local file via URI (osbuild/images#1684) * Author: Tomáš Hozza, Reviewers: Achilleas Koutsou, Brian C. Lane — Somewhere on the Internet, 2025-07-28 --- tag v0.166.0 Tagger: imagebuilder-bot <[email protected]> Changes with 0.166.0 ---------------- * customizations/subscription: conditionally enable semanage call (HMS-8866) (osbuild/images#1673) * Author: Sanne Raymaekers, Reviewers: Achilleas Koutsou, Michael Vogt * distro/rhel-10: versionlock shim-x64 in the azure-cvm image (osbuild/images#1697) * Author: Achilleas Koutsou, Reviewers: Michael Vogt, Simon de Vlieger * manifestmock: move container/pkg/commit mocks into helper (osbuild/images#1700) * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Simon de Vlieger * rhel9: `vagrant-libvirt`, `vagrant-virtualbox` (osbuild/images#1693) * Author: Simon de Vlieger, Reviewers: Michael Vogt, Sanne Raymaekers * rhel{9,10}: centos WSL refinement (HMS-8922) (osbuild/images#1690) * Author: Simon de Vlieger, Reviewers: Ondřej Budai, Sanne Raymaekers, Tomáš Hozza — Somewhere on the Internet, 2025-07-29 --- tag v0.167.0 Tagger: imagebuilder-bot <[email protected]> Changes with 0.167.0 ---------------- * RHEL/Azure: drop obsolete WAAgentConfig keys [RHEL-93894] and remove loglevel kernel option [RHEL-102372] (osbuild/images#1611) * Author: Achilleas Koutsou, Reviewers: Michael Vogt, Ondřej Budai, Sanne Raymaekers * Update dependencies 2025-07-27 (osbuild/images#1699) * Author: SchutzBot, Reviewers: Achilleas Koutsou, Simon de Vlieger * distro/rhel9: set default_kernel to kernel-uki-virt (osbuild/images#1704) * Author: Achilleas Koutsou, Reviewers: Ondřej Budai, Simon de Vlieger * distro: drop legacy loaders and update tests (osbuild/images#1687) * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Tomáš Hozza * distro: fix issues with yaml distro definitions and enable yaml checks (osbuild/images#1702) * Author: Achilleas Koutsou, Reviewers: Michael Vogt, Ondřej Budai, Simon de Vlieger — Somewhere on the Internet, 2025-07-30 --- tag v0.168.0 Tagger: imagebuilder-bot <[email protected]> Changes with 0.168.0 ---------------- * distro: fix bug in variable substitution for static distros (osbuild/images#1710) * Author: Michael Vogt, Reviewers: Achilleas Koutsou, Simon de Vlieger * rhel{9,10}: azure for non-RHEL (HMS-8949) (osbuild/images#1707) * Author: Simon de Vlieger, Reviewers: Achilleas Koutsou, Michael Vogt — Somewhere on the Internet, 2025-07-30 ---

test/scripts/imgtestlib.py: log ENV in gen_manifests()

cd52a56

This will make debugging much easier. Signed-off-by: Tomáš Hozza <[email protected]>

thozza force-pushed the fix-osbuild-manifest-tests branch from b88366a to 85cc3bc Compare July 23, 2025 14:06

test/scripts/dl-image-build-cache: log errors from gen_manifests()

2716cb6

This will help with debugging potential issues in CI. Signed-off-by: Tomáš Hozza <[email protected]>

thozza force-pushed the fix-osbuild-manifest-tests branch 2 times, most recently from c355154 to b6eae27 Compare July 23, 2025 14:53

thozza marked this pull request as ready for review July 23, 2025 14:53

thozza requested review from a team and achilleas-k as code owners July 23, 2025 14:53

thozza requested review from croissanne and mvo5 July 23, 2025 14:53

thozza marked this pull request as draft July 23, 2025 14:59

mvo5 previously approved these changes Jul 23, 2025

View reviewed changes

bcl previously approved these changes Jul 23, 2025

View reviewed changes

thozza dismissed stale reviews from bcl and mvo5 via c2e53b3 July 23, 2025 20:05

thozza force-pushed the fix-osbuild-manifest-tests branch from b6eae27 to c2e53b3 Compare July 23, 2025 20:05

thozza changed the title ~~Make test manifest generation stable when using custom files in BP with URI~~ test/manifests/all-customizations: do not embed local files via URI Jul 23, 2025

thozza force-pushed the fix-osbuild-manifest-tests branch from c2e53b3 to ec1f391 Compare July 23, 2025 20:12

thozza changed the title ~~test/manifests/all-customizations: do not embed local files via URI~~ test/manifests/all-customizations: don't embed local file via URI Jul 23, 2025

thozza marked this pull request as ready for review July 23, 2025 20:12

thozza requested review from bcl and mvo5 July 23, 2025 20:12

thozza added 2 commits July 23, 2025 22:14

test/manifests: add dedicated config for file customizations

5029b43

Add a dedicated config for file customizations, including embedding a local file using an URI. Use the config for AMIs, so that it will get boot-tested in images CI. Signed-off-by: Tomáš Hozza <[email protected]>

thozza force-pushed the fix-osbuild-manifest-tests branch from ec1f391 to 5029b43 Compare July 23, 2025 20:15

thozza mentioned this pull request Jul 23, 2025

Update images dependency ref to latest osbuild/osbuild#2150

Closed

achilleas-k approved these changes Jul 23, 2025

View reviewed changes

achilleas-k enabled auto-merge July 23, 2025 21:48

bcl approved these changes Jul 23, 2025

View reviewed changes

achilleas-k added this pull request to the merge queue Jul 24, 2025

Merged via the queue into osbuild:main with commit 42f92da Jul 24, 2025
21 of 22 checks passed

thozza deleted the fix-osbuild-manifest-tests branch July 24, 2025 05:21

achilleas-k mentioned this pull request Jul 30, 2025

go.mod: update osbuild/images to v0.168.0 osbuild/osbuild-composer#4799

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test/manifests/all-customizations: don't embed local file via URI #1684

test/manifests/all-customizations: don't embed local file via URI #1684

Uh oh!

thozza commented Jul 23, 2025 •

edited

Loading

Uh oh!

mvo5 left a comment

Uh oh!

mvo5 Jul 23, 2025

Uh oh!

thozza Jul 23, 2025

Uh oh!

bcl left a comment

Uh oh!

achilleas-k left a comment

Uh oh!

bcl left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

test/manifests/all-customizations: don't embed local file via URI #1684

test/manifests/all-customizations: don't embed local file via URI #1684

Uh oh!

Conversation

thozza commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mvo5 left a comment

Choose a reason for hiding this comment

Uh oh!

mvo5 Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

thozza Jul 23, 2025

Choose a reason for hiding this comment

Uh oh!

bcl left a comment

Choose a reason for hiding this comment

Uh oh!

achilleas-k left a comment

Choose a reason for hiding this comment

Uh oh!

bcl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

thozza commented Jul 23, 2025 •

edited

Loading