fix(sysext): stop merging extensions over the persistent partition

[ci-robbot]

Fixes #5076
Fixes #5132

Summary

/usr/local is where COS_PERSISTENT is mounted, and
99_sysext.yaml named six paths under it in
SYSTEMD_SYSEXT_HIERARCHIES. A successful sysext merge makes every
hierarchy it covers read-only: merge_hierarchy() in systemd's
src/sysext/sysext.c mounts the overlay and then calls
bind_remount_recursive(..., MS_RDONLY, MS_RDONLY, ...). So any machine
that merged one extension lost the writable half of its persistent
partition for the rest of the boot.

That is not only a test problem:

• Bundles install into /usr/local/bin. tests/install_test.go has one,
  and that is what reddened test-core / install and
  test-core-fips / install on master from run
  36892451653
  onwards, the first master run carrying test(sysext): restore the GRUB live media coverage on a verity-only extension #5110.
• 00_home_dir_owner_fix.yaml writes /usr/local/bin/fix-home-dir-ownership
  in the network stage, and 26_selinux.yaml writes three scripts into
  /usr/local/bin in initramfs. Both run after systemd-sysext.
• Per kairos-agent: a failed bundle install is completely silent, and RunBundles reports only the last failure #4780 those writes fail silently, so the symptom is a missing file,
  not an error.

Two commits:

1. Drop every /usr/local path from the three hierarchy lists the file
   declares (the UKI drop-in, the GRUB drop-in and the profile.d
   snippet), leaving the image-owned ones:
   /usr/bin:/usr/share:/usr/lib:/usr/include:/usr/src:/usr/sbin.
2. Rebuild both test extensions with their hello.sh payload at
   /usr/bin/ instead of /usr/local/bin/, so both specs keep running
   the command rather than falling back to reading the extension-release
   file. The GRUB image stays verity-only, the UKI image stays verity +
   signed with tests/assets/keys/db.key, so each still matches the
   image policy its boot path enforces. hello-broke.sysext.raw is
   untouched.

This is the direction @mudler asked for in
#5115 (comment), and
it answers the question #5076 was opened to decide.

Behaviour change worth calling out: anyone shipping an extension that
puts binaries in /usr/local/bin has to ship them in /usr/bin after
this. That is the convention anyway, and /usr/local belongs to the
machine.

How it was tested

• New kairos-init/pkg/bundled/sysext_hierarchies_test.go reads the
  shipped 99_sysext.yaml back through bundled.EmbeddedConfigs, and
  fails if any hierarchy is /usr/local or under it, if one of the
  image-owned ones goes missing, or if the three lists stop agreeing.
  Verified it catches a regression: putting /usr/local/bin back at the
  head of one list fails two of the four specs.
• go test ./kairos-init/pkg/bundled/... green, gofmt -l clean,
  CGO_ENABLED=0 go vet ./... clean in the kairos-tests module.
• The rebuilt images were checked against
  immucore/pkg/state/verity_signature_internal_test.go, which reads
  both fixtures and asserts on the verity signature partition: 13 specs
  green. Their GPT entries decode to the expected
  root-x86-64 / root-x86-64-verity / root-x86-64-verity-sig types,
  and the two partition UUIDs concatenate to the rootHash in the UKI
  image's signature blob.
• Read merge_hierarchy() in systemd v255 to confirm both halves: the
  read-only remount, and that a hierarchy is merged when the extension's
  copy of it is non-empty.

I cannot boot a VM here, so CI is the reproduction. The checks that matter
are test-core / install, test-core-fips / install, test-core / sysext
and the UKI cells.

AI disclosure

Written by an AI agent. No human has read the code yet.

[ci-robbot]

Status check, 2026-10-02 04:20Z: master is still red. Its newest run is 36892451653 at 8d04745f, which is current master HEAD, and test-core / install plus test-core-fips / install are still failing there, 12 hours on.

This PR cannot show it fixes them: authorize has been waiting since it opened, so no test job has ever started on it. @mudler it needs a CI approval before it can be merged. (The third red in that run, build-flavors / fedorariscv, is the separate mirror outage in #4986 and is not touched here.)

[ci-robbot]

Correction to my last note: I could regenerate them after all, so the fixture is now at /usr/bin/hello.sh as you asked and both specs run the command again (b6f0d374). GRUB stays verity-only, UKI stays verity + signed with tests/assets/keys/db.key, and immucore/pkg/state/verity_signature_internal_test.go reads both images green.

[ci-robbot]

Since my 04:20Z note master has gone red twice more on the same two jobs, so it is now every run since #5110 landed: 36892451653 (10-01 16:30Z), 36976440429 (10-02 07:01Z) and 37016179742 (10-02 13:54Z), ~22 hours with no green in between.

@mudler the fixture move you asked for is in b6f0d374 and that thread is resolved, so the only thing left is the CI approval: authorize is still waiting, so test-core / install has never run here.

[mudler]

@ci-robbot as a follow up Auroraboot should warn if seeing files that are not in the hierarchy listed here when creating extensions

[ci-robbot]

Filed as #5223. It is wider than /usr/local: auroraboot sysext packs all of /usr, so /usr/libexec (the Red Hat family's helper binaries) and /opt (which --include-path=/opt exists to pack) are already shipped and never merged today. Once this PR lands, /usr/local/* joins them, which is the case you were asking about.

[ci-robbot]

Picking this up for QA.

[ci-robbot]

QA Result: PASS

Head b6f0d374, unchanged since the PASS verdicts on #5076 and #5132. Those verdicts covered the GRUB path, and both said the UKI drop-in had never been booted. This cycle boots the UKI drop-in, and also checks for drift against current master.

Check	Result
Environment	Trusted boot (UKI) image: ghcr.io/kairos-io/hadron-trusted:v0.5.3 + kairos-init master-scratch, TRUSTED_BOOT=true. Two images that differ only in /system/oem/99_sysext.yaml: master af22a2ba or this PR. UKI ISOs from auroraboot:v0.27.1 build-uki with tests/assets/keys, and --overlay-iso with tests/assets/sysext-uki (master's copy for the master arm, the PR's copy for the PR arm), as _build-iso.yaml does. QEMU q35 with Secure Boot (OVMF secboot, keys auto-enrolled) and swtpm, x86_64 KVM.
Reproduction	Auto-install of each ISO with tests/assets/uki-install.yaml. Then boot the installed disk with no ISO attached (active_mode + uki_boot_mode). On the node, read the drop-in, profile.d, systemd-sysext status, /proc/mounts, run hello.sh, and touch in /usr/local/bin and /usr/local. Drift: git log 1c8f5bb7..up/master on the touched paths, git merge-tree onto af22a2ba, and go test ./kairos-init/pkg/bundled/... on the merge.
Expected	With the PR config, the signed work extension still merges under the UKI signed image policy, hello.sh runs from /usr/bin, the unsigned hello-broke is rejected, and /usr/local/bin stays writable.
Actual	Master config (control): the drop-in lists /usr/local/bin:.... work is merged on /usr/local/bin as sysext ... overlay ro. touch /usr/local/bin/x fails with Read-only file system. The bug reproduces on UKI too. PR config: the drop-in and profile.d list /usr/bin:/usr/share:/usr/lib:/usr/include:/usr/src:/usr/sbin. systemd-sysext.service is Result=success. work is merged on /usr/bin and /usr/lib. /usr/bin/hello.sh prints Hello world. /run/extensions has work.sysext.raw only (hello-broke rejected). /usr/local is /dev/mapper/vda3 ext4 rw with no overlay, and writes to /usr/local/bin and /usr/local work. Drift: the only master commit since the tested base that touches these paths is a4d118760 (sdk/signatures), which has nothing to do with sysext. The merge onto af22a2ba is clean. kairos-init/pkg/bundled passes, 97 of 97 specs, including the 4 new SysextHierarchies specs.
Proofs	Probe script and full output for both arms: https://gist.github.com/ci-robbot/ce206213b0e3d57e55012d4bf2c438fa. GRUB proofs from the earlier cycles: #5076, #5132.

Summary: The UKI drop-in behaves the same as the GRUB one. Under the master config a signed extension makes /usr/local/bin read-only, and under this PR's config it merges into /usr/bin and the persistent partition stays writable. The head has not changed since the GRUB PASS verdicts and still merges cleanly onto current master.

Not caused by this PR, seen the same on both arms: systemd-tpm2-setup.service fails, and mkdir /usr/lib/extensions: read-only file system is logged from 99_sysext.yaml. The second one is already noted in the #5076 verdict. The PR's e2e jobs are still waiting on authorize, so CI has not run them.

[ci-robbot]

Status, 2026-10-06 00:00Z: test-core / install and test-core-fips / install are still red on every run of the PR workflow, now since 2026-10-02, on the same spec and the same assertion (/usr/local/bin/usr/bin/edgevpn: No such file or directory, tests/install_test.go:197) — latest sightings are 37319948587 on triage/5229-reset-eof-no-panic and 37262892710 on mine, so it costs every open pull request two red legs.

This PR is approved, QA: pass and MERGEABLE; its authorize gate has never run, so the only thing left is a merge.