recovery/cos-img results to inconsistent builds

[davidcassany]

cos-toolkit version:

CPU architecture, OS, and Version:

Leap 15.3 x86_64

Describe the bug

After changing some package and bumping its version the command PACKAGES="system/cos recovery/cos recovery/cos-img" make build does not result in an updated recovery/cos-img package. It does for system/cos and recovery/cos but it doesn't for the recovery/cos-img.

To Reproduce

recovery/cos-img results to inconsistent builds

Since few days that I am really struggling to figure what is actually included in recovery/cos image build.

My process froma clean cache state is the following:

1. Checkout and build required artifacts with:

sudo PACKAGES="system/cos recovery/cos recovery/cos-img" time -p make build create-repo

2. I do some changes (e.g I change packages/installer/build.yaml to include a new file and I add a +1 in version from packages/installer/definition.yaml)

3. Rebuild with the same command as I previously did:

sudo PACKAGES="system/cos recovery/cos recovery/cos-img" time -p make build create-repo

Here I can see how utils/installer is being rebuild first for system/cos and the new version appears logs

    5/8 system/cos-0.5.3+4 ⤑ 🔨  build system/grub-config-0.0.6 ✅  Done
 📦  6/8 system/cos-0.5.3+4 ⤑ 🔨  build system/immutable-rootfs-0.0.19+14  starts
 >💧  copy< Package has multi-stage copy, generating required images
 >💧  copy< 1/1 ⤑ 🔨  build toolchain/yip->=0 generating multi-stage images for toolchain/yip->=0
 📦  Compiling toolchain/yip-0.9.2 .... ☕ 
 🌳  Build dependencies for toolchain/yip-0.9.2
  ↪  distro/base-0.20210528+2 🍃 
  ↪  build/golang-1.16.5 🍃 
 📦  1/2 toolchain/yip-0.9.2 ⤑ 🔨  build distro/base-0.20210528+2  starts
 📦  1/2 toolchain/yip-0.9.2 ⤑ 🔨  build distro/base-0.20210528+2 ✅  Done
 📦  2/2 toolchain/yip-0.9.2 ⤑ 🔨  build build/golang-1.16.5  starts
 📦  2/2 toolchain/yip-0.9.2 ⤑ 🔨  build build/golang-1.16.5 ✅  Done
 🚀  All dependencies are satisfied, building package requested by the user toolchain/yip-0.9.2
 📦  toolchain/yip-0.9.2  Using image:  quay.io/costoolkit/build-cache:7141be4ee383c3a7a0ebabbce7f2c949007405d2651a5c3cd38fb4b676271502
 >💧  copy< 1/1 ⤑ 🔨  build toolchain/yip->=0 ✅  Done
 📦  6/8 system/cos-0.5.3+4 ⤑ 🔨  build system/immutable-rootfs-0.0.19+14 ✅  Done
 📦  7/8 system/cos-0.5.3+4 ⤑ 🔨  build utils/installer-0.7.4+1  starts
 📦  utils/installer-0.7.4+1 🐋  Generating 'builder' image from quay.io/costoolkit/build-cache:39f57e220702fea2ea33ef3316f00ade4a32b2107cb8fdc3297c042d6139ed8e as quay.io/costoolkit/build-cache:builder-56a0d4b76fd895a21b3d887b312081ed with pr
elude steps
🚧  warning Failed to download 'quay.io/costoolkit/build-cache:builder-56a0d4b76fd895a21b3d887b312081ed'. Will keep going and build the image unless you use --fatal
🚧  warning Failed pulling image: Error response from daemon: manifest for quay.io/costoolkit/build-cache:builder-56a0d4b76fd895a21b3d887b312081ed not found: manifest unknown: manifest unknown
: exit status 1                                                                                                                                                                                                                                    
 🐋  Building image quay.io/costoolkit/build-cache:builder-56a0d4b76fd895a21b3d887b312081ed

Then I can see how in the context of recovery/cos the package is not rebuild.

...
 copy< 1/1 ⤑ 🔨  build toolchain/yip->=0 ✅  Done                                                                   
 📦  6/8 recovery/cos-0.5.3+4 ⤑ 🔨  build system/immutable-rootfs-0.0.19+14 ✅  Done                                 
 📦  7/8 recovery/cos-0.5.3+4 ⤑ 🔨  build utils/installer-0.7.4+1  starts
 📦  7/8 recovery/cos-0.5.3+4 ⤑ 🔨  build utils/installer-0.7.4+1 ✅  Done
 📦  8/8 recovery/cos-0.5.3+4 ⤑ 🔨  build system/selinux-policies-0.0.6+3  starts                       
 📦  8/8 recovery/cos-0.5.3+4 ⤑ 🔨  build system/selinux-policies-0.0.6+3 ✅  Done
...

Here my first round of doubts start, I would have expected it to be rebuilt since the previous build is part of system/cos tree not recovery/cos. Even if both recipes are exactly the same up to utils/installer layer I would have expected that different packages always differ in build cache regardless they share the same exact build.yaml or not. Anyway this was just my expectation, this is not yet broken, just unexpected behavior from my POV which might not be wrong.

On logs the same applies for recovery/cos-img I see equivalent logs including the bumped version of the installer without the need to actually recompile it.

...
 📦  6/9 recovery/cos-img-0.5.3+4 ⤑ 🔨  build system/immutable-rootfs-0.0.19+14 ✅  Done                                 
 📦  7/9 recovery/cos-img-0.5.3+4 ⤑ 🔨  build utils/installer-0.7.4+1  starts                                                                                                                                                                      
 📦  7/9 recovery/cos-img-0.5.3+4 ⤑ 🔨  build utils/installer-0.7.4+1 ✅  Done                                           
 📦  8/9 recovery/cos-img-0.5.3+4 ⤑ 🔨  build system/selinux-policies-0.0.6+3  starts                                                                                                                                                              
 📦  8/9 recovery/cos-img-0.5.3+4 ⤑ 🔨  build system/selinux-policies-0.0.6+3 ✅  Done                                                                                                                                                             
 📦  9/9 recovery/cos-img-0.5.3+4 ⤑ 🔨  build recovery/cos-0.5.3+4  starts
...

So far it still looks consistent, however the problem starts when start using recovery/cos-img, infact the build image does not include the new file included within the installer system/cos does, however recovery/cos-img doesn't. This is specially confusing when booting the image as there you don't have traces of which version of the utils/installer is actually included within the system. By unpacking recovery/cos I managed to see it actually includes the expected changes and also by directly unpacking and mounting the recovery/cos-img I managed to see how it does not include the latest changes.

Expected behavior

recovery/cos-img gets updated if I run a local build that includes changes in the former recovery/cos tree.

Logs

build.log

See also: #391

[mudler]

cos-toolkit version:

CPU architecture, OS, and Version:

Leap 15.3 x86_64

Describe the bug

After changing some package and bumping its version the command PACKAGES="system/cos recovery/cos recovery/cos-img" make build does not result in an updated recovery/cos-img package. It does for system/cos and recovery/cos but it doesn't for the recovery/cos-img.

To Reproduce

recovery/cos-img results to inconsistent builds

Since few days that I am really struggling to figure what is actually included in recovery/cos image build.

My process froma clean cache state is the following:

1. Checkout and build required artifacts with:

sudo PACKAGES="system/cos recovery/cos recovery/cos-img" time -p make build create-repo

1. I do some changes (e.g I change `packages/installer/build.yaml` to include a new file and I add a `+1` in version from `packages/installer/definition.yaml`)

2. Rebuild with the same command as I previously did:

sudo PACKAGES="system/cos recovery/cos recovery/cos-img" time -p make build create-repo

Here I can see how utils/installer is being rebuild first for system/cos and the new version appears logs

    5/8 system/cos-0.5.3+4 ⤑ 🔨  build system/grub-config-0.0.6 ✅  Done
 📦  6/8 system/cos-0.5.3+4 ⤑ 🔨  build system/immutable-rootfs-0.0.19+14  starts
 >💧  copy< Package has multi-stage copy, generating required images
 >💧  copy< 1/1 ⤑ 🔨  build toolchain/yip->=0 generating multi-stage images for toolchain/yip->=0
 📦  Compiling toolchain/yip-0.9.2 .... ☕ 
 🌳  Build dependencies for toolchain/yip-0.9.2
  ↪  distro/base-0.20210528+2 🍃 
  ↪  build/golang-1.16.5 🍃 
 📦  1/2 toolchain/yip-0.9.2 ⤑ 🔨  build distro/base-0.20210528+2  starts
 📦  1/2 toolchain/yip-0.9.2 ⤑ 🔨  build distro/base-0.20210528+2 ✅  Done
 📦  2/2 toolchain/yip-0.9.2 ⤑ 🔨  build build/golang-1.16.5  starts
 📦  2/2 toolchain/yip-0.9.2 ⤑ 🔨  build build/golang-1.16.5 ✅  Done
 🚀  All dependencies are satisfied, building package requested by the user toolchain/yip-0.9.2
 📦  toolchain/yip-0.9.2  Using image:  quay.io/costoolkit/build-cache:7141be4ee383c3a7a0ebabbce7f2c949007405d2651a5c3cd38fb4b676271502
 >💧  copy< 1/1 ⤑ 🔨  build toolchain/yip->=0 ✅  Done
 📦  6/8 system/cos-0.5.3+4 ⤑ 🔨  build system/immutable-rootfs-0.0.19+14 ✅  Done
 📦  7/8 system/cos-0.5.3+4 ⤑ 🔨  build utils/installer-0.7.4+1  starts
 📦  utils/installer-0.7.4+1 🐋  Generating 'builder' image from quay.io/costoolkit/build-cache:39f57e220702fea2ea33ef3316f00ade4a32b2107cb8fdc3297c042d6139ed8e as quay.io/costoolkit/build-cache:builder-56a0d4b76fd895a21b3d887b312081ed with pr
elude steps
🚧  warning Failed to download 'quay.io/costoolkit/build-cache:builder-56a0d4b76fd895a21b3d887b312081ed'. Will keep going and build the image unless you use --fatal
🚧  warning Failed pulling image: Error response from daemon: manifest for quay.io/costoolkit/build-cache:builder-56a0d4b76fd895a21b3d887b312081ed not found: manifest unknown: manifest unknown
: exit status 1                                                                                                                                                                                                                                    
 🐋  Building image quay.io/costoolkit/build-cache:builder-56a0d4b76fd895a21b3d887b312081ed

Then I can see how in the context of recovery/cos the package is not rebuild.

...
 copy< 1/1 ⤑ 🔨  build toolchain/yip->=0 ✅  Done                                                                   
 📦  6/8 recovery/cos-0.5.3+4 ⤑ 🔨  build system/immutable-rootfs-0.0.19+14 ✅  Done                                 
 📦  7/8 recovery/cos-0.5.3+4 ⤑ 🔨  build utils/installer-0.7.4+1  starts
 📦  7/8 recovery/cos-0.5.3+4 ⤑ 🔨  build utils/installer-0.7.4+1 ✅  Done
 📦  8/8 recovery/cos-0.5.3+4 ⤑ 🔨  build system/selinux-policies-0.0.6+3  starts                       
 📦  8/8 recovery/cos-0.5.3+4 ⤑ 🔨  build system/selinux-policies-0.0.6+3 ✅  Done
...

Here my first round of doubts start, I would have expected it to be rebuilt since the previous build is part of system/cos tree not recovery/cos. Even if both recipes are exactly the same up to utils/installer layer I would have expected that different packages always differ in build cache regardless they share the same exact build.yaml or not. Anyway this was just my expectation, this is not yet broken, just unexpected behavior from my POV which might not be wrong.

Up to here all sounds fine - even if you are targeting two separate images, if they share part of the dep tree then they are going to reuse it as much as possible - that's the benefit of having a sat solver doing this matching.

[mudler]

🤔 logs aren't showing much because it seems it reused most of all the images locally available. It needs a deeper 👀 with a run with the --debug flag on luet build

[mudler]

I think this is related to mudler/luet#234

[mudler]

I think we should check out if this is still the case with luet >=0.17.0. Given the fact that we have to adapt to the changes ( join is marked for deprecation, in favor of just using requires, https://luet-lab.github.io/docs/docs/concepts/packages/specfile/#requires_final_images )

[davidcassany]

I can confirm the issue I reported is already gone with current Luet 👍 Thanks for the fix!