Touch crossdeps-builder clone step

[sbomer]

Test to see what gets rebuilt when touching the arcade clone step

[am11]

When we only touch crossdeps-builder and the target cross dockerfiles, the change in llvm command does not reflect. e.g. when I added LoongArch in this file and touched cross/loongarch, it didn't located the new architecture. It required us to also touch crossdeps-llvm. Touching crossdeps-llvm builds everything under that netXX/cross.

Ideally, we should do this:

• Swap naming: crossdeps-llvm is more suitable name for crossdeps-builder; where we are actually building llvm, and vice-versa.
• Move arcade clone step to cross images. To accommodate: if someone made a change in arcade, which doesn't require rebuilding llvm and anything else.

[sbomer]

when I added LoongArch in this file and touched cross/loongarch, it didn't located the new architecture. It required us to also touch crossdeps-llvm

@mthalman is this expected? I would have thought that if I touch crossdeps-builder, images which depend on it (like crossdeps-llvm) would get rebuilt too. This seems like it might be a bug.

Touching crossdeps-llvm builds everything under that netXX/cross.

That is expected - if we make changes to the LLVM toolchain, we want the images that depend on it to get rebuilt.

Swap naming: crossdeps-llvm is more suitable name for crossdeps-builder; where we are actually building llvm, and vice-versa.

crossdeps-builder is a helper that contains build-time dependencies of the cross images that don't need to be part of the final images (llvm sources, llvm build dependencies, debootstrap, and the rootfs build scripts). crossdeps-llvm contains the runtime dependencies, including llvm, and serves as a base for the cross images - it's just the crossdeps image plus an install of llvm that we built in crossdeps-builder.

Move arcade clone step to cross images. To accommodate: if someone made a change in arcade, which doesn't require rebuilding llvm and anything else.

If we make an update to arcade, it should ideally rebuild all of the cross images. I agree it would be nice not to rebuild llvm. The arcade clone step comes after the llvm build which does locally allow reuse of the cached llvm build layer - except that our ci doesn't use a docker cache.

I'd like to keep the current architecture where one arcade clone is shared by all of the cross images, otherwise they could get out of sync and have different arcade versions. If we really need to work around the caching limitations, maybe we can separate the arcade clone into a separate image that is based on crossdeps-builder, so that the ci cache picks it up?

The current PR looks like it rebuilt llvm in only about 1h20m. I remember it used to take much longer, but I think 1h20m is an acceptable wait for the times we need to manually touch dockerfiles to flow arcade changes.

[am11]

crossdeps-builder is a helper that contains build-time dependencies of the cross images that don't need to be part of the final images (llvm sources, llvm build dependencies, debootstrap, and the rootfs build scripts). crossdeps-llvm contains the runtime dependencies, including llvm, and serves as a base for the cross images - it's just the crossdeps image plus an install of llvm that we built in crossdeps-builder.

I think there is a misunderstanding. Naming seem inverted. If the one with "llvm" in its name is building the actual llvm toolchian, it will remedy the confusion. The cross dockerfiles can use the new name (by updating their first line).

I'd like to keep the current architecture where one arcade clone is shared by all of the cross images, otherwise they could get out of sync and have different arcade versions. If we really need to work around the caching limitations, maybe we can separate the arcade clone into a separate image that is based on crossdeps-builder, so that the ci cache picks it up?

We can still achieve that by moving clone step to the middle layer (current: crossdeps-llvm, proposed: crossdeps-builder), then we don't need to rebuild llvm toolchain. It will rebuild the cross/ images only, which is better than touching multiple files.

@mthalman is this expected? I would have thought that if I touch crossdeps-builder, images which depend on it (like crossdeps-llvm) would get rebuilt too. This seems like it might be a bug.

Yes, it does seem unintentional / bug. For clarity, the first PR was #1306,, then we had to do another to rebuild the middle image #1309 because it wasn't built.

[sbomer]

If the one with "llvm" in its name is building the actual llvm toolchian

The crossdeps-builder images contains more than just llvm - it has all the shared dependencies used to build the cross images. The (current) crossdeps-llvm image contains just the runtime cross-build dependencies plus llvm. Why is crossdeps-builder a better name for that one?

We can still achieve that by moving clone step to the middle layer

The layering is designed so that the final cross images contain just what's needed to build runtime. They intentionally exclude assets only used during the image construction (like arcade, debootstrap, and the llvm sources). Moving the clone step to crossdeps-llvm would violate this.