docker: use multi-stage build

[lizan]

Use Docker multi-stage build and migrate all Docker image build process to docker buildx, this saves time and space by not loading images to Docker daemon during the build.

Risk Level: Low
Testing: CI
Docs Changes: N/A
Release Notes: N/A

Signed-off-by: Lizan Zhou lizan@tetrate.io

[repokitteh-read-only]

As a reminder, PRs marked as draft will not be automatically assigned reviewers,
or be handled by maintainer-oncall triage.

Please mark your PR as ready when you want it to be reviewed!

🐱

Caused by: #19789 was opened by lizan.

see: more, trace.

[moderation]

Relevant re: removing Alpine images - #19781

[phlax]

great idea for reducing size!

seems like there is a syntax error of some sort in ci/do_ci.sh

[lizan]

@phlax this should be good to go

[phlax]

lgtm, with a small nit about making the Dockerfile a little clearer - ill leave it to you to address/ignore

i wasnt aware of skopeo until now, looks v useful

the idea of prebuilding the image with the binary is really cool, altho it still seems to be sending a huge context - im wondering if it is worth building the image with the binaries and then deleting them before building the rest or similar. I just think it would speed it up quite a bit not copying out gbs of data

either way i think this is a great improvement on current, so lgtm - hopefully it address the size issue

[phlax]

lgtm, thanks

[lizan]

the idea of prebuilding the image with the binary is really cool, altho it still seems to be sending a huge context - im wondering if it is worth building the image with the binaries and then deleting them before building the rest or similar. I just think it would speed it up quite a bit not copying out gbs of data

Looking at the log I don't think BuildKit sends duplicate context, we have {envoy,envoy-contrib}-{release,debug}-{amd64,arm64} combination and it seems only sends them once each. For disk space it now ending the job with 19GB available vs 9.6GB available before this PR, the job starting with 34GB.

[PiotrSikora]

Thanks!