Publish multi-platform Docker from `develop`

[hugoclrd]

Have a GitHub workflow (triggered automatically or manually) that can build the multi-platform (arm64, amd64) Docker stacks-blockchain image from the develop branch.

The image can be hosted on GitHub image repository.

The tag could be something like next or preview or develop.

It would allow builders to test and preview new features locally. This will be especially useful to deploy Clarity 4 smart contracts in Clarinet Devnet

[wileyj]

we currently have most of what's requested here with https://github.com/stacks-network/stacks-core/blob/develop/.github/workflows/image-build-source.yml

it will need to be updated for the multi-arch image (unknown how much extra time it would take to compile), and a few other qol updates.

i'm hesitant to hardcode the tag though - today we use the branch name it's built from, and i think taht would be fine per your request. if building the image from develop, the image would be ../stacks-core:develop, if built from master -> ..stacks-core:master etc.

[hugoclrd]

Any update on this one @wileyj?
When do you think this could be ready?

[wileyj]

Any update on this one @wileyj? When do you think this could be ready?

I was able to get this working with a 1 line change, but i'm not impressed with the time it takes to perform what's being asked here. https://github.com/wileyj/stacks-core/actions/runs/18476617916/job/52643304933

it's run for 50+ minutes so far. there is another idea to possible speed this up, if we use the method we have for releases - in a matrix, we build the binaries separately and produce an artifact. then, the docker portion simply pulls in that artifact, extracts the binaries, and then pushes the multi-arch image.

timing on that should be closer to 30 minutes. in the above test, adding more cpu to the runner would help - but not that much. the second option would take longer to develop, but would result in faster builds.

[wileyj]

Any update on this one @wileyj? When do you think this could be ready?

2+ hours now. i don't think is a good option unless the idea is a nightly image?

[hugoclrd]

I agree this is kind of a lot (what takes so long though 😵)
As far as I'm concerned, the time it takes doesn't matter much, it's not something I've been needed in the past that much;

So, for instance, right now I'd like to be able to have this build available for builder to test Clarity 4 with the latest changes on develop, but I probably won't need it for another week or two.

That being said, once it's available, I may be tempted to use it more and more

[wileyj]

ok - i can PR this change, and we can see how well it holds up (i expect you'll see the wrinkles too!).
that said, the work to make it better aligns with what i was looking at for release builds - so i may update it to that other method regardless.

[hugoclrd]

Thanks.
Yeah so I guess the docker build just takes a very long time. And you can't really cache the dependencies in the scenario right?

[wileyj]

Thanks. Yeah so I guess the docker build just takes a very long time. And you can't really cache the dependencies in the scenario right?

That was something we looked at for a different build step - at the time, it simply wasn't possible. but it may be an option now that we could go over the allotted cache storage for a higher expense.
it would also be more complex to use that cache inside of docker (where the builds happen now), and there would need to be 2 copies (1 for each arch) to benefit here.

i don't think it's worth the work involved - i think the better option is we build the binaries from source on the runner VM, store the archive of the binaries, then a docker build step that downloads and uses that binary archive.

[hugoclrd]

it would also be more complex to use that cache inside of docker (where the builds happen now)

Yeah that's what I was thinking.

I think the better option is we build the binaries from source on the runner VM, store the archive of the binaries, then a docker build step that downloads and uses that binary archive.

👌 That sounds like a better plan indeed.

Well, thanks for opening a PR with the slow way in the meantime 🙏

[wileyj]

there is one other idea we could pursue if you'd like (i think it may reduce the time taken, but i don't know by how much).
if we build the image proposed here using the release-lite profile, it may reduce build time - locally i've seen around a 40% reduction in time to build but impossible to say on a smaller machine.