workflow to build container images #190

akostadinov · 2023-07-07T20:25:31Z

builds a multiarch (amd64 and arm64, you can add more if you want) image and pushes to ghcr.io

fixes #154

vi · 2023-07-08T07:23:34Z

Is it adapted for Websocat repository or just a template of a workflow that I should customize to be useful for the project?

(I haven't used Github Workflows yet)

akostadinov · 2023-07-08T07:48:50Z

Should already work as it uses variables that should already exist in all workflows. Unfortunately can't test it in your repo without being merged. If it fails after merging, I will look at fixing it.

Also it is possible to add a simple smoke test to make sure image is correct. But didn't want to spend time on it before seeing this merged. It wont be hard to add as I have an example I can adapt.

P.S. added some comments to highlight some points in the workflow definition

akostadinov · 2023-07-08T07:50:02Z

.github/workflows/container-image-buildah.yml

+      - name: Push To Container Registry
+        id: push-to-container-registry
+        uses: redhat-actions/push-to-registry@v2
+        if: github.event_name != 'pull_request'


will push image to ghcr.io only if the trigger was NOT a pull request

akostadinov · 2023-07-08T07:50:47Z

.github/workflows/container-image-buildah.yml

+          tags: |
+            type=schedule
+            type=raw,value=latest,enable=${{ github.ref_name == 'main' }}
+            ${{ github.ref_name == 'main' && 'type=raw,value=nightly' || 'type=ref,event=branch' }}


nightly and latest tags will be added if we build for the main branch.

Otherwise the branch name will be used. But trigger is configured to only respect the main branch so there should be no other branches that would be built (unless the trigger section is changed).

There is no "main" branch (as "master" was the default when Websocat was created), so it should probably be edited.

oh, I missed that, updated

akostadinov · 2023-07-08T07:53:11Z

.github/workflows/container-image-buildah.yml

+  push:
+    branches: [ main ]
+    tags:
+      - '*'           # Push events to every tag not containing /


an image will be built and pushed to ghcr.io for all tags that do not contain / in their name

Will existing, manually pushed images to gchr.io conflict with those?

If you build from a tag/branch that already has an image pushed in whatever way, the newly image will override the old image.

If you ask about the existing 2 images, unless you change the existing tags or trigger a build manually, then no new builds will automatically be created.

Does this answer your question?

akostadinov · 2023-07-08T07:53:51Z

.github/workflows/container-image-buildah.yml

+name: Container Image
+
+on:
+  workflow_dispatch: {}


This allows triggering a build manually for any branch/tag if so needed/desired.

akostadinov · 2023-07-08T07:54:40Z

.github/workflows/container-image-buildah.yml

+    tags:
+      - '*'           # Push events to every tag not containing /
+  pull_request:
+    types: [opened, reopened, synchronize]


image will be built for all pull requests but will not be pushed to registry, see the push step below

akostadinov · 2023-07-08T07:55:48Z

.github/workflows/container-image-buildah.yml

+
+concurrency:
+  group: ci-container-build-${{ github.ref }}-1
+  cancel-in-progress: true


if a new image build is triggered for the same reference (e.g. branch) as a build already in progress, then the older build will be canceled

vi · 2023-07-08T08:19:43Z

.github/workflows/container-image-buildah.yml

+  workflow_call: {}
+  schedule:
+    # every Wednesday morning
+    - cron: 7 7 * * 3


Will it keep building the same version again and again if there are is no activity for weeks?

Yes, it can be disabled. But I find it useful to build with updated base image potentially fixing security issues.

And making sure that image sill builds with latest base image.

vi · 2023-07-08T08:21:52Z

Unfortunately can't test it in your repo without being merged.

Do I need to configure credentials or do something else in Github web interface?

E.g. back in travis-ci.org days I needed to go to Travis and enable repository there for it to work.

vi · 2023-07-08T08:24:01Z

Where Cargo features for building are getting set?

In my release script different platforms (and editions e.g. "nossl") sometimes have different compile-time flags.

akostadinov · 2023-07-08T08:39:15Z

Do I need to configure credentials or do something else in Github web interface?

No, the permissions section of the file should take care of this. Temporary github actions credentials are being used so no need to create such statically. See secrets.GITHUB_TOKEN, that's generated per run automatically.

Where Cargo features for building are getting set?

I have zero idea what is cargo and how websocat is being built. How it is built depends on the Dockerfile content and I haven't modified that.

akostadinov · 2023-07-09T10:48:56Z

Actually the build ran in my fork but failed. Seems like a compilation issue. Seems like Dockerfile needs to be fixed somehow. See:

https://github.com/akostadinov/websocat/actions/runs/5499402760/jobs/10021569027

vi · 2023-07-09T13:20:23Z

~~It is building for musl (as specified in Dockerfile), but musl target is not installed in building environment.~~

~~I suppose it ignores the FROM rust:1.60-alpine3.15 line from Dockerfile?~~

Looks like the first build (for amd64) succeeds, but the second one fails, as target architecture is currently hard coded into the Dockerfile.

akostadinov · 2023-07-09T16:36:26Z

Ok, I disabled arm64 for the time being. It can easily be re-enabled. What worries me though is that the image I built locally segfaults.

$ podman run --rm -ti --entrypoint /bin/sh localhost/websocat-test:latest 
/ # websocat ws://ws.vi-server.org/mirror
Segmentation fault (core dumped)

This is same as ghcr.io/vi/websocat:0.11.0 ws://ws.vi-server.org/mirror

akostadinov · 2023-07-09T20:03:40Z

Ok, now I disabled arm64 build as it seems the Dockerfile will need some adjustments to support other archs. Can be easily re-enabled once this is done. Now build succeeds:

https://github.com/akostadinov/websocat/actions/runs/5500873352

So it can be merged and Dockerfile can be fixed later.

camaeel · 2023-10-10T12:11:02Z

.github/workflows/container-image-buildah.yml

+        uses: redhat-actions/buildah-build@v2
+        with:
+          tags: ${{ steps.meta.outputs.tags }}
+          platforms: linux/amd64


Why not add here 'linux/arm64' platform as well ?

See my last comment. Some adjustments will need to be done for other archs in the Dockerfile. But there is no point to investigate before the x86 segfault is fixed and this thing merged. Or the other way around:

merge this

fix dockerfile to build websocat properly

enable arm64

I know nothing about websocat though, I just wanted to help with github actions. Somebody else interested has to fix the Dockerfile as I don't have time for investigating the project build system.

@akostadinov Makes sense to first make it build and then add arm. I was not aware of Dockerfile changes needed. Sorry.

vi · 2023-10-10T17:18:52Z

Note that Dockerfile has been changed in 1.12 is probably (not checked) more portable.

I don't know how musl-based images work on non-amd64 though.

akostadinov · 2023-10-10T19:27:04Z

@vi , didn't notice. The new Dockerfile does work. You can try out the build from my fork:

$ podman run -i --rm ghcr.io/akostadinov/websocat:nightly ws://ws.vi-server.org/mirror <<< hohoho
hohoho

So this can be merged to have automated builds. I will prepare a couple more PRs afterwards.

vi · 2023-10-10T20:34:46Z

I should probably (someday?) educate myself about this whole Github CI thing to understand what does this pull request mean and how to use/verify it. Maybe starting with some small repository.

I remember using "travis-ci.org" for checking commits and tags. Maybe it's time to revisit the idea of non-localhost builds.

Does this pull request provide some advantage for Websocat users (i.e. they can install or use it in more ways than before or with more convenience) or is it just some development aid? Compared to e.g. just documenting how to use it with Github Workflows.

akostadinov · 2023-10-11T09:13:09Z

@vi , this enables automatic builds of websocat container image upon:

a schedule
master branch push
creating a tag
creating a pull request

It will also push this container image to the ghcr.io container registry so users can just pull image and use it in the case of:

master branch push
tag created

So it is an easy way for the users to just pull an image and use websocat instead of building locally or relying on their software vendor to provide a package for their OS.

If you merge this, one of the things I will add would be a simple test to the workflow that the image works. This is to prevent situations like before where websocat in the container image segfaulted. And add arm64 as we spoke previously.

akostadinov · 2023-10-11T09:14:40Z

Also image building can be added to the PR checks. This would be useful to avoid PRs that break container image builds. It can be done from project settings. It's not something to add in the workflow body.

vi · 2023-10-11T12:22:58Z

It will also push this container image to the ghcr.io container registry so users can just pull image and use it ... tag created

How does it interact with images I push manually, e.g. this one?

akostadinov · 2023-10-11T13:12:37Z

It will tag with the git tag name. If you have manually pushed an image with the same tag, it will be overwritten. If you push a manually built image after the workflow has run, apparently you will overwrite the image from the workflow.

But this workflow action should make it unnecessary to build and push official images manually.

vi · 2023-10-11T14:13:07Z

Is it practical to try to attain reproducible builds, so that local and CI builds would have the same hashes (hence, I suppose, the same container identifiers)?

akostadinov · 2023-10-11T14:44:08Z

Reproducible builds is to me something complicated with little to no benefits for an open source project. I don't know how to implement reproducible builds without a dedicated infrastructure to host all dependencies and build within environment without Internet access but only things from the cached dependencies being accessible.

Additionally to get exactly the same hash, the build should be void of any non-deterministic operation. And then still you would have different file creation times and layer build times which will still affect the final hash.

While I believe everything in this world if possible, I don't think this is feasible or even beneficial.

akostadinov mentioned this pull request Jul 7, 2023

Hope to provide the official docker image #154

Closed

akostadinov commented Jul 8, 2023

View reviewed changes

vi reviewed Jul 8, 2023

View reviewed changes

akostadinov force-pushed the master branch from acb25df to 8191b88 Compare July 9, 2023 10:37

camaeel reviewed Oct 10, 2023

View reviewed changes

akostadinov added 2 commits October 10, 2023 22:17

workflow to build container images

17bb292

disable arm64 until it is supported by Dockerfile

30083f9

akostadinov force-pushed the master branch from 060149f to 30083f9 Compare October 10, 2023 19:18

vi force-pushed the master branch from 845840c to ecb783f Compare October 15, 2023 21:26

vi merged commit 1ad7a89 into vi:master Mar 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

workflow to build container images #190

workflow to build container images #190

akostadinov commented Jul 7, 2023

vi commented Jul 8, 2023

akostadinov commented Jul 8, 2023 •

edited

Loading

akostadinov Jul 8, 2023

akostadinov Jul 8, 2023 •

edited

Loading

vi Jul 8, 2023

akostadinov Jul 9, 2023

akostadinov Jul 8, 2023

vi Jul 8, 2023

akostadinov Jul 9, 2023

akostadinov Jul 8, 2023

akostadinov Jul 8, 2023

akostadinov Jul 8, 2023

vi Jul 8, 2023

akostadinov Jul 9, 2023

akostadinov Jul 9, 2023

vi commented Jul 8, 2023

vi commented Jul 8, 2023 •

edited

Loading

akostadinov commented Jul 8, 2023

akostadinov commented Jul 9, 2023

vi commented Jul 9, 2023 •

edited

Loading

akostadinov commented Jul 9, 2023

akostadinov commented Jul 9, 2023

camaeel Oct 10, 2023

akostadinov Oct 10, 2023

camaeel Oct 10, 2023 •

edited

Loading

vi commented Oct 10, 2023 •

edited

Loading

akostadinov commented Oct 10, 2023

vi commented Oct 10, 2023 •

edited

Loading

akostadinov commented Oct 11, 2023

akostadinov commented Oct 11, 2023

vi commented Oct 11, 2023

akostadinov commented Oct 11, 2023

vi commented Oct 11, 2023 •

edited

Loading

akostadinov commented Oct 11, 2023

workflow to build container images #190

workflow to build container images #190

Conversation

akostadinov commented Jul 7, 2023

vi commented Jul 8, 2023

akostadinov commented Jul 8, 2023 • edited Loading

Choose a reason for hiding this comment

akostadinov Jul 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vi commented Jul 8, 2023

vi commented Jul 8, 2023 • edited Loading

akostadinov commented Jul 8, 2023

akostadinov commented Jul 9, 2023

vi commented Jul 9, 2023 • edited Loading

akostadinov commented Jul 9, 2023

akostadinov commented Jul 9, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

camaeel Oct 10, 2023 • edited Loading

Choose a reason for hiding this comment

vi commented Oct 10, 2023 • edited Loading

akostadinov commented Oct 10, 2023

vi commented Oct 10, 2023 • edited Loading

akostadinov commented Oct 11, 2023

akostadinov commented Oct 11, 2023

vi commented Oct 11, 2023

akostadinov commented Oct 11, 2023

vi commented Oct 11, 2023 • edited Loading

akostadinov commented Oct 11, 2023

akostadinov commented Jul 8, 2023 •

edited

Loading

akostadinov Jul 8, 2023 •

edited

Loading

vi commented Jul 8, 2023 •

edited

Loading

vi commented Jul 9, 2023 •

edited

Loading

camaeel Oct 10, 2023 •

edited

Loading

vi commented Oct 10, 2023 •

edited

Loading

vi commented Oct 10, 2023 •

edited

Loading

vi commented Oct 11, 2023 •

edited

Loading