chore(CI): Add QEMU based CI Workflow #986

Elbehery · 2025-06-12T12:01:33Z

Fixes #665

cc @ahrtr @ivanvc

ivanvc

Thanks for the pull request, @Elbehery. I left some comments :)

.github/workflows/cross-arch-test.yaml

Elbehery · 2025-06-12T21:25:57Z

/retest

Elbehery · 2025-06-12T21:26:36Z

the failure was due to server error in GH workflow

Run golangci/golangci-lint-action@4afd733a84b1f43292c63897423277bb7f4313a9
prepare environment
Error: Failed to run: Error: Unexpected HTTP response: 500, Error: Unexpected HTTP response: 500
    at /home/runner/work/_actions/golangci/golangci-lint-action/4afd733a84b1f43292c63897423277bb7f4313a9/dist/run/index.js:7965:25
    at Generator.next (<anonymous>)
    at fulfilled (/home/runner/work/_actions/golangci/golangci-lint-action/4afd733a84b1f43292c63897423277bb7f4313a9/dist/run/index.js:7878:58)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Error: Unexpected HTTP response: 500

Elbehery · 2025-06-12T21:37:44Z

/retest

Elbehery · 2025-06-12T21:37:59Z

failure is new

--- FAIL: TestTx_Rollback_Freelist (0.00s)
    btesting.go:54: Opening bbolt DB at: /tmp/TestTx_Rollback_Freelist3715326428/001/db
    db_failpoint_test.go:278: Populate some data to have at least 5 leaf pages.
    db_failpoint_test.go:302: Remove some keys to have at least 3 more free pages.
    db_failpoint_test.go:314: Close and then reopen the db to release all pending free pages.
    btesting.go:96: Closing bbolt DB at: /tmp/TestTx_Rollback_Freelist3715326428/001/db
    btesting.go:126: Reopening bbolt DB at: /tmp/TestTx_Rollback_Freelist3715326428/001/db
    db_failpoint_test.go:318: Enable the `beforeWriteMetaError` failpoint.
    db_failpoint_test.go:319: 
        	Error Trace:	/home/runner/work/bbolt/bbolt/tests/failpoint/db_failpoint_test.go:319
        	Error:      	Received unexpected error:
        	            	failpoint: failpoint does not exist
        	Test:       	TestTx_Rollback_Freelist
    btesting.go:96: Closing bbolt DB at: /tmp/TestTx_Rollback_Freelist3715326428/001/db
FAIL
FAIL	go.etcd.io/bbolt/tests/failpoint	0.022s
ok  	go.etcd.io/bbolt/tests/robustness	0.004s
?   	go.etcd.io/bbolt/tests/utils	[no test files]
?   	go.etcd.io/bbolt/version	[no test files]
FAIL
Error: Process completed with exit code 1.

Elbehery · 2025-06-12T21:50:23Z

@ahrtr hello

TestTx_Rollback_Freelist is consistently failing, also tried running it locally, and it fails !!!

--- FAIL: TestTx_Rollback_Freelist (0.00s)
    btesting.go:54: Opening bbolt DB at: /tmp/TestTx_Rollback_Freelist749001970/001/db
    db_failpoint_test.go:278: Populate some data to have at least 5 leaf pages.
    db_failpoint_test.go:302: Remove some keys to have at least 3 more free pages.
    db_failpoint_test.go:314: Close and then reopen the db to release all pending free pages.
    btesting.go:96: Closing bbolt DB at: /tmp/TestTx_Rollback_Freelist749001970/001/db
    btesting.go:126: Reopening bbolt DB at: /tmp/TestTx_Rollback_Freelist749001970/001/db
    db_failpoint_test.go:318: Enable the `beforeWriteMetaError` failpoint.
    db_failpoint_test.go:319: 
        	Error Trace:	/home/runner/work/bbolt/bbolt/tests/failpoint/db_failpoint_test.go:319
        	Error:      	Received unexpected error:
        	            	failpoint: failpoint does not exist
        	Test:       	TestTx_Rollback_Freelist
    btesting.go:96: Closing bbolt DB at: /tmp/TestTx_Rollback_Freelist749001970/001/db
FAIL
FAIL	go.etcd.io/bbolt/tests/failpoint	0.013s
ok  	go.etcd.io/bbolt/tests/robustness	0.004s
?   	go.etcd.io/bbolt/tests/utils	[no test files]
?   	go.etcd.io/bbolt/version	[no test files]
FAIL
Error: Process completed with exit code 1.

I think #973 is the only possible cause, please correct me if i am wrong

cc @ivanvc @fuweid @tjungblu

fuweid · 2025-06-12T21:53:25Z

@Elbehery some cases need to build with failpoint so I think we need to run make gofail-enable first

Elbehery · 2025-06-12T21:58:49Z

@Elbehery some cases need to build with failpoint so I think we need to run make gofail-enable first

used

 make gofail-enable && make test

hope this helps

.github/workflows/cross-arch-test.yaml

Elbehery · 2025-06-13T09:11:59Z

@ahrtr hello ✋🏽

following up on your comment

I could add the platforms to the workflow, but I found the following notes on them

Notes:

    aix/ppc64: Supported by Go, but note that AIX binaries are not commonly run/tested on CI environments like GitHub Actions. It may compile but won’t be runnable in the GitHub runner.

    android/arm64: Go supports Android cross-compilation, but compiling for Android may require additional NDK setup for complex use cases.

    solaris/amd64: Also supported by Go, but not natively testable on GitHub runners.

please let me know

cc @ivanvc @fuweid

.github/workflows/cross-arch-test.yaml

ahrtr · 2025-06-13T12:49:11Z

I could add the platforms to the workflow, but I found the following notes on them

Notes:

    aix/ppc64: Supported by Go, but note that AIX binaries are not commonly run/tested on CI environments like GitHub Actions. It may compile but won’t be runnable in the GitHub runner.

    android/arm64: Go supports Android cross-compilation, but compiling for Android may require additional NDK setup for complex use cases.

    solaris/amd64: Also supported by Go, but not natively testable on GitHub runners.

please let me know

My understanding is that it's exactly the reason why we need to use qemu. BTW, where did you see the notes?

Elbehery · 2025-06-13T14:07:39Z

I could add the platforms to the workflow, but I found the following notes on them
Notes:

    aix/ppc64: Supported by Go, but note that AIX binaries are not commonly run/tested on CI environments like GitHub Actions. It may compile but won’t be runnable in the GitHub runner.

    android/arm64: Go supports Android cross-compilation, but compiling for Android may require additional NDK setup for complex use cases.

    solaris/amd64: Also supported by Go, but not natively testable on GitHub runners.
please let me know
My understanding is that it's exactly the reason why we need to use qemu. BTW, where did you see the notes?

I used AI search, is it ok ? 🤔

Elbehery · 2025-06-13T14:17:14Z

/retest

ivanvc · 2025-06-13T17:47:33Z

If we can't run tests for some platforms (i.e., Android, Solaris, AIX). Could we at least cross-compile? If I understand correctly, errors like #988 should be caught at compile time.

Elbehery · 2025-06-13T18:28:44Z

If we can't run tests for some platforms (i.e., Android, Solaris, AIX). Could we at least cross-compile? If I understand correctly, errors like #988 should be caught at compile time.

yes indeed, it will compile, I am trying to fix the CI and then will add the 3 platforms in #988

Elbehery · 2025-06-13T19:09:57Z

the failing test is due to gofail not found

Run GOOS=linux GOARCH=mips64 make gofail-enable && make test
  GOOS=linux GOARCH=mips64 make gofail-enable && make test
  shell: /usr/bin/bash -e {0}
go install go.etcd.io/gofail
gofail enable .
make: gofail: No such file or directory
make: *** [Makefile:75: gofail-enable] Error 127
Error: Process completed with exit code 2.

I am adding a commit to use explicit path for gofail binary to be used within other targets 🙏🏽

.github/workflows/cross-arch-test.yaml

Elbehery · 2025-06-18T18:46:07Z

/retest

ivanvc

Thanks for the updates, @Elbehery.

I think this could be simplified further if we add a new template that receives the ARCHs and the OS. I can also improve this in a follow-up PR if it's not clear how to implement this.

Elbehery · 2025-06-19T06:33:42Z

Thanks for the updates, @Elbehery.

I think this could be simplified further if we add a new template that receives the ARCHs and the OS. I can also improve this in a follow-up PR if it's not clear how to implement this.

lgtm 👍🏽

Signed-off-by: Mustafa Elbehery <[email protected]>

.github/workflows/cross-arch-test.yaml

ivanvc

LGTM. Let's merge, and I'll improve this workflow in a follow-up.

ivanvc · 2025-06-20T14:20:15Z

/cc @ahrtr

ahrtr

LGTM

Thanks @Elbehery

k8s-ci-robot · 2025-06-20T15:32:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahrtr, Elbehery, ivanvc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ahrtr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added do-not-merge/work-in-progress size/M labels Jun 12, 2025

ivanvc reviewed Jun 12, 2025

View reviewed changes

Elbehery force-pushed the 20250612-use-qemu-github-workflow branch 2 times, most recently from 383220b to 5b3b4b4 Compare June 12, 2025 18:54

Elbehery force-pushed the 20250612-use-qemu-github-workflow branch from 5b3b4b4 to 9431051 Compare June 12, 2025 21:58

Elbehery changed the title ~~WIP: chore(CI): Add QEMU to CI Workflow~~ chore(CI): Add QEMU to CI Workflow Jun 12, 2025

k8s-ci-robot removed the do-not-merge/work-in-progress label Jun 12, 2025

Elbehery changed the title ~~chore(CI): Add QEMU to CI Workflow~~ chore(CI): Add QEMU based CI Workflow Jun 12, 2025

fuweid reviewed Jun 13, 2025

View reviewed changes

.github/workflows/cross-arch-test.yaml Outdated Show resolved Hide resolved

Elbehery force-pushed the 20250612-use-qemu-github-workflow branch from 9431051 to 0956a37 Compare June 13, 2025 06:17

ahrtr reviewed Jun 13, 2025

View reviewed changes

.github/workflows/cross-arch-test.yaml Outdated Show resolved Hide resolved

.github/workflows/cross-arch-test.yaml Outdated Show resolved Hide resolved

Elbehery force-pushed the 20250612-use-qemu-github-workflow branch from 0956a37 to 9ba6bae Compare June 13, 2025 14:09

Elbehery force-pushed the 20250612-use-qemu-github-workflow branch 3 times, most recently from df6ae3e to 6efa964 Compare June 13, 2025 19:27

Elbehery force-pushed the 20250612-use-qemu-github-workflow branch 9 times, most recently from 12c4527 to 42e181b Compare June 18, 2025 18:12

Elbehery commented Jun 18, 2025

View reviewed changes

.github/workflows/cross-arch-test.yaml Show resolved Hide resolved

Elbehery force-pushed the 20250612-use-qemu-github-workflow branch from 22a726f to df9a267 Compare June 18, 2025 19:06

ivanvc reviewed Jun 19, 2025

View reviewed changes

ivanvc mentioned this pull request Jun 19, 2025

chore(CI): add goimports to linter #993

Merged

chore(CI): Add QEMU to CI Workflow

8723401

Signed-off-by: Mustafa Elbehery <[email protected]>

Elbehery force-pushed the 20250612-use-qemu-github-workflow branch from df9a267 to 8723401 Compare June 19, 2025 09:13

Elbehery commented Jun 19, 2025

View reviewed changes

.github/workflows/cross-arch-test.yaml Show resolved Hide resolved

ivanvc approved these changes Jun 20, 2025

View reviewed changes

k8s-ci-robot requested a review from ahrtr June 20, 2025 14:20

ahrtr approved these changes Jun 20, 2025

View reviewed changes

k8s-ci-robot added the approved label Jun 20, 2025

ahrtr merged commit be68cc2 into etcd-io:main Jun 20, 2025
43 checks passed

Elbehery deleted the 20250612-use-qemu-github-workflow branch June 20, 2025 15:33

ivanvc mentioned this pull request Jun 24, 2025

Add template to cross arch build tests #999

Merged

This was referenced Jun 25, 2025

[release-1.4] Add QEMU based CI workflow #1000

Merged

[release-1.3] Add QEMU based CI workflow #1001

Merged

ivanvc mentioned this pull request Jun 25, 2025

Introduce qemu to verify all the existing platforms #665

Open

chore(CI): Add QEMU based CI Workflow #986

chore(CI): Add QEMU based CI Workflow #986

Uh oh!

Conversation

Elbehery commented Jun 12, 2025

Uh oh!

ivanvc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Elbehery commented Jun 12, 2025

Uh oh!

Elbehery commented Jun 12, 2025

Uh oh!

Elbehery commented Jun 12, 2025

Uh oh!

Elbehery commented Jun 12, 2025

Uh oh!

Elbehery commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fuweid commented Jun 12, 2025

Uh oh!

Elbehery commented Jun 12, 2025

Uh oh!

Uh oh!

Elbehery commented Jun 13, 2025

Uh oh!

Uh oh!

Uh oh!

ahrtr commented Jun 13, 2025

Uh oh!

Elbehery commented Jun 13, 2025

Uh oh!

Elbehery commented Jun 13, 2025

Uh oh!

ivanvc commented Jun 13, 2025

Uh oh!

Elbehery commented Jun 13, 2025

Uh oh!

Elbehery commented Jun 13, 2025

Uh oh!

Uh oh!

Elbehery commented Jun 18, 2025

Uh oh!

ivanvc left a comment

Choose a reason for hiding this comment

Uh oh!

Elbehery commented Jun 19, 2025

Uh oh!

Uh oh!

ivanvc left a comment

Choose a reason for hiding this comment

Uh oh!

ivanvc commented Jun 20, 2025

Uh oh!

ahrtr left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Jun 20, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

Elbehery commented Jun 12, 2025 •

edited

Loading