ci: sporadic failures "Exit 127"

[mudler]

cos-toolkit version:
Not relevant

CPU architecture, OS, and Version:

Describe the bug
Sporadically CI fails with ERRO[0000] fatal error: exit status 127 or more generically with exit status 127 in different places. This is happening very frequently, and becomes more noticeable now

To Reproduce
Happened to me locally too (only once). My guess is a simple looping on the binaries generated with UPX will eventually fail with the above

Expected behavior
To not fail

Logs

Additional context
I suspect it's due to part of our binaries (luet and extensions) compressed with UPX. See also upstream issues about it: upx/upx#385 upx/upx#224 upx/upx#225 .

I'll use this issue to track all the occurences

[mudler]

https://github.com/rancher-sandbox/cOS-toolkit/runs/2959979887
https://github.com/rancher-sandbox/cOS-toolkit/runs/2956146346?check_suite_focus=true

[davidcassany]

I suspect it's due to part of our binaries (luet and extensions) compressed with UPX. See also upstream issues about it:

If so I'd suggest to add a toggle to use UPX or not and then just give it a try for some time without. On CI this happens quite often, I mostly saw it on make deps step. If the issue disappears without it I'd say we have to forget about UPX until it gets fixed upstream.

[mudler]

I suspect it's due to part of our binaries (luet and extensions) compressed with UPX. See also upstream issues about it:

If so I'd suggest to add a toggle to use UPX or not and then just give it a try for some time without. On CI this happens quite often, I mostly saw it on make deps step. If the issue disappears without it I'd say we have to forget about UPX until it gets fixed upstream.

I do agree, but I would give a deeper look as we could also try to downgrade upx/upx#225 (comment) or try to use a different upx option, I think --brute is causing issues

[mudler]

I think we can close this for now and re-open it if needed, doesn't seems happening anymore after disabling upx