BUG: look into replacing Travis CI with GitHub actions

[pcmoore]

There are plenty of articles on this, but see The Register article below for some background on this issue:

• https://www.theregister.com/2020/11/02/travis_ci_pricng

With Travis CI becoming potentially unreliable for libseccomp, we should investigate moving out CI testing to GitHub actions:

• https://docs.github.com/en/free-pro-team@latest/actions

[drakenclimber]

Here's a guide for migrating from TravisCI to Github Actions. I am hoping to experiment with this over the next few days
https://docs.github.com/en/free-pro-team@latest/actions/learn-github-actions/migrating-from-travis-ci-to-github-actions

[drakenclimber]

I'm really liking the look of Github Actions. I just sent out a patchset to switch libcgroup from Travis CI to Github Actions.

https://github.com/drakenclimber/libcgroup/tree/issues/github_actions
https://sourceforge.net/p/libcg/mailman/message/37165461/

I'll try to put together a patchset to transition libseccomp this week.

[pcmoore]

That sounds good @drakenclimber, thanks!

[drakenclimber]

Here's my current status. I have Github Actions working on amd64 with some slight deviations from our current Travis CI solution, but I have concerns about other architectures.

Pros:

• Easy to use. I was able to quickly setup a continuous integration workflow in Github Actions
• It's also easy to encapsulate operations into an action. These actions can then be reused within the workflow. (Strangely an action cannot invoke another action.)
• Acceptable speed (for amd64 at least). The entire test suite completed in 22 minutes
• Good GUI and failing steps are clearly delineated
• I really like the parallelization options of Github Actions, and it provides a feature to collate results once other jobs complete. I am using this feature in libcgroup to finalize its code coverage numbers after all of the tests have run

Cons:

• No native support for other architectures
  • A github employee provided this workaround to run an arm64 Docker container. I tried it out, and with some fiddling I was able to get the basic test suite mostly working. (Test 52 - basic load - failed for some odd reason.) But this setup is difficult to reproduce locally and debugging is a significant challenge. I wasn't able to get the python tests working. Oh, and it's really, really slow - it took an hour to run a pared down set of tests on arm64. ppc64el took 6 hours before I killed it :(
  • A github user created this action that also uses Docker containers to run non-native architectures. The syntax is somewhat clunky and would require us to duplicate code. Its supported architecture and platform list is pretty good, though
  • Github Actions supports using self-hosted runners, so one (ugly) option would be to find a dedicated arm64, ppc64le, etc. box and use it to run the tests. The advantage of this is that debugging would be much more straightforward than a Docker container

Other:

• Coveralls.io has created a Github Action. Our existing TravisCI solution, cpp-coveralls, does work on Github Actions but it struggled with parallel builds. I have the Coveralls action working in parallel in libcgroup
• To utilize Coveralls' action, I had to invoke lcov directly to generate an lcov.info file. This file is then uploaded to coveralls.io. Note that using lcov directly produces slightly different coverage numbers. For example, the Travis CI solution doesn't specify whether this line was covered or not while the Github Actions solution explicitly calls it uncovered. I believe Github Actions is correct in this case and our current solution is misreporting its coverage number slightly
• Note to self - I don't have the --exclude src/arch-syscall-check.c lcov flag working. No clue why

[oxr463]

Here's my current status. I have Github Actions working on amd64 with some slight deviations from our current Travis CI solution, but I have concerns about other architectures.

Pros:

• Easy to use. I was able to quickly setup a continuous integration workflow in Github Actions
• It's also easy to encapsulate operations into an action. These actions can then be reused within the workflow. (Strangely an action cannot invoke another action.)
• Acceptable speed (for amd64 at least). The entire test suite completed in 22 minutes
• Good GUI and failing steps are clearly delineated
• I really like the parallelization options of Github Actions, and it provides a feature to collate results once other jobs complete. I am using this feature in libcgroup to finalize its code coverage numbers after all of the tests have run

Cons:

• No native support for other architectures

  • A github employee provided this workaround to run an arm64 Docker container. I tried it out, and with some fiddling I was able to get the basic test suite mostly working. (Test 52 - basic load - failed for some odd reason.) But this setup is difficult to reproduce locally and debugging is a significant challenge. I wasn't able to get the python tests working. Oh, and it's really, really slow - it took an hour to run a pared down set of tests on arm64. ppc64el took 6 hours before I killed it :(
  • A github user created this action that also uses Docker containers to run non-native architectures. The syntax is somewhat clunky and would require us to duplicate code. Its supported architecture and platform list is pretty good, though
  • Github Actions supports using self-hosted runners, so one (ugly) option would be to find a dedicated arm64, ppc64le, etc. box and use it to run the tests. The advantage of this is that debugging would be much more straightforward than a Docker container

Other:

• Coveralls.io has created a Github Action. Our existing TravisCI solution, cpp-coveralls, does work on Github Actions but it struggled with parallel builds. I have the Coveralls action working in parallel in libcgroup
• To utilize Coveralls' action, I had to invoke lcov directly to generate an lcov.info file. This file is then uploaded to coveralls.io. Note that using lcov directly produces slightly different coverage numbers. For example, the Travis CI solution doesn't specify whether this line was covered or not while the Github Actions solution explicitly calls it uncovered. I believe Github Actions is correct in this case and our current solution is misreporting its coverage number slightly
• Note to self - I don't have the --exclude src/arch-syscall-check.c lcov flag working. No clue why

What about using Vagrant on macOS?

[pcmoore]

What about using Vagrant on macOS?

This issue is specifically about moving our CI testing from Travis CI to GitHub Actions and not the general development and testing of libseccomp. I'm not sure MacOS is an option for GitHub Actions, and even if it was it would likely be a poor choice for us due to the lack of kernel support (our "live" tests are limited, but important).

[pcmoore]

Here's my current status. I have Github Actions working on amd64 with some slight deviations from our current Travis CI solution, but I have concerns about other architectures.

Thanks for looking into this @drakenclimber, the limited architecture support is very disapppointing. Since we aren't actually seeing a lot of problems with Travis CI at the moment, perhaps we continue with Travis until it becomes disruptive?

Regarding the lcov/Coveralls comments; I'd noticed similar things in the past, but didn't worry to much about the differences since they were minor. I wonder if it would be possible to use lcov in Travis and upload the lcov file as part of the Travis build without leaking any creds in the Travis config? If nothing else that would bring consistency across local and Travis use, and it might make things a little easier if/when we migrate way from Travis CI.

[oxr463]

What about using Vagrant on macOS?

This issue is specifically about moving our CI testing from Travis CI to GitHub Actions and not the general development and testing of libseccomp. I'm not sure MacOS is an option for GitHub Actions, and even if it was it would likely be a poor choice for us due to the lack of kernel support (our "live" tests are limited, but important).

I'm quite familiar with GitHub Actions. They do support macOS (See: https://github.com/actions/virtual-environments#available-environments). Specifically, macOS is the only environment that comes with Vagrant and VirtualBox (See: actions/runner-images#433).

In my experience, it requires a little more work to set up, but running inside a virtual machine ensures a more consistent environment for CI/CD pipelines. Not to mention, it is more portable, since anyone can run the Vagrant/VirtualBox images locally. It also makes migrating to a new CI/CD solution easier, since the configuration is typically written in a script, independent of the vendor-specific YAML declarations.

This is just my two cents :)

[pcmoore]

Thanks @oxr463, that's good to know about GH Actions. At this point I'd prefer not to have the extra management overhead of a virtual environment, but it is something to consider if Travis CI ever becomes a problem for us.

As our Travis activity is relatively light, I'm hopeful we wont run into the Travis problems that some other projects have seen.

[drakenclimber]

Thanks for looking into this @drakenclimber, the limited architecture support is very disapppointing. Since we aren't actually seeing a lot of problems with Travis CI at the moment, perhaps we continue with Travis until it becomes disruptive?

Yes, I think that's the best and safest answer.

Regarding the lcov/Coveralls comments; I'd noticed similar things in the past, but didn't worry to much about the differences since they were minor. I wonder if it would be possible to use lcov in Travis and upload the lcov file as part of the Travis build without leaking any creds in the Travis config? If nothing else that would bring consistency across local and Travis use, and it might make things a little easier if/when we migrate way from Travis CI.

Yes, this should be possible. I created Issue #309 and assigned it to myself. I'll try to pick this up in the next week or two.

Thanks

In my experience, it requires a little more work to set up, but running inside a virtual machine ensures a more consistent environment for CI/CD pipelines. Not to mention, it is more portable, since anyone can run the Vagrant/VirtualBox images locally. It also makes migrating to a new CI/CD solution easier, since the configuration is typically written in a script, independent of the vendor-specific YAML declarations.

Thanks, @oxr463. I also hope we don't have to go that route, but it's good to know that there are other options out there.

[pcmoore]

@drakenclimber I'm going to drop this from a release milestone and drop the priority down to low since we're adopting a "wait until it breaks" approach to this, if you disagree feel free to shout or simple adjust the labels accordingly.

[drakenclimber]

@drakenclimber I'm going to drop this from a release milestone and drop the priority down to low since we're adopting a "wait until it breaks" approach to this, if you disagree feel free to shout or simple adjust the labels accordingly.

Sounds good to me. Thanks!

[pcmoore]

@drakenclimber one thing did just occur to me - we should consider trying to migrate from travis-ci.org to travis-ci.com as the ".org" is supposedly going away "within weeks".

[drakenclimber]

Oh! Didn't know that. Thanks!

I'll try to pick this up next week then.

[pcmoore]

It looks like we can close this out now.