[WIP] Booting/testing something in CI

[ijc]

Very incomplete right now, will work on it as time allows.

[ijc]

This is hitting issues with the linuxkit build getting randomly SIGKILL'd, I think it is getting squashed by the OOM killer and raised moby/tool#191 to reduce the memory overheads. I'm also experimenting with the resource_class to see if that helps in the meantime (as a short term band-aid, although it looks like it will become a paid only feature before too long).

[ijc]

Early indications are that medium+ (3 CPU 6GB RAM) is insufficient while large (4 CPU 8GB RAM) is enough.

In moby/tool#191 I observed the initial RAM usage before my changes was 6.7GB which seems consistent with getting SITH on a 6GB limit. I also noted that tar used more like 2GB, which explains why they mostly work since the default medium has 4GB RAM, although they do still fail occasionally so perhaps there can be spikes or differences around other factors like content trust.

[GordonTheTurtle]

Please sign your commits following these rules:
https://github.com/moby/moby/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:

$ git clone -b "ci-boot-something" [email protected]:ijc/linuxkit-kubernetes.git somewhere
$ cd somewhere
$ git rebase -i HEAD~842354248512
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

[ijc]

We have no MacOS workers in circle (seems it is a paid only feature) so the jobs just hang forever waiting to be assigned.

It's been suggested we could try using the same gcp account as the linuxkit/linuxkit CI and use linuxkit run gcp. Need to take care not to leak VMs though and to thoroughly cleanup.

[errordeveloper]

It's been suggested we could try using the same gcp account as the linuxkit/linuxkit CI and use linuxkit run gcp. Need to take care not to leak VMs though and to thoroughly cleanup.

This makes sense. The only concern that I have is how long would it take to upload a build from Circle to GCP... We would also have to cleanup image, as I'm pretty sure they charge for storing them. The alternative would be to build and run on a VM that supports nested virt. IIRC there is a new instance type GCP that supports nested virt.

[ijc]

@rn is experimenting with the nested virt stuff on the main linuxkit repo now, will wait and see how he gets on.

It might indeed be too much data to upload for each PR, may turn out better to deploy a VM to build the images whether we then test them nested or in a new VM altogether.

[errordeveloper]

Also, if use nested virt, it means someone can reproduce the tests locally very easily, we could even provide a generalised script for folks to re-use in private LinuxKit projects. Just reiterating what I've said earlier...

[rn]

linuxkit/linuxkit#2871 has the right runes for enabling nested virt. The image needs to have a "special license" (some string in the Licenses field) and the instance needs to be Haswell or newer.