Configured Travis CI to run cookbooks via Test Kitchen#266
Configured Travis CI to run cookbooks via Test Kitchen#266migurski wants to merge 6 commits intoopenstreetmap:masterfrom
Conversation
With Kitchen-Dokken in place, Travis should now successfully run both the Cookstyle script and individual Kitchen tests as separate matrix builds. Matrix builds will run in parallel, and the global Travis 50min limit should apply to each separately ensuring that all cookbooks get tested. https://docs.travis-ci.com/user/build-matrix/
|
This rather overlaps with the work I've been going to try and run the tests under docker but as expected my experience is that not all of them can be run under docker so unless you have a magic solution for that I'm not sure this is a feasible approach? |
|
Incidentally I've never heard of the "dokken" driver - what made you chose that over the standard docker driver? |
|
Andy recommended the Dokken driver: it’s designed to emulate complete systems, has a functioning init, and is built specifically for Test Kitchen and Chef. @jalessio and I went some distance down the path of the plain Docker driver and found that we were duplicating existing work from Dokken. |
@tomhughes I can't definitively say yet that all of these will run under Docker, but one of the main issues I've encountered while testing Chef cookbooks via Docker is that you often need an init system running in order to test daemons such as MySQL. We got over this hurdle by utilizing the line I have a branch which uses this code to test the mysql cookbook from this repo, which demonstrates that the MySQL daemon starts successfully as part of the CI suite using Docker on Travis. It's part of this build (https://travis-ci.org/jalessio/openstreetmap-chef/builds/648127647) and sample MySQL logs are here (https://travis-ci.org/jalessio/openstreetmap-chef/jobs/648127652). My kitchen-add-passing-cookbooks-to-ci branch currently has builds working on Travis for 6 of the 14 cookbooks in this repo. On that branch, I only added cookbooks which passed without any modification (as an easy starting point). I haven't dug into each of the failing cookbooks yet, but I don't think the failures are necessarily related to using Docker since they are failing for reasons like missing test Data Bags (meaning they probably fail under Vagrant too). As for utilizing Dokken, it does take care of some things we'd have to build anyway to make this work and my confidence was boosted when I saw it being used extensively in the check-cookbooks repo. |
|
Yes the init system is the main issue but I did actually have that running and still had some issues. Plus I was actually using I'll try and have a look at this at some point this week. |
|
Is there a reason why you have added I'd rather not have the complexity of supporting two different platforms for the tests if we can avoid it, and it we can make the tests run in a container environment then that would seem to be the better option from a performance point of view. Personally I would have preferred it if this PR was split in two so that we could address the question of which driver to use separately from the question of running the tests in travis - aside from anything else I'm not sure it's actually practical to run the tests in travis once we have more testing just because of the length of time it would take. I've had a brief look at |
|
Re: separate platform, Jamie and I had set up Docker as a complete Vagrant replacement originally and then looked for a way to make this change less invasive. This PR introduces Docker specifically for use in CI without upending any existing Vagrant setup you or anyone else from the OWG or sysadmins teams might have. A future version of Re: split PR, I’m unsure it’s possible to split these two ideas. Only a Docker-based driver will work on hosted CI like our Travis integration. We previously tried to make this change work without a new driver but I think these two modifications have to go hand-in-hand. It’s possible to address the length of time by increasing our Travis parallelism with a paid plan: most of these test suites are quite speedy. Glad to hear Docker is not a deal-breaker for you! |
|
The thing is that I'm reluctant to commit to a specific technology until we know it will work across the board, which is why I'd rather look at switching technology to something that will work in travis first. Once I've reinstalled docker here I'll have a go with this and see how it goes across the whole range of test suites to get some idea of what the status is. I have my own notes from trying kitchen-docker as a baseline to compare to. |
|
Unfortunately it turns out that Fedora 31 can't run docker anymore, at least not without rebooting to go back to groupsv1, so I'm not going to be able to test this right now. |
|
Kitchen-dokken is our preferred method of testing chef cookbooks in CI systems. Kitchen-docker's PID 1 won't allow you to test most services. Kitchen-dokken will properly allow starting services. That coupled with the dokken images, which are "fat" containers allowing for full service testing, and you have fairly complete testing setup that can run in most CI environments. |
Implements @Firefishy's suggestion: openstreetmap#266 (comment)
…jalessio/kitchen-separate-platform
|
I believe I've now fixed all the outstanding issues with the tests, and have a configuration at https://github.com/tomhughes/openstreetmap-chef/tree/dokken that can run them all in docker and in travis, as seen here: https://travis-ci.org/tomhughes/openstreetmap-chef/builds/649655674 |
|
Exciting! Do you plan to accept this compatible PR which includes comments in the config files explaining how things work, or would you like for us to close #266 out so you can use the version you’ve written? |
|
I took the commits from this PR and rearranged them slightly then added my work on top so that preserve as best as possible attribution for the work - that is all now pushed on master. The only thing I haven't taken is the comments - many of them were either no longer relevant or were just documenting stuff that was boilerplate from the dokken instructions so I left them out. Hopefully that all works for you, and thanks for your work on this, and for finding dokken! |
|
This works, and I’m excited to see new expanded test coverage on master! |
6 participants
In #259, we noted that observing Test Kitchen outputs required manual sharing of logs and some conversation about working Test Kitchen installations. This PR introduces the possibility of testing Chef cookbooks right in Travis CI, thanks to the use of Docker under Kitchen-Dokken which will successfully complete in a virtual environment such as Travis.
@jalessio and I have started with the small Apt cookbook suite as a proof of concept. Cookstyle and Kitchen tests now run in parallel with logs that can be shared and observed between editors in PRs to this repo. The end-user display in Travis is pretty nice:
If accepted, future work would include adding each cookbook suite (Wiki, Website, etc.) in turn to the Travis config and ensuring that it passes tests. @gravitystorm has recommended that we break this work up over time and add remaining suites in followup PRs.
Notes:
.kitchen.ymlcould be simplified a lot by abandoning Vagrant entirely, but for now this change should be completely backwards-compatible with existing Vagrant setups.vagrant --versionreturns a valid version string. Since we can’t actually run Vagrant/Virtualbox on Travis, we cheat a little here by generating a one-line/usr/bin/vagrantto echo a fake version string. We could alsoapt-get install vagrantto be more correct, with some startup time cost.