Automated tests of docker images

[retrohacker]

Run iojs and npm through their paces before considering them ready to deploy.

[yosifkit]

Would this be helpful: official-images test suite? We are working on having more tests for more images. If it can work for you, feel free to add tests and help us improve the tool for everyone 😃.

[retrohacker]

@yosifkit probably! I'll take a pass at it soon.

[Starefossen]

An automated build on Docker Hub would give us build errors. Another option is Wercker which seams very Docker oriented – but they do not currently support building your own Dockerfiles on their infrastructure [1].

[1] http://devcenter.wercker.com/docs/faq/can-i-build-dockerfiles.html

[termie]

Hi hi, one of the wercker guys here. It's true that we don't support building dockerfiles at the moment, though there are plenty of places that will do that part (we're still debating supporting it considering how easy it is to get done and how non-standardized the commands people tend to use in them are).

The two common use cases we see are 1) people using that built dockerfile image as a base on wercker for running their tests and 2) people building and testing their code using wercker and then deploying to the hub/etc using a pristine base image (you can build/test using one image and then drop that output into small image without build deps and push that).

Happy enough to help out if there are more questions :)

[retrohacker]

Thanks @termie! @Starefossen I was considering creating a Dockerfile that variably inherits from each of the images we would like to test, and that file have the unit tests in it. Docker Hub may not be a good candidate for that since it requires shipping before testing.

I was considering setting up a build server on D.O. that handles it, but it would be a bit before I can get to that point.

[Starefossen]

Did you have any thoughts on which tests to run @wblankenship? My first thought was to run the io.js core test suite – but this actually crashes v8 at some point along with a bunch of other test failures!

This was my Dockerfile setup:

FROM iojs:1.6.3

RUN curl -L https://github.com/iojs/io.js/tarball/v1.6.3 | tar -xz 
RUN mv iojs-io.js* /usr/src/app
RUN ln /usr/local/bin/iojs /usr/src/app/out/Release/iojs

WORKDIR /usr/src/app

CMD python tools/test.py

[retrohacker]

I want to run the io.js test suite and nodesource/distributions#12 (comment) at a minimum.

[termie]

the io.js tests are pretty intensive, I get about 28 failures a bunch of which are based on the test being run a little differently than when io.js runs them normally (which is to say there is supposed to be a bit more stuff in the Release dir than just iojs).

Anyway, I wanted to see how the new wercker stack handles it and in general poke at your use case:

https://app.wercker.com/#build/551f13156dda5c9817000207

You can see the wercker.yml I used for the build (basically just a copy of your dockerfile and the test @Starefossen described above) in the setup environment step: https://app.wercker.com/#buildstep/551f13156dda5c9817000208

or in my branch https://github.com/termie/docker-iojs/blob/wercker/wercker.yml, I tried to annotate it to explain what things were doing.

[termie]

(on the plus side, v8 doesn't fail :) )

[jbergstroem]

@termie: Do you mean building v8 or testing it? Testing v8 against musl would be interesting to hear about.

[Starefossen]

@jbergstroem I think @termie v8 comment was directed towards my comment that I managed to get a V8_Fatal error when running the io.js test suite inside of the official Docker image for io.js.

[jbergstroem]

@Starefossen ah, ok.

[retrohacker]

Here is my current CI setup, its a sandbox for the iojs/docker-iojs project (it is really rough around the edges): http://104.236.43.123:8080/job/test/11

It is hooked up to the repo: retrohacker/test#5

[retrohacker]

Congratulations @termie and the wrecker group: http://blog.wercker.com/2015/04/02/Introducing-our-docker-stack.html

😄

[Starefossen]

Wow! Very nice indeed @termie 😄👏

[retrohacker]

I've been a blocker on this for a while. Unassigning myself incase someone else wants to take a pass at it.

So far, I've created a tool for automated testing of Docker images: https://github.com/wblankenship/dante

And put together some smoke tests for the Docker image, which follows the work of rvagg: https://github.com/wblankenship/docker-iojs/blob/tests/tests/iojs/Dockerfile

These tests are executed according to the inventory file here: https://github.com/wblankenship/docker-iojs/blob/tests/inventory.yml

The tests currently do not run the Node.js project's test suite, which may be a good idea.

We have an open pull request for the build group to provision us a Jenkins server that we can wire this workflow up to: nodejs/build#168

The Jenkins slave should trigger the test on a new PR to this repo. The test is run by simply calling dante in the base of the project directory. If the tests pass, dante will exit with a status 0, else it will exit with a status of 1. dante creates tagged images for each build for historical reference. This will chew up HDD space over time so we will need some sort of garbage collection as well.

</braindump>