Feature request: ability to have Ginkgo tests run with a different test runner

[fabioy]

Allow tests written using Ginkgo's DSL to run with a different test runner.

A scenario that's being envisioned is to write end-to-end tests using Ginkgo, but to be able to reuse the same tests in a long-running or continual stress environment. The stress runner would do things like randomize test selections (with optional selection weights); run them repeatedly (and in parallel); and other monitoring. It's not envisioned that all tests could be run with a different runner, but it's something that could be enabled.

One way to achieve this would be to extend the current runner with a lot of other options, but it seems like a cleaner approach would be to set a different test runner run the Ginkgo tests. I think some of this could be done by having constructs that expose the "global***" vars in "ginkgo_dsl.go", and maybe add a few more abstractions/constructs.

[onsi]

A lot of this can be accomplished today by scripting the ginkgo cli. Exposing the tree of tests and allowing custom test runners is a cool idea but I wonder if it's overkill w.r.t the problem you are actually trying to solve - namely: keep running all/some of these tests and report on failure/success statistics.

I agree that adding yet more options to the cli is probably not the way to go - though even there we are already most of the way there: ginkgo --untilItFails --randomizeAllSpecs -p --focus="{STRESS}", for example, is not too far from what you are asking for -- wrap it in a while loop (or call it programmatically) and you've effectively solved the problem at hand.

[fabioy]

Scripting ginkgo cli only goes so far. For the type of scenarios envisioned, access to the tree is the right approach. For example, it's easier to setup and share external resources among the tests, as well as do things like change the behavior on panics (some may be acceptable to continue, some should stop all testing for stress failure investigation).

While a lot of scenarios can be done with the current driver, burdening it with ever more options doesn't seem to be the right thing.

[onsi]

Apologies for the length of this post. I don't mean to be difficult - just want to make sure we identify the actual problems you are trying to solve. If gaining access to the tree is the best solution to the problems you want to solve I am open to that implementation choice (note, however, that writing a runner is not completely trivial so this adds a substantial burden on the runner maintainer). At this point, I'm not convinced that this is the best solution for the problem at hand.

Here are the problems you've raised so far:

1. Construct a long-running/continual stress environment.
2. Randomize test selections during these stress tests.
3. Run the tests in parallel.
4. "Other monitoring".
5. Setup and share external resources among the tests.
6. Control whether a test failure should cause the long-running test suite to fail or to keep going.

I'm unconvinced that a custom runner is the best solution to these problems. I do believe that Ginkgo today, in an unmodified form, gives you most of these things - and that there are probably straightforward ways to extend Ginkgo to give you all of these things. I'd value the latter approach as it would give more people the kind of sophisticated tooling that they need.

Let's go at these one at a time:

1. Construct a long-running/continual stress environment.
   This is possible today with ginkgo --untilItFails and a while loop. I could imagine extending the Ginkgo CLI to make it explicitly support this usecase. ginkgo --stress or, far better, a new subcommand ginkgo stress that takes all the relevant existing flags.

   This is actually generically useful and would be valuable for most consumers of Ginkgo.

2. Randomize test selections during these stress tests.
   This is already the case - moreover Ginkgo is careful to make this randomization reproducible and gives you the randomization seed you need to reproduce the particular test run in question. That sort of thing would have to be managed by the custom test runner - an unnecessary burden.

3. Run the tests in parallel.
   This is something ginkgo gives you today and it's highly non-trivial. A part of the burden of making ginkgo -p so seamless falls on the runner - and making the runner extensible could push some of this burden on the end user. Again - I'm open to making this change but I want to avoid unnecessary complexity and make sure you're solving the problem you want to solve.

   ginkgo stress -p sounds like a pretty sweet feature that I'd be happy to add to Ginkgo ;)

4. "Other monitoring"
   I'm not sure what you are envisioning here. These is something that can be most easily accomplished today with a custom reporter. I've written custom reporters that do things like generate CSV data. You could easily add a custom reporter that hits an API endpoint to build reporting. In fact, the Reporter interface will have all the information you'll need whereas a custom runner will have to collect and aggregate this information manually.

5. Setup and share external resources among the tests.
   This is something that we've invested a fair amount of effort in already and I think putting this sort of thing in a custom runner is an anti-pattern. At the very least it would entail that there is a tight coupling between the runner and the test suite and would make the two less separable and, therefore, less generic. This is why BeforeSuite and AfterSuite exist.

   In particular, setting up shared external resources is especially challenging when you want to run tests in parallel. This is why SynchronizedBeforeSuite and SynchronizedAfterSuite exist - they are most likely what you want to use for your test setup and Ginkgo is very careful to call these things correctly when failures occur and when the test suite is interrupted.

   The docs cover this particular issue in detail.

6. Control whether a test failure should cause the long-running test suite to fail or to keep going.
   This would be a new feature request and I can see the value in it. I'd want to dig into this one some more. I don't think a custom runner would actually help here -- when you ask an individual Ginkgo spec to run it manages detecting failures and produces a simple report as to the status of the test. A custom panic wouldn't make its way to the custom runner with the way Ginkgo is built now.

   However, one could imagine being able to annotate an It with a statement of intent like:

   It("...", func() {
   
   }).AbandonSuiteOnFailure()
   

   or something - that would be one way to tell Ginkgo that failures emanating from this particular test should short-circuit the test suite. (Again, I'd want to hear more about the underlying problem before proposing a concrete solution).

[zmerlynn]

@fabioy: I'm actually with @onsi here. (@onsi, as a disclaimer, we're both on the Kubernetes project, so you've seen our questionable runner). We need to actually port our stuff over to the default ginkgo runner. There are technical blockers for us to do so and we just haven't gotten through them yet, but I think we either need to embrace the main Ginkgo runner, or (to be blunt) abandon the framework (and I don't want to do the latter).

[onsi]

Yep - would love to hear more about your technical blockers and help y'all work through them. We have an equally (if not more) complex set of Ginkgo test suites for Diego.

[zmerlynn]

It probably makes more sense to do that as a either a separate issue, or an email thread. I'm happy to open an issue on our side and @ you on it, because it's something I've been threatening to do for a while.

[fabioy]

Let me dig a bit more into this. Meanwhile, the actual problem I'm trying
to solve is designing a stress tool for for Kubernetes (see
kubernetes/kubernetes#5601).

Where I'd like to get to is to be able to say, "If you write your e2e tests
using these components (i.e. use this to setup, this to tear down, etc),
then you can do and it'll become part of the stress suite". The
stress suite would then allow you to arbitrarily run a combination of
tests, while optionally doing some fault injections at various
(configurable) places to test for resiliency and recovery.

Asking for access to the test tree was one way to get there. I think a lot
(if not all) of the requirements could be met with a pluggable scheduler
into the runner. This would allow my code to induce fault injections while
tests are running, pause/stop tests under some conditions, and other things
like a random weighed selection of tests (i.e. run this test 1% of the
time, this other 50%, etc). Though the last option could just be a standard
ginkgo feature.

I have a WIP change for my tool that prototypes some of this. If I have
time, I'll dig a bit more into gingko and see if I can propose a PR for it.

Thanks.

On Thu, Mar 19, 2015 at 2:06 PM, Zach Loafman notifications@github.com
wrote:

It probably makes more sense to do that as a either a separate issue, or
an email thread. I'm happy to open an issue on our side and @ you on it,
because it's something I've been threatening to do for a while.

—
Reply to this email directly or view it on GitHub
#146 (comment).

-- Fabio Yeon

[zmerlynn]

Heh. I miss fail(9) so much. (So much that I ported it out from the company I was at to FreeBSD. :P)

@fabioy, I think what you're looking for is just a fixture that can be queried for whether failures should occur at particular junctions or not. You could easily imagine something as dumb as:

if (shouldFail(theFailmaster, "some context")) {
  aggghhhh
}

but maybe that's not what you had in mind? Maybe you wanted more implicit fault injection?

[onsi]

alrighty - sorry for the delay but I ended up spending a bit of time prototyping something.

https://github.com/onsi/composition-ginkgo-example presents one way to organize Ginkgo tests such that they can be shared between a traditional integration-style test and something more complex like a stress test.

I'll be the first to admit that the stress test I wrote doesn't quite give you all the fine control you're envisioning @fabioy. One thing I did not include, for example, is the ability for the test writer to request failures at specific points in the code. This would be fairly trivial however (add something like SharedContext.InvokeTheFailMaster("some context") to the SharedContext that is passed into the tests and have it decide what to do). I'd be happy to describe how additional features could be layered in on top.

but

I also want to step back and dig into your proposed stress test approach. It sounds like you aren't proposing a "chaos-monkey" style approach where a standard integration test runs in the foreground while an autonomous process wreaks havoc on the object under test. Instead, it sounds like you want to have a lot of fine-grained control over when failures can occur and, perhaps, what the nature of the failures is, etc.

We've taken a different approach. While we'd like to have the "chaos-monkey" style that I just described (we don't have it yet) we do have a test suite that exercises the system under test through specific high-value failure modes. For example: we request that the cluster take an action and then immediately shut down one of its components. We then bring the component back up and assert that, eventually, the system attains eventual consistency.

We've found this sort of testing to be very useful and (most importantly) substantially easier to debug than a more chaotic approach. Granted this puts the onus on the developers to characterize the various failure modes of the system (and, no doubt, we've missed many!) - nonetheless I would argue that it's actually very valuable and guards against regressions to the various mechanisms that ensure robustness in the face of common failures (VMs disappearing, network partitions forming, etc...)

Anywho -- take a look at the example if you have time. I'm not suggesting that it's ideal or perfect - just that Ginkgo does give you some flexibility here.

[fabioy]

(I'm on vacation and replies will be very delayed). Thanks for the example. I've cloned it and will take a look when I can. I don't think our approaches are all that far off. I look at "predicted failure and recovery type of test" as another type of integration test. Useful to ensure product works and to catch regressions. A category that perhaps falls a bit above it, but not quite chaos-monkey, is what I call "resiliency" test. I can induce failures at a random rate (though influenced by what the test is doing), ensuring that the random factor may expose other flaws. The hope is that this will help expose more of the edge cases that may have been overlooked. Anyway, again thanks for the example.