[WIP] Fix how we call grunt with multiple tasks

[ycombinator]

This way, if we invoke npm test with options like npm test -- --esvm-no-fresh, the options are available to all grunt tasks, not just the last one.

[ycombinator]

jenkins, test it

[jbudz]

Any reason for not adding this to the grunt test command? Not running on jenkins? https://github.com/elastic/kibana/blob/master/tasks/test.js#L18

[cjcenizal]

👍 Awesome! Thanks for the improvement.

[ycombinator]

@jbudz I see no reason not to do it that way. I didn't know the original intent of doing it in package.json so I left it that way and made a minor improvement. @cjcenizal would it be okay to move test:visualRegression as a sub-task of test:quick in the grunt task file instead?

[cjcenizal]

@ycombinator After talking about it with @spalger, I put the task into package.json so that the visualRegression task would run regardless of the tests failing. I think if it's a sub-task, grunt will exit after a test fails, and visualRegression won't run. The long-term goal is to be able to run this on Jenkins and push the resulting visual regression gallery to S3. But I haven't been able to get access to Jenkins's configuration yet, so I haven't been able to set that up.

[ycombinator]

@cjcenizal I see. Then I think we need a way to accomplish both these constraints: run the visualRegression task regardless of tests failing and be able to pass any command-line options to all grunt invocations. Perhaps the solution lies in moving visualRegression into the grunt tasks file but not as a sub-task of test:quick.

[spalger]

The reason this was done this way is that we want the test:visualRegression task to run even if the test task fails. using --force won't work because it won't differentiate between the sub-tasks of test (which is shouldn't run if test fails) so the only way I can think of to do this is using the ; command separator.

[ycombinator]

@spalger The current behavior with using the ; separator seems broken to me. My specific use case is that I want to be able to call npm run test -- --esvm-no-fresh and not have ESVM download a fresh Elasticsearch build. Currently that's broken because the effective command becomes grunt test; grunt test:visualRegression --esvm-no-fresh. Note that the first grunt command never gets passed the --esvm-no-fresh option so it ends up downloading a fresh Elasticsearch build anyway.

I understand we want the test:visualRegression task to run even if the test task fails. But I also think any commandline options should be available to both tasks.

[spalger]

Yeah, that's fair, which is why I didn't close the pr :) we just can't merge it as is

[ycombinator]

Agreed. I'll mark it as WIP and remove the review label so its clearer that it should not be merged in its current state.

[ycombinator]

I looked into using grunt-continue but that simply toggles --force around specific tasks via continue:on and continue:off, which isn't going to cut it as Spencer noted in an earlier comment. Continuing to look into other ways of achieving both constraints.

[epixa]

I don't understand. Why not just add visualRegression to test?

[ycombinator]

@epixa Unless I'm misunderstanding your question, I think the answer is in this comment from CJ above: #7461 (comment).

BTW, I think I have something that works while satisfying both constraints. @spalger and others, mind giving this another review?

[ycombinator]

This does mean development has to be done on a system with bash on it, which may be too restrictive.

[epixa]

Yeah, that's a no go. The dev tasks need to be able to run on windows and such.

[ycombinator]

Okie doke. Marking this PR as WIP again.

[epixa]

Our tests don't bail once a single failure is found, so I'm not sure how adding visualRegression to grunt test would prevent it from running if there were tests failures.

[cjcenizal]

@epixa I did an experiment by adding the visualRegression task to test:ui:

grunt.registerTask('test:ui', [
  'esvm:ui',
  'run:testUIServer',
  'downloadSelenium',
  'run:seleniumServer',
  'clean:screenshots',
  'intern:dev',
  'test:visualRegression',
  'esvm_shutdown:ui',
  'stop:seleniumServer',
  'stop:testUIServer'
]);

I then hard-coded a test to fail and tried running grunt test. The intern:dev completes and reports the failure, but then grunt exits without running test:visualRegression and subsequent tasks.

[epixa]

In that case, if tests do bail once one task fails, why do we want visualRegression to behave differently than the other test suites?

[cjcenizal]

I think the idea is that we could use the generated screenshots/visual_regression_gallery.html to do some debugging. So you could run your tests, take a look at the gallery, and notice some UI error that's breaking the tests. At least that's the theory. 😄 I don't think the value of this has been proven yet...

[cjcenizal]

If this is blocking people or costing an inordinate amount of attention, I suggest we remove the visualRegression task from the npm scripts and refrain from adding it to the grunt test tasks. At this point, I think I'm the only person really using it, so I'm fine with just running grunt visualRegression locally for now.

[epixa]

For consistency's sake, we should just toss it at the end of grunt test and be done with it. You can always add a test:visualRegressions npm script to explicitly run this test if that's desired.

[ycombinator]

@cjcenizal Based on #7461 (comment), I'm going to close this PR (unmerged) so you can do a separate, clean PR instead.