docs: sauce issues

[laurelnaiad]

I had a pull request all set for this, but I realized that I haven't done a CLA since starting at my current employer. So, it's more expedient to just point this out.

I had assumed that if I opened a Sauce Connect tunnel, I could just point at localhost as if I was running against a local selenium, and so I still had a seleniumAddress file in my config, and I was wondering why I was getting errors that my user and key hadn't been provided.

Poking in the code, I discovered that seleniumAddress overrides sauceUser and sauceKey, which isn't documented on line 58 of server-setup.md.

Additionally, browser-setup.md says on line 75 that if multiCapabilities is set then capabilities is ignored, but this is not entirely true. Protractor errors out if the capabilities key doesn't associate an object that minimally specifies a browserName. This one seems like a bug to me. In fact, the sauce driver doesn't seem to support multiCapabilities at all... is that so? Or is it not working because I'm manually creating the "protractor/lib/runner" instance in my gulp process and not doing something that I should be?

I'll get to work in my company with signing a CLA so I'm not handcuffed in the future.

[laurelnaiad]

@juliemr since you tagged this for docs, does that mean that mulitCapabilities is unsupported for Sauce configurations?

Due to that and some other reasons on my side, I'm trying to manually run through each browser we support in one gulp command.

It always fails on the 2nd, 3rd, 4th, etc. browsers.

The error stack looks like this:

Error: This driver instance does not have a valid session ID (did you call WebDriver.quit()?) and may no longer be used.
    at checkHasNotQuit (/Users/ssalsbur/Documents/_ws/myproj/browser/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/webdriver.js:334:13)
    at [object Object].webdriver.WebDriver.schedule (/Users/ssalsbur/Documents/_ws/myproj/browser/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/webdriver.js:281:3)
    at [object Object].webdriver.WebDriver.Navigation.to (/Users/ssalsbur/Documents/_ws/myproj/browser/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/webdriver.js:969:23)
    at [object Object].webdriver.WebDriver.get (/Users/ssalsbur/Documents/_ws/myproj/browser/node_modules/protractor/node_modules/selenium-webdriver/lib/webdriver/webdriver.js:661:26)
    at [object Object].Protractor.get (/Users/ssalsbur/Documents/_ws/myproj/browser/node_modules/protractor/lib/protractor.js:1221:15)
    at World.go (/Users/ssalsbur/
// etc. etc. , now we're in cucumber land

Boiling down the automation code that kicks this off and leaving the protractor related steps would yield this:

var Runner = require('protractor/lib/runner');
var ptorConfig = {
  stackTrace: false,
  getPageTimeout: 20000,
  allScriptsTimeout: 40000,
  baseUrl: 'http://localhost:3000/',
  rootElement: 'html',
  chromeOnly: false,
  framework: 'cucumber',
  specs: require('globule').find(
    path.resolve(ctx.paths.projectRoot, 'specs/features/**.feature')
  ),

  cucumberOpts: {
    require: path.join(ctx.paths.rootDir, 'test/cucumber-support/**/*.js'),
  },

  sauceUser: 'stu-salsbury',
  sauceKey: 'mykey',
  sauceSeleniumAddress: 'localhost:4445/wd/hub',
  capabilities: {
    // this is from the set that I loop over -- these values are changed in my loop
    platform: 'Windows 7', browserName: 'chrome', version: '33', deviceName: ''
  }
};

// now if you imagine I loop over my supported browsers and run this code in the loop.
// (this stuff is in an async function that is called repeatedly after deep cloning the base 
// ptorConfig and then tweaking the capabilities

var runner = new Runner(ptorConfig);
try {
  runner.run().then(
    function complete (exitCode) {
      // do "stuff" with the results
      cb();
    },
    function failed (exitCode) {
      // do "stuff" with the failure info
      cb(exitCode);
    }
  );
}
catch (err) {
  cb(err);
}

It seems to me that the runner must either be "overly" cleaning up after itself or not doing enough cleaning up after itself, and at a high-level that it is winding up in an inconsistent state after the run -- almost as if its prototype got messed with in a bad way, since I'm reinstatiating the runner instance for each attempted run.

Alternative, maybe there's something I need to say to selenium about the fact that I'm trying to change the capabilities before I start using the new runner instance?

Note -- I'm not actually hitting the "failed" callback because at this level, it looks like success -- the tests complete and individually fail with the above stack, the overall execution of the run function results in the success callback nonetheless.

[juliemr]

multiCapabilities is supported for Sauce. Protractor uses this for our own CI tests. Here's an example https://github.com/angular/protractor/blob/master/spec/ciConf.js

I suspect that something is going wrong with how you are using the protractor/lib/runner instance. See lib/launcher.js for how Protractor manages multiple capabilities.

[laurelnaiad]

I did look at that. The main difference I see is that the launcher seems to use a separate process and exits/starts a new one for each run. Does that mean that reusing the prototype to get a new instance of Runner within the same process isn't intended to work?

[juliemr]

Correct, that was never an intended use case - we don't test with it, so there may be a way to make it work but I'm not 100% sure without spending more time testing it out. Runner sets up some global values, so those are probably conflicting: https://github.com/angular/protractor/blob/master/lib/runner.js#L146

[laurelnaiad]

Gotcha -- I missed those. Glutton for punishment that I am, I'm going to try to deal with that instead of relenting and going out-of-process.

The reason I'm actually more interested in separating the runs outside of protractor runner than doing multiCapabilities has to do with the fact that cucumber has no facilities for writing results to a file, and I have to hook the process.stdout and write the JSON manually to files in order to capture the results and then turn each of them into XUnit/JUnit files for our nightly test harness to consume.

It would be great if there were a way in protractor to redirect test output to a file, but it seems the architecture relies on the test framework for that. So, barring that, doing it this way makes it easier for me to isolate the results output from the other messages and to keep the results of the multiple runs separated.

Thanks for your help!

[juliemr]

Re: test output to JSON, we're working on it! See #1485

[laurelnaiad]

But not for cucumber I imagine... I was hoping that this effort would be focused on doing it at a level above the test framework, such that it doesn't care which framework you're using... the one thing cucumber gets right in this regard is that they have JSON output, but they refuse to add a flag to write to a file.

They think this is supposed to be protractor's responsibility: cucumber/cucumber-js#90 (comment)

People who are wanting to run angular e2e tests with cucumber are stuck in a tug of war. :(

[hankduan]

Yes, the #1485 attempts to unify the test output for all test framework into a single format. Cucumber hasn't been done though. I'll look at it now to see how feasible it is.

[laurelnaiad]

Ah, ok, great! I just saw the jasmine related code and imagined this was a jasmine-only thing. Thanks for taking a look @hankduan. I dont' know how you're going about it but the best I was able to come up with from my automation-level code was to hook stdout. If you look at the issue for cucumber I linked above, you can see some other ideas that were floated which may be more available to you within protractor than they were to me at the higher level.

[hankduan]

Alright, so cucumber works with protractor to produce a bare-bone result output now. It looks something like this:

[
  {
    "description": "I run Cucumber with Protractor",
    "assertions": [
      {
        "passed": true
      }
    ],
    "duration": 325723
  },
  {
    "description": "it should still do normal tests",
    "assertions": [
      {
        "passed": false,
        "errorMsg": "expected true to equal false",
        "stackTrace": "AssertionError: expected true to equal false"
      }
    ],
    "duration": 1966454
  },
  {
    "description": "the title should equal \"My AngularJS App\"",
    "assertions": [
      {
        "passed": true
      }
    ],
    "duration": 1111641923
  }
]

Three caveat about cucumber (which are all fine with Jasmine/Mocha).

1. Unlike jasmine, it doesn't expose assertion-level detail about the failures, only step-level, so the results can't capture which assertion failed.
2. the duration is pretty weird as the number ranges are all over the place. I used what they provide for duration.
3. the stacktrace isn't really a real stacktrace. (From the 1hr I played with this, I can't find where they store the stacktrace info, but if it really affects you, feel free to send a PR).

[laurelnaiad]

Thank you very much, @hankduan and @juliemr . That was incredibly fast! I will take a look as soon as I can.

Stacktraces are a sticky point, too. I have really wrestled with the cucumber/webdriver combination. cucumber/cucumber-js#157 (comment). The upshot of what I discovered and documented in that thread is that webdriver wants to take apart and rebuild stack traces, but cucumber manually writes stack traces that don't quite match the regex webdriver uses to parse them, and everything tends to come back as <at <anonymous> because webdriver can't comprehend the strings. I was able to fix it by creating a prepareStackTrace function override that cleans up the strings to bring them in line with webdriver's expectations. I suspect that loosening webdriver's regexp a little would be an alternative, but if I remember correctly I didn't find an efficient way to monkey patch it (perhaps the function was "relatively" private from my automatio code, or maybe I just never contemplated that option). Either way, it doesn't seem like something that I would blame on protractor, even if protractor code happens to be in a better position to establish compatibility than my code is.

Anyway, with this I can try to return to making multiCapabilities work, and that would be super if it enables me to take advantage of the 3 simultaneous sessions that Sauce provides for OSS. So again, thanks!!!

[hankduan]

@stu-salsbury Sorry but we're not actively supporting the cucumber framework (it's more community-driven). I just included cucumber output as part of that PR since it was easy and allows me to create test cases with that. Feel free to submit cucumber PRs and we will review them and pull them in though!

[laurelnaiad]

Thanks @hankduan . Yes, I realize that cucumber isn't officially supported. For me, the bigger issue now is that I find that the center of gravity has moved even further out of the runner and into the launcher and so I'm left with a lot of functionality that isn't suited to running in my gulp process. I guess that means that I need to target at least the launcher as my entry point if not attempt to harness the CLI itself. I was really hoping not to add to the already lengthy list of out-of-process steps in my build.

[laurelnaiad]

I'm sorry that I didn't look very closely at the example JSON above. Shoot.

I started running with the cli in order to be able to do multiCapabilities and to write the file, and while it's great that I'm able to get multiCapabilities to work there, the json output doesn't match cucumber's so I can't use it -- I need the whole thing because it goes into the XUnit conversion library to become XUnit results. Also, all of the abbreviated results are together in a big array with no identifying properties to distinguish between browsers/OSs.