Missing failing teststeps in junit report using nightwatchjs 0.7

[MateuszJeziorski]

After upgrading nightwatch to 0.7 version, there are missing information about failing assertion in junit report. I'm using junit report with jenkins. Because of this bug, all tests are marked as "passed" in jenkins.

Example test file:

module.exports = {
    tags: ['FailingReport'],

    'passed': function (browser) {
        browser.assert.ok(true, "true");
    },

    'failed': function (browser) {
        browser.assert.ok(false, "failed");
    }
};

JUnit report:

<?xml version="1.0" encoding="UTF-8" ?>
<testsuites errors="0"
            failures="1"
            tests="2">
    <testsuite name="ReportTests"
    errors="0" failures="1" hostname="" id="" package="ReportTests" skipped="0"
    tests="2" time="1.305" timestamp="Mon, 22 Jun 2015 23:02:42 GMT">
        <testcase name="passed" time="1.302" assertions="1">
        </testcase>
        <testcase name="failed" time="0.003000" assertions="1">
        </testcase>
    </testsuite>
</testsuites>

Seems in nightwatchjs 0.7 is messed up (or missing) failure field.

module in nightwatch 0.6.x

{
  "completed": {
    "passed": {
      "passed": 1,
      "failed": 0,
      "errors": 0,
      "skipped": 0,
      "assertions": [
        {
          "message": "Passed [ok]: true",
          "stacktrace": "",
          "failure": false
        }
      ],
      "time": "1.338"
    },
    "failed": {
      "passed": 0,
      "failed": 1,
      "errors": 0,
      "skipped": 0,
      "assertions": [
        {
          "message": "Failed [ok]: (failed)",
          "stacktrace": "Failed [ok]: (failed): Expected \"true\" but got: \"false\"\n    at Object.ok...",
          "failure": "Expected \"true\" but got: \"false\""
        }
      ],
      "time": "0.006000"
    }
  },
  "skipped": [],
  "time": "1.344",
  "timestamp": "Mon, 22 Jun 2015 23:17:20 GMT",
  "tests": 2,
  "failures": 1,
  "errors": 0
}

module in nightwatchjs 0.7

{
  "completed": {
    "passed": {
      "passed": 1,
      "failed": 0,
      "errors": 0,
      "skipped": 0,
      "assertions": [
        {
          "message": "Passed [ok]: true",
          "stacktrace": "",
          "elementNotFound": false
        }
      ],
      "time": "1.323"
    },
    "failed": {
      "passed": 0,
      "failed": 1,
      "errors": 0,
      "skipped": 0,
      "assertions": [
        {
          "message": "Failed [ok]: (failed)",
          "stacktrace": "Failed [ok]: (failed): Expected \"true\" but got: \"false\"\n    at Object.ok ...",
          "elementNotFound": "Expected \"true\" but got: \"false\""
        }
      ],
      "time": "0.005000"
    }
  },
  "skipped": [],
  "time": "1.328",
  "timestamp": "Mon, 22 Jun 2015 23:21:21 GMT",
  "tests": 2,
  "failures": 1,
  "errors": 0
}

Seems I found place in code responsible for this problem. Check commit: bc3b36167f3be15ca26856f45019fd748ed10d37, file: lib/core/assertion.js

Changed

-      failure : failure !== '' ? failure : false
+      elementNotFound : failure !== '' ? failure : false

Seems corresponding changes was done in lib/runner/reporter.js, but not in junit.xml.ejs. So this 'if'

if (assertions[i].failure)

is always false using nightwachjs 0.7.

Was this change suppose to be done? I don't see any reason to change 'failure' to 'elementNotFound'

[sknopf]

Thanks for finding and looking into, we'll put a fix in soon.

[MateuszJeziorski]

What about "lib/runner/reporter.js"? Shouldn't it be reverted as well?

           test.assertions.forEach(function(a) {
-            if (a.failure !== false) {
+            if (a.elementNotFound !== false) {
               if (this.options.start_session) {
-                console.log('     ' + a.message + ' - ' + a.failure);
+                console.log('     ' + a.message + ' - ' + a.elementNotFound);

and thank you for quick fix :)

[beatfactor]

Yes, also. But that doesn't affect the JUnit reports. We'll publish a patch later on today.

[vivian-ho]

Ran into the same issue, awaiting your fix, thanks.