Failing tests returning code 0

[digitalsadhu]

Not sure if this is my bad but my tests have failures yet $? tells me the exit code was 0. Because of this CI thinks everything went fine. Using nightwatch ~0.4.4

Test:

'use strict';

module.exports = {

  'Login': function (client) {
    client
      .url(process.env.BASE_URL + '/login')
      .waitForElementVisible('body', 1000)
      .assert.title('TMP For CHCH - Login')
      .setValue('#LoginForm_username', process.env.USER1)
      .setValue('#LoginForm_password', process.env.PASS1)
      .click('input#login')
      .assert.title('TMP For CHCH - Manage TMPs')
  },

  'List TMPs': function (client) {
    client
      .elements('css selector', '.items tr', function (result) {
        client.assert.ok(result.value.length > 0)
      })

    client
      .getAttribute('.items td a', 'href', function (result) {
        var match = result.value.match('tmp/trafficManagementPlan/[0-9]*')
        client.assert.notStrictEqual(null, match)
      })
      .end()
  }

};

Response:

TEST FAILURE: 1 assertions failed, 0 passed and 5 skipped.
Step "List TMPs" skipped.
FAILED: 1 assertions failed and 5 skipped

echo $? # -> 0

[beatfactor]

Yes, this will be fixed soon. But in any case, the CI server should parse
the test reports and fail the build if at least one test failed, no?

On Fri, Mar 21, 2014 at 3:27 PM, Richard Walker [email protected]:

Not sure if this is my bad but my tests have failures yet $? tells me the
exit code was 0. Because of this CI thinks everything went fine. Using
nightwatch ~0.4.4

Test:

'use strict';
module.exports = {

'Login': function (client) {
client
.url(process.env.BASE_URL + '/login')
.waitForElementVisible('body', 1000)
.assert.title('TMP For CHCH - Login')
.setValue('#LoginForm_username', process.env.USER1)
.setValue('#LoginForm_password', process.env.PASS1)
.click('input#login')
.assert.title('TMP For CHCH - Manage TMPs')
},

'List TMPs': function (client) {
client
.elements('css selector', '.items tr', function (result) {
client.assert.ok(result.value.length > 0)
})

client
  .getAttribute('.items td a', 'href', function (result) {
    var match = result.value.match('tmp/trafficManagementPlan/[0-9]*')
    client.assert.notStrictEqual(null, match)
  })
  .end()

}
};

Response:

TEST FAILURE: 1 assertions failed, 0 passed and 5 skipped.
Step "List TMPs" skipped.
FAILED: 1 assertions failed and 5 skipped

echo $? # -> 0

Reply to this email directly or view it on GitHubhttps://github.com/beatfactor/nightwatch/issues/99
.

[digitalsadhu]

nope, circle ci works by watching the error code. If its a 0, build passes.
I have put a hack in place for the time being as follows:

#capture test output
output=`npm test`;

#output test output
echo -e $output;

#check for test failure
echo $output | grep "FAILED:"

#if grep succeeds (code 0) then FAILED was found, fail the tests
if [[ $? = 0 ]]; then
  exit 1
fi

Pretty nasty so will look forward to your fix.

[beatfactor]

It is :). Which CI are you using?

On Fri, Mar 21, 2014 at 4:27 PM, Richard Walker [email protected]:

nope, circle ci works by watching the error code. If its a 0, build passes.
I have put a hack in place for the time being as follows:

#capture test outputoutput=npm test;
#output test outputecho -e $output;
#check for test failureecho $output | grep "FAILED:"
#if grep succeeds (code 0) then FAILED was found, fail the testsif [[ $? = 0 ]]; then exit 1fi

Pretty nasty so will look forward to your fix.

Reply to this email directly or view it on GitHubhttps://github.com/beatfactor/nightwatch/issues/99#issuecomment-38287063
.

[digitalsadhu]

Circle CI
Btw, I've checked with support and they confirmed that 0 means success, anything else means fail for them.

[beatfactor]

so they don't support junit xml output?

On Fri, Mar 21, 2014 at 4:37 PM, Richard Walker [email protected]:

Circle CI
Btw, I've checked with support and they confirmed that 0 means success,
anything else means fail for them.

Reply to this email directly or view it on GitHubhttps://github.com/beatfactor/nightwatch/issues/99#issuecomment-38288331
.

[beatfactor]

Fixed in da46284. Will be in npm soon also.

[digitalsadhu]

nice, you da man ;)

[pbiggar]

CircleCI dev here: junit.xml support is coming :)

[beatfactor]

Cool :), do you support other formats?

On Fri, Mar 21, 2014 at 5:11 PM, Paul Biggar [email protected]:

CircleCI dev here: junit.xml support is coming :)

Reply to this email directly or view it on GitHubhttps://github.com/beatfactor/nightwatch/issues/99#issuecomment-38292438
.

[pbiggar]

@beatfactor junit will be the first format (actually, we do support it, but need to make some schema changes before we can support it for everyone). After that we'll add other formats. Anything in mind?

[sorry to hijack the thread folks, please feel free to reach out to [email protected] with questions if you prefer]

[beatfactor]

No, nothing in particular. I was just wondering because nightwatch only
supports junit.

On Fri, Mar 21, 2014 at 6:51 PM, Paul Biggar [email protected]:

@beatfactor https://github.com/beatfactor junit will be the first
format (actually, we do support it, but need to make some schema changes
before we can support it for everyone). After that we'll add other formats.
Anything in mind?

[sorry to hijack the thread folks, please feel free to reach out to
[email protected] with questions if you prefer]

Reply to this email directly or view it on GitHubhttps://github.com/beatfactor/nightwatch/issues/99#issuecomment-38303665
.

[beatfactor]

Fixed in v0.4.7.

[digitalsadhu]

thanks! awesome!

[scalvert]

@digitalsadhu what does your circle.yml file look like? I can't seem to get this working...

[digitalsadhu]

circle.yml

machine:
  node:
    version: 0.10.22
test:
  pre:
    - npm run selenium:
        background: true
    - sleep 5

[scalvert]

Thanks! I managed to get it working. We are using a grunt task to run the tests, so I needed to bubble up the exit code to the topmost process to get this working.

[mmahalwy]

I am experiencing this problem when trying to boot up my own server in the global:

var app;

module.exports = {
  before: function(done) {
    if (process.env.NODE_ENV === 'test') {
      var server = require('../../bin/server');

      if (typeof server == 'function') {
        return server(function(serverScope) {
          app = serverScope;
          done();
        });
      }
    }

    return done();
  },

  after: function(done) {
    console.log(arguments);
    if (process.env.NODE_ENV === 'test') {
      // done();
      return app.close(function() {
        done();
      });
    }

    done();
  }
};

always exits with 0