Add validation for Ember.onerror in testing.

[rwjblue]

Ember.onerror intentionally is allowed to swallow errors, this is a very useful tool when in production to ensure that your application can be a tad bit more resilient to things like network issues and whatnot.

However, having an improperly configured Ember.onerror can lead to severe issues where your tests are passing but things in reality are completely broken.

This change leverages the validateErrorHandler utility function added in emberjs/ember-test-helpers#247 to automatically add a test that will fail if Ember.onerror does not rethrow the error it is provided. Importantly, this new test adding behavior is on by default (in order to help folks that are unaware they are in a broken state), but can easily be disabled (see below for details).

Example of Valid Ember.onerror

An example of a properly setup Ember.onerror is the following:

// app/app.js
import Ember from 'ember';

Ember.onerror = function(error) {
  if (Ember.testing) {
    throw error;
  }

  // ...desired development and production behaviors here...
};

Testing Production-like Behavior

In many cases, you will actually want to test that the swallowing does occur and the app is able to recover (e.g. in an acceptance test that intentionally rejects one of your network requests). To make that scenario easier to handle, we have created ember-test-friendly-error-handler which exposes a very simple API to allow specific tests to opt-in to the "production" behavior. That might look like:

// app/app.js
import Ember from 'ember';
import buildErrorHandler from 'ember-test-friendly-error-handler';

Ember.onerror = buildErrorHandler('Ember.onerror', (reason) => {
  reportErrorToService(reason);
  // whatever else you might want here...
});

// test/acceptance/something-test.js
import { module, test } from 'qunit';
import { setupApplicationTest } from 'ember-qunit';
import { visit } from '@ember/test-helpers';
import { 
  squelchErrorHandlerFor,
  unsquelchAllErrorHandlers
} from 'ember-test-friendly-error-handler';

module('some good description', function(hooks) {
  setupApplicationTest(hooks);

  hooks.afterEach(() => {
    unsquelchAllErrorHandlers();
  });

  test('network failure message is displayed', async function(assert) {
    squelchErrorHandlerFor('Ember.onerror');
    
    await visit('/some-route-here');
    
    // assert things here...
  });
});

Opting Out of Validation

In general, this is a bad idea. The validation exists specifically to protect applications from completely invalid test suites.

If however, you have good reason to disable validation you can opt-out by setting the setupEmberOnerrorValidation option when calling ember-qunit's start() method in tests/test-helper.js:

import Application from '../app';
import { setApplication } from '@ember/test-helpers';
import { start } from 'ember-qunit';

setApplication(Application.create({ autoboot: false }));

start({ setupEmberOnerrorValidation: false });

[argote]

Is there a way to disable validation on a per-test basis? Seems this conflicts with using onerror to test for expected errors in helper rendering integration tests.
See https://discuss.emberjs.com/t/how-to-catch-errors-in-component-rendering-test/14854

module('Integration | Helper | my-helper', function(hooks) {
  setupRenderingTest(hooks);

  test('it throws error and renders nothing when null passed in', async function(assert) {
    this.set('inputValue', null);

    const done = assert.async();
    Ember.onerror = function(error) {
      assert.ok(error.message.includes('null passed in'));
      done();
    };

    await render(hbs`{{my-helper inputValue}}`);

    assert.equal(this.element.textContent.trim(), '');
  });
}

[rwjblue]

You should add an afterEach to reset Ember.onerror, otherwise all of your other tests will be doing that assertion (and swallowing errors) whenever an error happens. This scenario is exactly the kind of thing we are trying to catch with this assertion.

[argote]

Thanks! That's actually what I ended up doing, however I was not aware that the Ember instance is shared across all tests. I was expecting each test to be atomic and reset all Ember values to the default.

[ppcano]

If the following test is executed at the end of our test suite, Ember.onerror will be reset and the ember-qunit: Ember.onerror validation: Ember.onerror is functioning properly test fails.

import { test, module } from 'qunit';
import { setupTest } from 'ember-qunit';

module('xxx', function(hooks) {
  setupTest(hooks);

  test('', function() {
    ...
  });

});

Perhaps, there is an expected configuration in our app or test suite. I am asking for the recommended way to fix this situation; I ended up using a custom setupForTest which is a setupTest wrapper:

import Ember from 'ember';
import { setupTest } from 'ember-qunit';

const setupForTest = function(hooks) {
  setupTest(hooks);

  hooks.beforeEach(function() {
    Ember.onerror = function(error) {
      throw error;
    };
  });

};

export default setupForTest;