RFC: Data driven testing

[SimenB]

The one feature I miss the most from other test frameworks is a nice, clean way of testing multiple inputs and multiple outputs. I first came over this pattern when using Spock, where they call it Data driven testing. It looks basically like this (example taken from link):

def "maximum of two numbers"(int a, int b, int c) {
    expect:
    Math.max(a, b) == c

    where:
    a | b | c
    1 | 3 | 3
    7 | 4 | 7
    0 | 0 | 0
}

This provides a very concise and readable test, perfect for testing pure functions without resorting to looping through values manually.

In Jest, I think the best way would be to implement it similar to how jest-each works today:

jest.each([
  [1, 1, 2],
  [1, 2, 3],
  [2, 1, 3],
])([a, b, c]) => {
  expect(add(a, b)).toBe(expected);
});

Potentially by parsing a template string, so it's closer to Spock's implementation:

jest.each`
  a | b | expected
  0 | 1 | 1
  1 | 1 | 2
`(({a, b, expected}) => {
  expect(a + b).toEqual(expected);
});

(notice the destructuring matches the names in the headers)

jest-each allows us to define custom test names, which I think is a good idea. If not, we basically have this:

[
  [1, 1, 2],
  [1, 2, 3],
  [2, 1, 3],
].forEach(([a, b, expected]) => {
  test(`${a} and ${b} is ${expected}` => {
    expect(a + b).toBe(expected);
  });
})

I think the big benefit with supporting it out of the box in Jest is that we can provide static analysis of it, and maybe better messages. We might even be able to hook it into fuzzy testing or other improvements.

It might just boil down to simple sugar around [].forEach and a template string, but encouraging the pattern by having built-in support might improve developer's test suite.

---

There is also babel-plugin-gwt, but I don't think introducing custom syntax is something we want to do in Jest core. If we do implement this feature though, that plugin can target whatever mechanism we come up with here, for a potentially better interop.

/cc @mattphillips

[mattphillips]

I second this! Data driven testing is something I've wanted in JS (especially in Jest) for a while.

As you can see I've tried to make it possible with jest-each and babel-plugin-gwt, it's hard to find the right balance between being intuitive, not verbose and expressive enough to make dynamic, but unique test titles.

I really like the suggestion of using tagged template literals to try to mimic Spocks data tables API - without the need to do it at a compiler level like in babel-plugin-gwt.

My only concern with using tagged template literals is everything will be a string and the underlying library (possibly Jest) will have to convert them into actual types.

I've just had a play with jest-each (over here) to change its API to use tagged template strings, with one requirement - all data must be passed in as interpolations. This way the library doesn't care what the data is. Here is an example:

describe('.add', () => {
  each`
    a    | b    | expected
    ${0} | ${0} | ${0}
    ${0} | ${1} | ${1}
    ${1} | ${1} | ${2}
  `.test('returns $expected when given $a and $b', ({ a, b, expected }) => {
    expect(a + b).toBe(expected);
  });
});
↓↓↓↓↓↓↓↓↓↓

I'd be more than happy to build this as part of Jest's core, if approved and once an API is agreed 😃

[thymikee]

Hm, I dont't think I'm not sold on template strings for that one, as I'd need Prettier to support formatting it :D

[SimenB]

My only concern with using tagged template literals is everything will be a string and the underlying library (possibly Jest) will have to convert them into actual types.

Yeah, didn't think of that. I think it makes sense, if not we'd have to call JSON.parse or something on every value.

I'd be more than happy to build this as part of Jest's core

That's awesome! @cpojer asked me to write this up, so hopefully he'll chime in with his thoughts on the API.

I'd need Prettier to support formatting it :D

Yeah, prettier support for the template string would be awesome. It's great to hit format in IntelliJ and have the tables in Spock magically align. Could probably reuse whatever logic Prettier has for tables in markdown

[cpojer]

Yeah, I'm wondering if the initial example with jest-each is enough and whether we should just ship that as part of Jest? I'm not sure what the value of naming the fields in the template literal is when you have to name the variables separately in the callback function.

[mattphillips]

@cpojer one nicety of naming the fields in the template String is that we can use the name to interpolate the test title with the value of the field - I agree it can seem a bit verbose to have the names both in the template String and in the callback.

[SimenB]

Starting out with jest importing jest-each might make sense then. With the added support of template strings. Then we can look into variable names etc later if it makes sense

@mattphillips I think that's a good starting point for a PR 🙂

Should it be jest.each or expect.each? I'm thinking jest.each as it will implicitly depend on globals exposed by jest

[cpojer]

I think expect.each makes more sense for now.

[mattphillips]

I think it would make more sense to be a property on test (and any aliases): test.each.

If we put it on expect users may think that they can use it inside of a test body.

jest-each currently wraps test and exposes it as a property on each:

describe('.add', () => {
  each([
    [0, 0, 0],
    [1, 0, 1],
    [2, 1, 1]
  ]).test('returns %s when given %s and %s', (a, b, expected) => {
    expect(a + b).toBe(expected);
  });
});

If we exposed jest-each inside of Jest's core on test we could write the tests like:

describe('.add', () => {
  test.each([
    [0, 0, 0],
    [1, 0, 1],
    [2, 1, 1]
  ])('returns %s when given %s and %s', (expected, a, b) => {
    expect(a + b).toBe(expected);
  });
});

[SimenB]

#6102