Abstract test case

[panhania]

Sometimes we have some interface and we would like to write a generic test suite for it, so that concrete implementations just reuse it.

The problem with Python's unittest (and transitively absltest) is that the TestCase class serves as a "marker" telling the test executor that it should be executed. This is unfortunate, because abstract test suites should not run. Consider this example:

class FooTest(absltest.TestCase):

    @abstractmethod
    def new_foo(self) -> Foo:
        pass

    def test_bar(self) -> None:
        foo = self.new_foo()
        self.assertEqual(foo.bar(), "bar")


class QuuxTest(FooTest):

    def new_foo(self) -> Foo:
        return Quux()


class NorfTest(FooTest):

    def new_foo(self) -> Foo:
        return Norf()

Here, the test executor will instantiate FooTest, QuuxTest and NorfTest. However, FooTest is abstract and it is not possible to create an instance of it. There are three workarounds for this that I am aware of.

The first one is to configure the test executor to ignore specific test classes or prefixes (possible in pytest). However, this is awkward and requires modifying external configuration files.

The second one is described here. Basically, we del the base class once child classes are defined. This feels very wrong and works only if all the concrete test cases are defined in the same module.

The last one is to use a mixin approach. Instead of making FooTest derive from absltest.TestCase, we "mark" only concrete classes with it:

class FooTest:

    @abstractmethod
    def new_foo(self) -> Foo:
        pass

    def test_bar(self) -> None:
        foo = self.new_foo()
        self.assertEqual(foo.bar(), "bar")


class QuuxTest(FooTest, absltest.TestCase):

    def new_foo(self) -> Foo:
        return Quux()


class NorfTest(FooTest, absltest.TestCase):

    def new_foo(self) -> Foo:
        return Norf()

The problem here is that it doesn't work with static type checkers: FooTest now doesn't inherit from TestCase but uses methods like self.assertEqual.

This last solution seems like the only "correct" one except for the mentioned issue. Instead, Abseil could define an abstract test class and make the normal test case class implement it:

class AbstractTestCase(ABC):

    @abstractmethod
    def assertEqual(self, this, that):
      ...

    ...

class TestCase(AbstractTestCase):
    ...

Then, it would be possible to inherit from absltest.AbstractTestCase in the abstract test case, making the type checker happy:

class FooTest(absltest.AbstractTestCase):

    @abstractmethod
    def new_foo(self) -> Foo:
        pass

    def test_bar(self) -> None:
        foo = self.new_foo()
        self.assertEqual(foo.bar(), "bar")


class QuuxTest(FooTest, absltest.TestCase):

    def new_foo(self) -> Foo:
        return Quux()


class NorfTest(FooTest, absltest.TestCase):

    def new_foo(self) -> Foo:
        return Norf()

[yilei]

The most common way of doing this I have seen is, use your first approach, but put the abstract class FooTest(absltest.TestCase): definition in a separate library. It makes sense as this is a common library for testing, not concrete tests. It works since FooTest is defined in a library so it won't run.

Another solution is to leverage the load_tests protocol to filter out the test case instances of type FooTest.

Does this help?

[rickeylev]

For a commonly reused base class, moving to another module makes sense. More often I'm creating a base local to my test, though, and need to share asserts or tests; like panhania, creating a mixin by omitting the proper base class bothers me.

I've previously had the idea to have a e.g. @ignore_test decorator or something, but I like this "ignore abstract classes" idea better -- that seems much more natural and correct. Unfortunately, I think the find-classes logic is pretty deeply baked into the unittest loader? I've poked around in there a few times and its pretty tricky to selectively override small parts (at least in 3.6; I think later versions have a few changes that make it slightly more modular) without re-implementing the whole loader.

[yilei]

The "ignore abstract classes" approach works because FooTest doesn't inherit from unittest.TestCase, so no additional loader logic is needed. My concern with this is that, it requires keeping the definition in sync with unittest.TestCase thus it's difficult to maintain.

A decorator e.g. @skip_base_class("Skip reason: abstract class") could be implemented by defining a def setUpClass(cls) method on the decorated class, and raise unittest.SkipTest if it's from the base class, otherwise no-op so tests from subclasses will run.

[rickeylev]

FR in stdlib from many years ago: https://bugs.python.org/issue17519#msg388510

[yilei]

Oh I misunderstood #166 (comment), if it means to skip all abstract classes by inspecting it in the test loader like https://bugs.python.org/issue17519#msg184959 suggests (as opposed to define a AbstractTestCase suggested in OP), I like this better too.

[jerub]

Yes: I want a feature that does this, but no: I think abstract classes do not successfully implement what we need.

I think that using abc.ABCMeta style abstract base classes is a dangerous idea that will result in testcases classes being skipped when not intended.

Consider where we get the test loader to ignore any class for which inspect.isabstract returns True, such that this pattern works appropriately:

class AbstractTestCase(unittest.TestCase, metaclass=abc.ABCMeta):
  @abc.abstractmethod
  def foo(self):
    ...

class FooTest(AbstractTestCase):
  def foo(self):
    return 1

But then consider these two failure cases:

class AbstractTestCase(unittest.TestCase, metaclass=abc.ABCMeta):
  ...

class FooTest(AbstractTestCase):
  def foo(self):
    return 1

In this example, having the metaclass set up for AbstractTestCase doesn't have any effect, because an abstract class without any abstract methods is not abstract. So any testcases defined will run, setUp will run etc.

There is a second failure case, where we accidentally skip concrete classes. Consider what happens if you add a second abstract method (or misspell a method name!)

class AbstractTestCase(unittest.TestCase, metaclass=abc.ABCMeta):
  @abc.abstractmethod
  def foo(self):
    ...

  @abc.abstractmethod
   def bar(self):
    ...

class FooTest(AbstractTestCase):
  def foo(self):
    return 1

Because FooTest doesn't implement 'bar' it will be silently skipped. The tests will 'pass', your unit tests are green, and everything's fine. This ends up being unintended action-at-a-distance.

Typically abstract classes raise an exception when they're created in case-2 to tell you you've done it wrong, but when the test loader skips abstract classes, you don't get that safety.

I have an alternate approach I've prototyped for absl, which is to instead add a new decorator, absltest.skipThisClass(reason) - which skips just one specific class, and treats all its subclasses as concrete testcases. I think that will fulfill the requirements appropriately.