Python unit tests with nose test generators

I recently came across the case where I had to write unit tests that were very repetitive, which was very annoying. The python unit test library provides some ways to reduce repetitive code, namely for setup and teardown, but sometimes this is not enough. In this post I will go through the problem and the different approaches I learned about using a simplified example:

Assume that you specified a interface for a data structure, e.g. a set of items:

class ISet(object):
    def add(self,item):
        """
        Add an item to the set
        """
        pass

    def contains(self,item):
        """
        Check if an item is contained in the set
        """
        pass

You write an implementation for this and to make sure it behaves like you expect, you write tests:

class SimpleSetTest(unittest.TestCase):
    def setUp(self):
        self.s = SimpleSet()

    def testContainment(self):
        self.s.add("Holla")
        self.assertTrue(self.s.contains("Holla"))

    def testNotContainment(self):
        self.assertFalse(self.s.contains("Holla"))

All tests pass, great! So you try another fancy set data structure and write an alternative implementation. To extend the tests to this new implementation, I see the following possibilities

Copy and Paste

Copy paste the test code and change all places where you create instances of set from one class to the other. If you use fixtures chances are that there are not so many places, but you still have to copy the test code. Meh.

Inheritance

Use inheritance to define a base test class and override the setUp method. This is already much nicer, as it allows to reflect the inheritance structure in the tests:

class ISetTest(unittest.TestCase):
    def testContainment(self):
        self.s.add("Holla")
        self.assertTrue(self.s.contains("Holla"))

    def testNotContainment(self):
        self.assertFalse(self.s.contains("Holla"))

class SimpleSetTest(ISetTest):
    def setUp(self):
        self.s = SimpleSet()

class MoreComplicatedSetTest(ISetTest):
    def setUp(self):
        self.s = MoreComplicatedSet()

Unfortunately, this causes problems with test discovery: The base class for the test is not meant to be instantiated, but the test discovery protocoll doesn't know about this.

Fold all tests into one

Write your tests in such a way, that every test tests all implementations. You have to write a loop into every test, so this still causes boilerplate code of O(n) in the number of tests that you write, although it is only one line. The loss of the nice organisational structure into TestCases, and the less clear error analysis in case of a test failure is more problematic:

class AllSetsTest(unittest.TestCase):
    def setUp(self):
        self.sets = [SimpleSet(),MoreComplicatedSet()]

    def testContainment(self):
        for s in self.sets:
            s.add("Holla")
            self.assertTrue(s.contains("Holla"))

    def testNotContainment(self):
        for s in self.sets:
            self.assertFalse(s.contains("Holla"))

Nose test generator

Use the test generator feature of nose test generators. This allows you to generate tests functions programatically, so we can actually parametrize our tests:

def checkContainment(s):
   s.add("Holla")
   assert s.contains("Holla")

def checkNotContainment(s):
   assert not s.contains("Holla")

def test_cartesian_generator():
      for test in [checkContainment,checkNotContainment]:
            for s in [SimpleSet,MoreComplicatedSet]:
               yield test, s()

However, this is a nose-only feature and we loose the organisation into test cases. Also care has to be taken with naming: the name of the generator function has to start with test and end with generator, and the tests have a different prefix check to avoid problems with test discovery.

Summaryant

It turns out that I am a bit stupid. I actually wanted to write about class decorators, but it turns out that the example I chose to demonstrate this approach is actually not so well suited, so I will have to write another post to make my point. Luckily I have a backup point: Nose is great, use it!

Also the unittest library in python caused me a lot of pain. For example the heavy reliance on naming conventions turns this into a rather magical thing, that makes for some very surprising behaviour. And it causes the test discovery mechanisms to be very complicated, and feel not very extensible. Of course, you can hook into many phases of a test run or implement your own versions of test runners, loaders and other stuff. But all this magic makes me not want to do that. Probably I don't do the unittest library justice. In the end, it is not so ugly to use as long as you don't want to do something out of the ordinary, and I probably don't understand the reasons behind all these seemingly strange design decisions, and the ecosystem around it (like nose) is actually rather powerful. But I still find it surprisingly unpythonic and ugly.

The Pseudotron