This is a response to Hynek Schlawack's article titled "Why you should document your tests".

Hynek's recommendation

In his article, Hynek argues that you should document your unit tests, since a test may not clearly show what's actually being tested. He provides a compelling example of this, but I want to try demonstrating the same point with my own example:

def test_hash_parity():
    assert hash('a') == hash(hash('a'))
    assert hash(-1) == hash(-2)

When documenting tests, Hynek specifically recommends using docstrings instead of comments. This allows you (and your team) to enforce a test documentation policy using a tool.

Here's the same test, now with documentation.

def test_hash_parity():
    """Confirm parity with CPython's hash algorithm.

    CPython's hash algorithm always returns an int,
    and the hash of an int is always that same int
    (except for -1, which is reserved).
    """

    assert hash('a') == hash(hash('a'))
    assert hash(-1) == hash(-2)

When docstrings aren't sufficient

Hynek is 100% right that you should document your tests, but I want to add that you should document your test assertions, too. This is especially important when using parametrized test functions.

At work I frequently parametrize unit tests using pytest. When a new corner case is discovered, I simply add another test case to the list of parameters. As the number of test cases grows, it becomes increasingly likely that a test function's docstring will be reduced to "Confirm correct behavior".

For example:

@pytest.mark.parametrize(
    'text, expected',
    (
        ('a', 'a'),
        ('a1', 'a 1'),
        ('a-1b', 'a b'),
        # 15 additional test cases
    ),
)
def test_parser(text, expected):
    """Confirm correct behavior."""
    assert parse(text) == expected

Hynek explicitly points out that your function docstring shouldn't be useless, but in this situation the docstring alone isn't sufficient. Do you put comments next to each test case? Do you expand the docstring with each new test case?

Luckily, Python's assert can really help here: it supports an oft-overlooked second expression. This second expression will be used as the argument to the AssertionError if the test fails. Here's an example of the syntax:

assert int('Zz', 36) == 1295, 'int() must be case-insensitive'
assert int('-z', 36) == -35, 'int() must support negative letters'

To use this with parametrized functions, you can add an additional message argument to the list of parameters, which will put your documentation inline with your test cases. Here's the same pytest parametrized test as above, now with documentation messages:

@pytest.mark.parametrize(
    'text, expected, message',
    (
        ('a', 'a', 'letters must be accepted'),
        ('a1', 'a 1', 'positive numbers must be accepted'),
        ('a-1b', 'a b', 'negative numbers must be ignored'),
        # 15 additional test cases
    ),
)
def test_parser(text, expected, message):
    """Confirm that parse() handles known inputs."""
    assert parse(text) == expected, message

In fact, Python's unittest module supports messages, too!

class TestCompatibility(unittest.TestCase):
    def test_int(self):
        self.assertEqual(int('Zz', 36), 1295, 'int() must be case-insensitive')

Conclusion

When you're writing test cases, particularly parametrized tests, consider adding messages to your assertions. They serve as documentation when you're reading the test code and they provide helpful feedback when tests start failing.

Further reading

• pytest parametrized test functions
• assert statement