This is a response to Hynek Schlawack's article titled "Why you should document your tests".
In his article, Hynek argues that you should document your unit tests, since a test may not clearly show what's actually being tested. He provides a compelling example of this, but I want to try demonstrating the same point with my own example:
def test_hash_parity(): assert hash('a') == hash(hash('a')) assert hash(-1) == hash(-2)
When documenting tests, Hynek specifically recommends using docstrings instead of comments. This allows you (and your team) to enforce a test documentation policy using a tool.
Here's the same test, now with documentation.
def test_hash_parity(): """Confirm parity with CPython's hash algorithm. CPython's hash algorithm always returns an int, and the hash of an int is always that same int (except for -1, which is reserved). """ assert hash('a') == hash(hash('a')) assert hash(-1) == hash(-2)
When docstrings aren't sufficient
Hynek is 100% right that you should document your tests, but I want to add that you should document your test assertions, too. This is especially important when using parametrized test functions.
At work I frequently parametrize unit tests using pytest. When a new corner case is discovered, I simply add another test case to the list of parameters. As the number of test cases grows, it becomes increasingly likely that a test function's docstring will be reduced to "Confirm correct behavior".
@pytest.mark.parametrize( 'text, expected', ( ('a', 'a'), ('a1', 'a 1'), ('a-1b', 'a b'), # 15 additional test cases ), ) def test_parser(text, expected): """Confirm correct behavior.""" assert parse(text) == expected
Hynek explicitly points out that your function docstring shouldn't be useless, but in this situation the docstring alone isn't sufficient. Do you put comments next to each test case? Do you expand the docstring with each new test case?
assert can really help here: it supports an
oft-overlooked second expression. This second expression will be
used as the argument to the
AssertionError if the test fails.
Here's an example of the syntax:
assert int('Zz', 36) == 1295, 'int() must be case-insensitive' assert int('-z', 36) == -35, 'int() must support negative letters'
To use this with parametrized functions, you can add an
message argument to the list of parameters,
which will put your documentation inline with your test
cases. Here's the same pytest parametrized test as above,
now with documentation messages:
@pytest.mark.parametrize( 'text, expected, message', ( ('a', 'a', 'letters must be accepted'), ('a1', 'a 1', 'positive numbers must be accepted'), ('a-1b', 'a b', 'negative numbers must be ignored'), # 15 additional test cases ), ) def test_parser(text, expected, message): """Confirm that parse() handles known inputs.""" assert parse(text) == expected, message
In fact, Python's
unittest module supports messages, too!
class TestCompatibility(unittest.TestCase): def test_int(self): self.assertEqual(int('Zz', 36), 1295, 'int() must be case-insensitive')
When you're writing test cases, particularly parametrized tests, consider adding messages to your assertions. They serve as documentation when you're reading the test code and they provide helpful feedback when tests start failing.