Contract Coverage

If you write code with a sufficiently rigorous TDD process, then any code that is neither covered by your tests¹ nor mandated by your compiler, can simply be deleted. After all, according to your tests, the system will still work.

If you follow this way of writing code, you will naturally end up with a very high test coverage. Scoring between 98% and 100% instruction or basic-block coverage of your product code, as measured by a tool, is not at all unlikely. However, you will also find that you still have bugs in your code, in spite of this rigorous process and high coverage.

It turns out that covering every instruction in your program by tests, is not enough to rid it of bugs. The reason, we soon discover, is often that there are tests that we simply didn’t write — certain interactions that we did not combine. Certain behaviours that we did not specify.

If we think of our tests as an executable specification, then the bugs we encounter at 100% instruction coverage would be unspecified behaviour, and the solution is to specify a behaviour that makes sense for that given case. This, however, does not solve our problem of missing tests. We need to get into a mental state where we discover the missing tests. Discovering missing tests by discovering bugs is not good enough. We want to prevent the bugs from getting into the released product in the first place.

The best technique I have found for discovering these missing tests, is one that incorporates the writing of the API documentation as part of the TDD cycle. Updating the documentation² becomes a step alongside refactoring. I do not think it matters much whether you choose to update the documentation before or after the refactoring step, as long as you do it.

Simply writing a little bit of documentation is not enough, however. The documentation has to be comprehensive and complete. Every case, feature, limitation and behaviour must be fully spelled out. This will put you in a mental state where you try to think of all the little details that might be relevant to a potential user of the API. What happens if I pass in a null for this parameter? What happens if I call this method while my thread is interrupted? What happens if I pass in a call-back function that blocks forever? What happens if I run out of stack space, or heap space? What if I pass in an empty collection, an immutable one, or one of infinite size? Or what if something stateful has been shut down, released or closed? What happens at all conceivable edge cases?

As you write tests for new behaviours, you update the documentation to specify what happens in that particular case you tested for. And as you think up new behaviours you want to specify, you make sure to write a test that verifies that your code actually behaves that way.

The contract of the unit is its API, defined in terms of observable behaviour. As the process goes on, the contract becomes increasingly specified, and this specification is kept fully tested. This is what I mean by the words “contract coverage.” There are unfortunately no tools to measure this contract coverage, so we have to rely on gut feel here. However, the nature of the process gives me hopes that a gut feeling founded in experience will be fairly accurate in this case.

An additional benefit of discovering tests like this, is that they (the tests) become primarily concerned with observable behaviour as opposed to details of the particular implementation. There is no point in testing that the implementation works a certain way, if clients of the API can’t possibly tell either way. When tests do not rely on details of the implementation, then we are free to change the implementation in any way we like, as long as all observable behaviours of the code remain unchanged. Our tests are, in other words, less brittle and more relevant.

Adopting a rigorous TDD process meant a big step up in the quality of my code. I have used this technique cleanly and intentionally on thus far a single project, but the early results indicate yet another significant increase in code quality³. I think this is a useful adaptation of the TDD process, and worth exploring further.

The immediate downside, I should mention, appear to be an increase in the time it takes to develop software. However, this is only to be expected: not only is the technique new to me, there are also simply more tests to write, more bugs to fix (because they are discovered) and more documentation to write. Still, I think this extra time is well invested. There are those who would talk about diminishing returns as the test coverage closes in on 100%, and here I am talking about keeping on writing more tests for your code even after you have reached 100% test coverage. But a bug in your code is a bug in your code, and bugs in your code must be eliminated — that is my attitude anyway. I really don’t like bugs, and this kind of rigour helps me eliminate them.