Code Coverage Does Not Imply Test Coverage

A code coverage tool watches your program executing and reports which lines of code were executed and which were not. Testers are sometimes tempted to use code coverage tools to assess test coverage. And some testers are tempted to set code coverage goals. If you feel these temptations, be careful how you interpret the code coverage tool's reports.

You can be sure that if a line of code was not executed during a test run, then it certainly was not tested by that run.

But what of a line of code that was executed by the tests? Unfortunately, you can't tell, just from the fact that it was executed, whether the line was tested.

Elisabeth Hendrickson and I developed a workshop on unit testing. The work of the workshop centered on a small application we had written, a rudimentary HTTP server. Our initial code had exactly thirteen tests, just enough to illustrate a few basic tools and techniques that we'd be teaching in the workshop.

When we ran a test coverage tool called NCover to watch our test suite, it reported that our thirteen tests executed 65 percent of the server's code. Does that mean that we achieved 65 percent test coverage? Not on your life. Our thirteen tests barely scratched the surface of the responsibilities of even our very simple HTTP server.

If our tests tested so little, why was code coverage so high? Because though we our suite tested little of the code, it executed a lot of the code.

For example, one of our tests sent a GET request to the server and evaluated the response. As the server executed the request, it called a logging function to log information about the request and its response to a file. The logging function was minimal, and did not deal with any of the zillions of possible file system errors it might encounter. It expected the happy path, and nothing but the happy path. So this one test, which did not in any way assess the logging feature, executed all of the logging code. The logging code was 100 percent executed and zero percent tested.

Code coverage does not imply test coverage. If you use code coverage tools to help assess your test coverage, keep that in mind.