When moving from the traditional waterfall process with manual testing towards BDD with automated execution of scenarios, one of the often asked questions is “how do you measure your test coverage?”. What it really wants to say is “we have high test coverage with manual testing, can you guarantee the same or more with the BDD?”. From there on usually comes the suggestion to transfer one-to-one manual tests to the BDD format. That’s not a good idea since the result would be less than adequate transformation to the different format without any tangible change. BDD is not only about automation but about the way we construct and deliver software.
Code Coverage Analysis
Main usages of code coverage analysis is to:
- Find areas of software under test that were not exercised by a set of test cases.
- Help in creation of additional test cases to increase the coverage.
- Provide quantitative measure of test coverage as indirect measure of quality.
- Identify redundant test cases that do not increase coverage
- Identify parts of the code that is not in use
Code coverage is a way of ensuring that tests are actually testing the code. It helps ensuring the quality of tests, not the quality of the product under test. All tests might pass successfully but if the code coverage is low, the quality of tests is low. They do not cover sufficient percentage of the code. The bigger the percentage of the code passed through while testing, the bigger the probability of the absence of bugs.
Big projects might require analysis to be done after different types of tests are run: manual tests, automated unit tests, BDD functional tests, etc. Unit tests tend to be fast to execute. On the other hand, manual tests might take hours, days or even weeks to run. Test coverage analysis should be enabled during the whole test iteration.
Testing processes and code coverage
Two most common testing processes are black-box and white-box testing. While code coverage is mostly used with white-box testing, with the shift from the waterfall process and team members working in silos towards shorter iterations, cross-functional teams and the increase in automated functional testing (i.e. ATDD and BDD), the usage of code coverage has been increasing within black-box testing. While testers continue to design and run tests as white-box, later on code coverage review can bring up which areas were missed or not explored enough. In those cases code coverage review should be done as a joint venture with developers.
Another misconception is that code coverage should be used only with automated tests. While it is true that in many cases it is run together with automated unit tests, there are no restrictions. We strongly recommend that as many tests are automated as possible but the reality is that many companies have testing departments that are predominantly “old school” with manual black-box testing as major practice. No matter whether test are automatic, manual or a combination of both, code coverage can and should be used in all of those cases.
Code coverage metrics
There are different metrics that can be used to determine how well the code is covered with tests. They might be different from one framework to another. Some of the commonly used metrics are how many classes, methods, functions, instructions and branches were missed.
It is recommended to start working towards reaching desired coverage percentage of simple metrics like classes, methods and functions. Reaching 100% coverage would be relatively easy and it provides a good base for the fulfillment of desired coverage of other, harder to accomplish metrics like instructions and, finally, branches. As a general rule, start small and easy and move towards bigger and more complicated metrics.
One of the commonly asked questions is whether 100% is the desirable code coverage. There is no one fits all type of the answer. It depends on the size of the project, the way existing code is written, maturity of tests and many other aspects. While risking to generalize too much, classes, methods and functions should be covered 100%. Some instructions might prove too complicated to reach from tests without providing enough benefits. Around 95% of instructions coverage should be acceptable in most situations. Branches are most complicated to work with due to the sheer number of combinations they might produce. 90% of branches coverage can be considered as a worthy goal.
More important than numbers and percentages provided with different metrics is the review of results. It is OK to skip covering some instructions or branches as long as that was a conscious decision made during the review of the results.
- Black-box vs White-box Testing
- Quality Assurance is not Quality Checking
- Behavior Driven Development (BDD): Value Through Collaboration (Part 1: Introduction)