Test cases in and of themselves may not be the most important thing when it comes to testing.  However, neither is test case and requirements test coverage a trivial or inconsequential aspect of product quality. In earlier posts we discussed a multiple perspective approach to product testing and that included exploratory testing, combinatorial testing and stress testing.  Test cases are often associated with the product requirements.  The execution of those test cases confirms or refutes the achieving of those requirements.  From a project management perspective, it is arguable that the execution of these test cases fills the role of scope confirmation which is part of the closing activities for the project.  Not that this testing necessarily happens only at project closure (testing should be all along the development path in incremental builds) or solely used for project closure.

Test coverage as a percentage of the total test cases is only a part of the story.  Since this may only include the requirements testing (not the exploratory or the stress tests) defining test coverage total as just the tests conducted toward requirements is inaccurate. However, what can be an interesting metric is the percentage of test cases conducted compared to the number and severity of the faults found.  For example, let us say that we are 30% through our test cases.  Let us also say in that testing we found 10 failures of a moderate severity and 6 catastrophic failures. What we can extrapolate from that, is that our product is not fit for production and there are likely a few more failures perhaps twice as many as we presently see or even more.

We track the test case rate of execution and over lay that with the rate of problems found.

As test cases are typically mapped to product documented specifications, the test case coverage is part of our project closure activities, specifically, the contract closure.  Contract closure requires confirmation that the customer’s contractual expectations are met. We demonstrate that we have fulfilled our contractual obligations by knowing the test case coverage and the passing or failing of those test cases.  Those areas we do not have test cases mapped, we will take other actions to ensure we have met the contract requirements (reviews and evaluations).

Percent coverage of test cases is not as helpful by itself as it often excludes the exploratory and combinatorial testing volume. However, testing percent coverage, number of faults, arrival rate and severity of defect is more than a little bit of a clue as to what the remaining product quality may be.  The use of test cases (juxtaposed) to requirements is not only necessary for the product development engineers but for the project management discipline as well. This information about conformance or congruence with the project scope is necessary. It will be part of the assuring we meet the quality targets for the product / project as well as serving as proof of delivery for contract closure.