Not a Verification Problem

I recently had a flash of a project from the past. The project had a fixed delivery date.  The project was to deliver a system through iterative and incremental deliveries. Sounds pretty good right? An iterative and incremental delivery of sub systems and components to the verification group in a way that allows for controlled testing, opportunity learning about the systems and the product, and make adjustments to the product based upon that learning.  In this case, the verification had gone through a number of time consuming loops.  The system was immense, and each test phase took about 30 days.

Iterative and Incremental Deliveries

As each increment of the system iteration shows no great improvement in quality, an executive goes to the verification manager asking him to speed up the testing.  The verification manager explains that the rate of test case execution will improve as more automation of the test cases that are requirements based can be scripted.  Once all of the content is put into the product, we would expect that with each test loop and subsequent development correction, would show a decrease in the number of faults or bugs reported.  In this instance we do not see a continued reduction; on the contrary there are new bugs arriving and recurring old bugs as well (not shown and severity not illustrated).  Even when no new content is added to the software we still see a growth in bugs found.  This is not likely a verification problem but a product development problem. 

Product Development has a Role to Play

We should probably go look at how the product development is executed.  There could be control issues with build management or configuration management that allows a once working piece of software to cease to function. There are clues to be found in seeing an open bug, closed only to be opened again in a subsequent build.  It could be issues with the test case execution, and that should be (and was) explored and ruled out.  At that point pressuring the test group is not an appropriate course of action.  The real problem is not where you witness the symptom which is in testing.  This is similar to bailing the water out of a sinking boat at a slower rate than the water is entering.

Erratic Defect Arrival Rate

Predictable?

Consider the chart below. This is more like what we would like to see. We see that every build as each subsequent delivery eliminates some of the failure sources; it appears as if we are not injecting a plethora of other failures.  Our over all trend is negative sloped meaning we are improving the product in each of the iterations, putting us in a position to be able to predict when ready for launch.  It appears as if the testing and the product development portions of the team are working quite well.

Defect Arrival Rate that is consistent

Conclusion

Defect arrival rate can inform us on much more than just the state of the product.  If we are paying attention to the entire picture we will know where to put our effort for the best results.  Recurring defects, solved then reappearing and solved and reappearing informs you there is more to what is happening in verification.  Product development goes hand in hand with testing and verification.  Just because you discover the problem in test, does not mean you have a testing or verification problem.  Refusing to entertain that the problem can be somewhere other than testing impedes progress and will mean resolution of the real problem will not be prompt.

Testing of Complex and Embedded Systems