Regression Testing and Scope

A brief exchange on twitter on the topic of regression testing leads me to this post.  Twitter, is not the place to really discuss this topic as there are too few characters and there are many factors that go into determining the scope of regression test.  Regression testing is the testing of portions of the product that were built previous to this iteration.  The testing is to ensure that in the addition of the new features to the product we have not inadvertently broken the product.  A good definition of regression testing can be found from ISTQB.

Risk Aversion 

One of the things we will consider in determining the scope of our regression testing will be the level at which our company is comfortable with the risk associated with the product in the field.  If our company makes medical or perhaps automotive or aerospace products, we may generally be risk averse since the consequences of our mistakes can be catastrophic to our customers. If this is the case we may adopt a policy of 100% regression testing.

Historical Record 

Our approach to regression testing may be based on our organization’s historical record with product updates.  For example, perhaps in the past, we have delivered product updates to the customer that have had defects, not only in the new release content but in the prior built portion of the product.  Consider a parameter that is accidentally altered or an error in the build process that allows a particular outdated software module to be built into the new product.  It could be a simple “fat finger” mistake that corrupts the old portion of the software.  If we have experienced many of these types of events, we may opt for 100% regression testing or as close to it as we can get.

Level of Automation

It might seem strange that level of automation may play a role in determining our approach to regression testing; it is not that strange in fact.  There are often two project-level constraints upon our test that require us to consider what level of regression testing will be performed. Those two constraints are; 1) the amount of time available for regression testing, and 2) the money (often labor) available for regression testing that does not ruin our business case.

For the first situation, we are up against the delivery deadline and must cram some 50 pounds of testing in a 10-pound amount of time to mix metaphors.  If we only have manual testing we have to decide what is most likely to cause serious problems to the customer and conduct that testing only so we can launch on time.  The second situation is the amount of money available for spending on regression testing that will not destroy the business case for the software update or change.

In both cases, automation can make this decision easier. Effective automation means people are not as heavily involved in the regression test execution.  There is human involvement in the creation of automation, setting up the specific test and interpretation of the test results.  However, if we have been building our automated regression testing suite all along as we developed the product, the investment in time has already been made.  Also, machines do not get tired, do not mind working late at night or even many hours or days in a row.  If we have significant automation we may opt to conduct a larger scope of regression testing as the impact to our team and the business case may not be significant. This is a safe approach.

Consequences of the Changes

This is really where the heart of the matter resides.  Regression testing is performed to confirm that what once worked is continuing to perform and that our updates have not caused any errors elsewhere in the product or system.  Using this as the approach, we consider the updates to the software and what features or functions would possibly be impacted by that alteration.  If the nature of our change is only one software module minimally connected to or depending upon other parts of the software the scope of regression testing may be small. If the change has a larger potential for impact, for example,  an operating system change, we will consider the consequences of that change on the software as a system and the scope of the regression testing will be larger. 

In addition to prioritizing by what was changed in the product, we consider the consequences of a failure upon the customer.  Using this approach, those grievous potential failures will be the features upon which we focus our regression test.  Our limiting the scope of the testing saves time and money, but as we are making a decision about what gets tested based upon the severity of impact .  Less severe issues will likely go undetected which means a defect can make it to the customer, at which time corrective actions must be administered to the field.

Summary

Personally, I have seen that defects can creep into the development work at any time and from any place, even into the old portions of the code. After all, we are updating the entire software release even when our intention is to add one software module or add to an existing software module.  Errors can creep in all along the delivery cycle and can impact anywhere in the product.  With the cost of automation coming down and the ability to quickly test with this automation, regression testing can be easily accommodated and cover a great range of the scope of the product with little damage to the test personnel via long hours.  The problem with this approach is you cannot decide today to automate regression testing and have it available tomorrow. The organization must have an understanding at the strategic level for developing the regression testing and pursue with a conscious and deliberate set of actions.