We strongly recommend automated testing whenever this approach is feasible. The test team can use a language designed for this kind of testing; for example, the National Instruments product Labview. The software should be able to read in the documented test cases, execute the test cases against the product, and finish by producing a meaningful and complete test report. It is also possible to execute the tests manually and record the actions taken, thereby creating the source for the next run of the tests to be automated.  Any requirements based testing must be automated allowing the best use of test engineer time to perform tests that cannot be automated.  As we might guess, it is nearly impossible to automate stochastic or exploratory testing but is usually reasonable to add the great ones to the automated test suite.

            One of the most difficult parts of automating testing lies in the ability to determine if we are seeing a failure. Do we look for a standard set of known failure modes or do we have a means that allows us to capture new failure modes? We suggest that anything but the expected behavior is effectively a failure mode, even if it is allegedly benign.

            Automated test suites—indeed, all test suites—should be under complete configuration management. That means we must also record the following about every piece of embedded software we test:

• Software version
• Specific parameter configurations
• Hardware version, including printed circuit board version
• Compiler name and type
• Compiler version
• Suite version