By: Jon M. Quigley

Introduction

We watch Air Disasters on The Smithsonian Channel. You should watch the program to understand the root cause analysis. It is not just about root cause analysis but also risks and how circumstances and events stack up to produce unwanted results. A recent episode I watched was about Atlantic Southeast Airlines Flight 2311, a turboprop airplane that had an accident coming into land.  In this case, it was not just the root cause of interest, but implications extended to testing.

Product development, lab testing, and real-world testing are critical phases ensuring a product’s functionality, safety, and reliability. While both methods have merits, understanding their differences can significantly impact the product’s success in the market.

What is Lab Testing?

Lab testing involves evaluating a product under controlled conditions within a laboratory setting. This process allows engineers to simulate various scenarios and stress conditions the product might face in the real world.

Benefits of Lab Testing

• Controlled Environment: Variables can be precisely managed to ensure consistency.
• Repeatability: Tests can be repeated multiple times under the same conditions.
• Early Detection: Potential issues can be identified early in the development process.

Downsides of Lab Testing

Despite its advantages, lab testing has several downsides:

• Lack of Realism: Lab conditions may not accurately reflect real-world environments.
• Over-Simplification: Complex interactions and unpredictable factors are often simplified.
• Cost and Time: High costs and lengthy setup times can delay product development.

Now, to return to the show, Air Disasters. I do not want to give too much away here, but the engines were tested in a lab. The evidence uncovered in the investigation (root cause analysis) seemed to point to a failure that could not happen according to the controls and the testing results. The lab test was executed again to explore the hypothesis, and the system passed.  To not give too much away and to state the obvious, mounting an engine in a lab is not the same as mounting it on a plane, the ground is not the same thing as flying through the air.

Steps to Creating Lab Testing of the System

Creating effective lab testing for a system involves several key steps:

1. Define Objectives

Clearly outline what you aim to achieve with the testing, including identifying the specific aspects of the product you want to evaluate.

2. Design Test Scenarios

Develop scenarios that simulate real-world conditions as closely as possible. Consider various stress factors and environmental conditions.

3. Set Up Equipment

Ensure you have the necessary equipment to carry out the tests. This includes measurement tools, simulation devices, and safety gear.

4. Conduct Preliminary Tests

Run initial tests to ensure the setup works correctly and to refine test procedures if necessary.

5. Execute Tests

Carry out the tests according to the predefined scenarios. Record all data meticulously for analysis.

6. Analyze Results

Evaluate the data collected to identify any issues or areas for improvement. This step is crucial for understanding how the product performs under test conditions.

7. Iterate and Improve

Adjust the product and the testing process as needed based on the test results. Repeat the tests to validate improvements.

Real-World Testing: Complementing Lab Testing

Although lab testing is essential, it should be complemented by real-world testing. This involves evaluating the product in actual environments where customers will use it. Real-world testing helps identify issues that lab testing might miss, such as user interaction problems and unforeseen environmental factors.

Benefits of Real-World Testing

• Authentic Feedback: Provides genuine user feedback and uncovers real-world issues.
• Holistic Evaluation: Tests the product in various conditions and use cases.
• Market Readiness: Ensures the product meets customer expectations and is ready for the market.

Comprehensive Approach

We have long dwelled on a multifaceted approach to testing.

The Test, Inspection & Evaluation Master Plan Organized adds an extra method to the Test and Evaluation Master Plan [TEMP—specified in IEEE 1220 and MIL-STD-499B (Draft)] to support product quality. TIEMPO expands the concept of a Test and Evaluation Master Plan by focusing on staged deliveries with each product/process release  a superset of the previous release.  

Additionally, we have another IEEE article on the Five Approaches to Product Testing.

Conclusion

Balancing lab testing with real-world testing is crucial for developing a reliable and successful product. Lab testing provides a controlled and repeatable environment to detect early issues, real-world testing offers insights into how the product performs under actual conditions. Understanding the limitations and benefits of both methods, product developers can ensure their products are robust and user-friendly.  Our advice regarding product testing is to start at the lowest increment and components and work toward the entire system. Systems testing in a lab tells us something, but we remind the reader,

“A ship in a harbour is safe, but that is not what ships are built for” ~ often attributed to John A. Shedd

Carefully consider whether lab testing is the best approach to testing the system.  The real world is much more complicated than our simulation models and lab testing for those controllable attributes.