Transparency in Embedded-System Development and Testing

I was part of a fireside chat at the automotive testing expo on transparency.  The chat was about transparency, and when I think of that I think of Feynman’s Cargo Cult Science lecture.  Of course I included this in my chat.  I was surprised to find that almost nobody in the audience had heard of Richard Feynman.  My expectation was the audience was a mass of engineers and other science aficionados.  Add to that a physics student at local university was walking around and heard me mention Feynman and chatted with me after the event.

But Why

The reason this lecture is noteworthy is where the chat goes to scientific integrity.   Fact over implication.  To me this is akin to transparency, or to having this level of integrity requires transparency and honesty.  No matter what you want to call it, the testing of the product requires these principles.  We have long held that testers and testing should be treated as productive challenges to the development assertion, yeah verily the product sound and of good quality.

Central argument

Embedded systems can pass every planned test and still fail after integration. Hardware, software, calibration, networks, sensors, actuators, and operating environments interact in ways that isolated testing may not reveal. Feynman’s ideas about scientific integrity provide a useful standard: disclose not only the evidence supporting a conclusion, but also the assumptions, limitations, anomalies, and competing explanations that could make the conclusion wrong.

1. Feynman’s standard of scientific integrity

Feynman argued that scientific integrity requires more than reporting correct numbers. Investigators should reveal everything that might weaken or invalidate their conclusions, including alternative explanations, experimental limitations, and contradictory evidence.

His warning—“you must not fool yourself—and you are the easiest person to fool”—is particularly relevant to embedded-system development.

An embedded-system team can unintentionally fool itself when:

• Software passes because the simulated environment did not reproduce actual vehicle timing.
• An ECU passes bench testing without the network traffic present in the integrated system.
• A calibration works under nominal conditions but fails near temperature, voltage, or timing limits.
• A fault cannot be reproduced, so it is classified as an anomaly rather than investigated.
• Successful HIL or SIL results are treated as proof of vehicle-level performance.
• A software update invalidates previously completed verification, but the existing results remain attached to the release decision.

Scientific integrity requires the team to explain what the test demonstrated—and what it did not.

2. Honesty, openness, and transparency in embedded development

These terms are connected but serve different purposes.

Honesty

Honesty means that requirements, results, defects, and test status are reported accurately.

Examples include:

• Not changing acceptance criteria after seeing the results.
• Not classifying a failed test as incomplete merely to protect the pass rate.
• Not claiming coverage for conditions that were simulated but not physically tested.
• Not representing an intermittent fault as resolved when it can no longer be reproduced.
• Not treating a temporary workaround as a verified corrective action.

Honesty is essential, but an embedded-system report can contain no false statements and still omit information that would change the release decision.

Openness

Openness is the willingness to expose the design, and to a very real degree one’s thinking processes and the conclusions to questions.  It is not easy leading with one’s metaphorical chin.

Embedded development requires openness across:

• Hardware engineering.
• Software and firmware development.
• Systems engineering.
• Cybersecurity and functional safety.
• Test and verification.
• Manufacturing.
• Suppliers.
• Vehicle integration teams.

An open team permits someone to ask:

• What assumption supports this requirement?
• What happens if this message arrives late, out of sequence, or corrupted?
• Which component owns the system response?
• What evidence connects this software build to the tested hardware?
• What happens when multiple faults occur simultaneously?
• What evidence would prove our current conclusion wrong?

Openness makes technical challenge possible without treating the challenge as a personal or organizational attack.

Transparency

Transparency puts honesty and openness into practice. It provides timely access to the evidence and context required to evaluate a conclusion.

For embedded systems, transparency should connect:

• System and software requirements.
• Interfaces and network definitions.
• Hardware revision.
• Software and firmware build.
• Calibration and parameter set.
• Bootloader and operating-system version.
• Test procedure and automated script version.
• Test equipment, models, and simulation assumptions.
• Environmental and electrical conditions.
• Raw results and transformed data.
• Anomalies and defects.
• Corrective actions and regression results.
• Release decision and residual risk.

Without this chain, a passing result may be accurate but no longer applicable to the product being released.

3. Transparency requires leaning toward disclosure

Feynman’s standard creates a positive obligation to reveal evidence that might weaken the preferred conclusion. In embedded testing, this means reporting more than pass or fail.

A transparent test report should identify:

• The exact hardware, software, firmware, and calibration configuration tested.
• Assumptions made by SIL, MIL, HIL, simulation, and digital-twin models.
• Differences between the model, test bench, subsystem, and actual vehicle.
• Requirements or operating conditions that were not tested.
• Tests that were waived, shortened, deferred, or modified.
• Unexpected resets, timing deviations, communication errors, or diagnostic events.
• Intermittent failures that could not be reproduced.
• Data that were filtered, removed, or classified as invalid.
• Measurement-system accuracy, uncertainty, sampling rate, and limitations.
• Defects accepted for release and the reasoning behind the decision.
• Changes made after testing that could invalidate previous evidence.

This is especially important because embedded failures frequently emerge at interfaces. Every individual component may meet its specification while the integrated feature still fails.

A transparent conclusion should therefore say:

The system passed these tests in this configuration, under these conditions, with these limitations and remaining uncertainties.

That statement is more useful than simply declaring the system “validated.”

4. What transparent embedded-system testing looks like

Begin with a falsifiable hypothesis

Define the expected system behavior and the evidence that would contradict it.

For example:

The braking-control ECU will receive, process, and respond to the requested torque command within the specified time under nominal and degraded network conditions.

The test should then identify timing, network load, voltage, temperature, sensor validity, competing traffic, and fault conditions that could disprove the hypothesis.

Maintain a known system configuration

Record the complete configuration of the item being tested:

• ECU hardware revision.
• Application-software build.
• Bootloader version.
• Calibration dataset.
• Network database.
• Sensor and actuator revisions.
• Supplier component versions.
• Test-script and tool versions.

Configuration management is what connects the evidence to the product.

Test across levels of integration

Transparency requires distinguishing evidence produced at different levels:

1. Unit and software-component testing — verifies individual logic.
2. MIL and SIL testing — evaluates algorithms and software against models.
3. HIL testing — examines ECU behavior with simulated inputs and loads.
4. Bench and subsystem testing — introduces physical hardware and interfaces.
5. System and vehicle integration testing — evaluates interactions among ECUs, networks, sensors, actuators, and operating conditions.
6. Field and operational evidence — reveals behaviors that controlled environments may not reproduce.

Success at one level does not automatically demonstrate success at the next.

Use multiple testing approaches

Transparent verification should show how and why we considered multiple approaches.  We have written much about this starting Testing Complex and Embedded Systems, up to recent article for IEEE Reliability Magazine and more.

• Compliance: Did the system meet defined requirements?
• Combinatorial: Which combinations of states, configurations, and inputs were evaluated?
• Stochastic or exploratory: What unexpected behaviors emerged when the system was exercised beyond scripted cases?
• Extreme: What happens near environmental, electrical, timing, processing, or communication limits?
• Attack: How does the system respond to intentional misuse, malformed messages, cybersecurity threats, and fault injection?

Reporting only compliance testing can create a false impression of complete coverage.

Preserve anomalies and contradictory evidence

Unexpected results should remain part of the evidence even if they cannot be reproduced immediately. Logs, traces, memory contents, network traffic, environmental conditions, and system configurations should be preserved.  We saw how modern tools and the use of AI can help quickly find clues to these rather than pour over so many data traces.

An intermittent failure that disappears is not necessarily a resolved failure.  From experience it can be that the test itself may have been the issue, and not the product, but also from experience this should not be the default setting.  The more severe the event, the more important to explore why and what.

Make residual risk visible

The release decision should identify these things below. Doing so gives us a basis for our decision. A basis greater than intuition.

• What has been demonstrated.
• What has not been demonstrated.
• Which assumptions remain unverified.
• Which defects remain open.
• Which operating conditions present uncertainty.
• Who accepted the residual risk.
• What monitoring, containment, or follow-up testing is required.

Conclusion

Embedded-system transparency is not simply sharing test reports. It is maintaining an evidence chain from the requirement and risk through the configuration, test environment, result, anomaly, corrective action, and release decision.

Honesty prevents the evidence from being falsified. Openness allows the evidence and assumptions to be challenged. Transparency ensures that the people making decisions can see the complete technical story.

A passing ECU test is evidence—not proof. The real question is whether the evidence remains credible when the ECU becomes part of the complete vehicle system.

 

Design. Build. Learn. Improve.

 

For more information, contact us:

The Value Transformation LLC store.

Follow us on social media at:

Amazon Author Central https://www.amazon.com/-/e/B002A56N5E

Follow us on LinkedIn: https://www.linkedin.com/in/jonmquigley/

https://www.linkedin.com/company/value-transformation-llc

Follow us on Google Scholar: https://scholar.google.com/citations?user=dAApL1kAAAAJ