-- DanielZrymiak - 01 Apr 2020

Medical Device Software Testing

Test Considerations

To summarize the section for Verification and Validation below, the test cases and results have to be documented in a way that traces back to and demonstrates that design and development attributes (i.e. requirements, risks, user stories) have been explicitly addressed and affirmatively verified and validated.

Test Categories

SOFTWARE TEST TYPE	INPUTS / DEPENDENCIES	TEST CONTROL DOCUMENTS
Functional Testing	User stories, functions	Test Strategy
Confirmation of Specifications	Documented specifications	Configuration Audit
Traceability to Requirements	Documented and approved Technical and Business Requirements	Requirements Traceability Matrix
Data accuracy, precision, repeatability, and reproducibility	Data Quality specifications	Data Quality Test Approach
User Needs (based on Intended Use)	Intended Use of Software/Device Clinical Classification	User Acceptance Testing Clinical Test Protocols
Operational Requirements (i.e. Clinical Use environment)	Operational Specifications	Operational Readiness Testing Portability Testing
Risk, Safety, and Hazards mitigation	Risk Profile, Hazards Identification (i.e. Failure Mode Effects Criticality Analysis	Risk-Based Testing Testing for Critical Errors
Acceptable failure and degraded modes	Potential Failure Modes Standard for Minimum Viable Product	Stress Testing
Boundary conditions, equivalence classes, and exceptions	Data classfication (i.e. Orthoganal Arrays) Representative Samples	Data determination
Regression and Change Control testing	Change Control Practices (i.e. CAB approval)	Regression Tests Pre-Change Test Protocols
Maintenance and Backwards Compatibility	Portfolio of prior versions and configurations	Maintenance Regression Testing Test of newly deployed changes Backwards Compatibility Testing

These categories are not exclusive to Medical Devices, and can be leveraged toward other types of Software Quality projects and engagements.

Extensions From Conventional Software Testing

Medical device testing exists as a supplemental layer extended from conventional software testing. Once the application has reached a level of stability and robustness, the official SaMD tests can be applied to affirmatively demonstrate bi-directional traceability and fulfillment linking test results back to risks, requirements, operating conditions, intended uses, and clinical considerations. There should be no “orphaned” tests without corresponding cross-references, and no documented requirements, specifications, or risks without an associated successful software test.

Verification and Validation

http://www.imdrf.org/docs/imdrf/final/technical/imdrf-tech-151002-samd-qms.pdf

The verification and validation (V&V) activities should be targeted towards the criticality and impact to patient safety of the SaMD, as discussed in IMDRF/SaMD WG/N12.

Typically, verification (providing assurance that the design and development activity at each development stage conforms to the requirements) and validation (providing reasonable confidence that the software meets its intended use/user needs and operational requirements) activities ensure that all elements from the SaMD design and development—including any changes made during maintenance/upgrades—have been implemented correctly and that objective evidence of this implementation is recorded.

A defined set of V&V activities should focus on the interface of the SaMD to the operating system, outsourced components, and other dependencies related to the computing platform.

Patient Safety and Clinical Environment Considerations

These V&V activities should include scenarios that cover the clinical user/use environment (usability, instructions for use, etc.). This can be accomplished, in part, through structured human factors testing using a subset of patients/clinicians. These activities should confirm that software safety elements work properly (i.e., patient safety / clinical use risk elements, etc.). These activities are also commonly included as part of user acceptance testing (UAT). Confirmation of acceptable failure behavior in the clinical environment should be established. This may include confirmation of the ability of the software to continue to operate in the specified degraded modes (e.g., fail-safe, fail-secure, or fail-soft).

Consideration of a variety of user groups to ensure software can be used by persons of different demographics.

Technology and Systems Environment Considerations

The extent of test coverage should be driven by the risk profile of the device determined by the intended use and SaMD definition statement. Interoperability of components and compatibility to other platforms/devices/interfaces, etc. with which SaMD works should be considered. Adequate coverage and traceability to the known hazard-related functions of SaMD should be provided. The coverage of boundary conditions and exceptions (robustness, stress testing, data security, integrity, and continuity of SaMD availability) should be included. Companies should employ rigorous impact analysis of changes made to SaMD (i.e., regression testing) to ensure updates do not compromise the safety, effectiveness, and performance of SaMD.

http://www.imdrf.org/docs/imdrf/final/technical/imdrf-tech-170921-samd-n41-clinical-evaluation_1.pdf

Analytical / Technical Validation of a SaMD

Analytical validation measures the ability of a SaMD to accurately, reliably and precisely generate the intended technical output from the input data.

Said differently, analytical validation:

Confirms and provides objective evidence that the software was correctly constructed – namely, correctly and reliably processes input data and generates output data with the appropriate level of accuracy, and repeatability and reproducibility (i.e., precision); and

Demonstrates that (a) the software meets its specifications and (b) the software specifications conform to user needs and intended uses.

The analytical validation is generally evaluated and determined by the manufacturer during the verification and validation phase of the software development lifecycle using a QMS.

Analytical validation is necessary for any SaMD.

References

• imdrf-tech-151002-samd-qms (1).pdf: Software as Medical Device QA Plan

• SaMD Clinical Evaluation.pdf: Software as Medical Device Clinical Evaluation