End to End Test Metrics

End to End (E2E) testing replicates real user-facing scenarios from the point they begin interacting with your system, to the point that their transaction or experience is complete. Because customer experience can depend on the coordinated behavior or multiple interfaces, services, and databases, more localized tests may not catch issues that impact the customer. As a result, E2E tests are created from the perspective of user, who is using the software or system to realize a particular goal or outcome. An E2E test usually consists of multiple interconnected test cases, each of which completes an action that contributes to the overall goal.

When performing E2E testing, consider reporting metrics related to:

• successful business outcomes from creating and running the tests, including
  • reduced frequency and severity of bugs
  • reduced frequency and severity of customer complaints
  • longer mean time between failures (MTBF)
  • risk reduction, and
• the conduct and ROI of the tests themselves

---

Measuring Business Outcomes

Reduced Frequency & Severity of Bugs

If the frequency of new bugs (and emergency fixes) that relate to the E2E business process decrease once the E2E test is introduced and regularly executed, then the E2E test has served as an effective control for detection of new problems. Figure out each interface, service, or data store that a business process touches, and arrange to monitor the bugs per appropriate unit time (day, week, month - look at the history; if only one bug is reported per month on average, it doesn't make sense to track by day). Monitor bug frequency per component app and total bug frequency.

Reduced Frequency & Severity of Customer Complaints

If the frequency of customer complaints that relate to the E2E business process decrease once the E2E test is introduced and regularly executed, then the E2E test has served as an effective control for detection of new problems. Since bugs may be recorded when customer complaints are processed, and since complaints are usually submitted less frequently than bug reports, this is a less informative metric than bug frequency.

Risk Reduction

An E2E test can be used to reduce the risk associated with steps of a business process. This means that a baseline risk assessment needs to be in place first (the "as-is" state) so you have something to compare it to after you've completed your manual or automated E2E tests. My favorite way of doing this is with Software FailureModeEffectsAnalysis (FMEA). Take each step of the business process, and evaluate the criticality of that step in terms of Severity (how bad it will be if that process step fails, with 1=not bad and 10=very bad), Opportunity (how likely it is that the process step fails; 1=never and 10=very often, and (non)-*Detection* (how likely it is you will miss a problem if it arises; 1=we never miss it; 10=we usually miss it because we can't detect it). Multiply all those numbers together to get RiskPriorityNumber (RPN).

If running your manual or automated E2E tests reduces the RPN, the E2E test has added value by reducing the risk of that business process (as served by the software systems). This will usually happen by making it more likely to detect issues.

---

Measuring Test Outcomes

Feel free to add to this list as you discover and use additional metrics. The lean management SQDIP categorization was used to help us understand different kinds of test metrics.

• Security/Effectiveness of tests
  • Pass rate (Number passed/Number executed)
  • Compare frequency of defects between defect categories (Number defects in a particular category/Total defects; plot all in a ParetoChart to see dominant groups)
• Quality of tests - how well do E2E tests pick up defects
  • Defect detection rate (Number of defects discovered through test process/Number of defects discovered total
• Delivery - how effectively is value provided to customer (usually on-time delivery and related metrics)
  • NA
• Inventory
  • Backlog/burndown - how effectively is the test team producing manual or automated tests compared to plan
• Productivity
  • E2E tests created per unit time

-- NicoleRadziwill - 06 Apr 2020