·
8 min
Root Cause Analysis in Software Development: How to Find Why Tests Fail
Roman Kirchmeier - Autemos
Root cause analysis (RCA) in software development finds the condition that made software defects and failed tests possible in the first place. ISTQB defines it as “The process used to identify the root cause of a defect” (ISTQB Glossary, as of 2026). For QA teams it usually starts with a red test run and one question: is the fault in the product, the test, the environment or the test data? At Google, about 84% of observed pass-to-fail transitions involved a flaky test (Google Testing Blog, 2016). The analysis is one step in defect management in software testing.
TL;DR: Root cause analysis separates symptom, trigger and root cause. For a failed test, you secure the evidence first, reproduce the run and assign it to one of five categories: product defect, test defect, environment, test data or flaky test. Only then do 5 Whys and Ishikawa help.
Figure 1: Five categories for a failed test run
What is root cause analysis according to ISTQB?
RCA is an investigation method that identifies the fundamental reason for a defect. The ISTQB syllabus puts it this way: “A root cause is a fundamental reason for the occurrence of a problem (e.g., a situation that leads to an error).” (ISTQB CTFL Syllabus 4.0.1, 2024)
The syllabus describes a chain: “Human beings make errors (mistakes), which produce defects (faults, bugs), which in turn may result in failures.” It adds that environmental conditions can cause failures too. So a red automated test can point to a defect in the product or to a broken environment.
The syllabus places RCA where a failure occurs or a defect is identified. ISTQB expects that addressing the root cause can prevent similar failures or reduce their frequency. Our article on the defect life cycle covers the states up to closure.
What is the difference between a trigger and a root cause?
A trigger is the event that makes a problem visible; the root cause is the condition that made it possible. The Google SRE Workbook states: “Identifying the root causes and trigger is one of the most important reasons to write a postmortem.” (Google SRE Workbook, 2018)
Figure 2: Trigger and root cause compared
A sample postmortem in the workbook describes a longstanding input-checking bug in the Traffic Admin server that “was triggered by the manual reexecution” of an earlier action. The old bug is the root cause. The manual rerun is the trigger.
In testing, a deployment, a database update or a new browser build is the typical trigger. Google SRE observed internally that roughly 70% of outages come from changes in a live system (Google SRE Book, 2016). That is a Google figure with no published methodology, not an industry statistic.
How do you run a root cause analysis for a failed test?
An RCA for a failed automated test or a production defect follows eight practical steps, from securing evidence to regression testing. The CTFL syllabus sets out no fixed RCA procedure; this sequence comes from test automation practice:
Scope the problem. Which test, since when, in which environment, on which build? Note the impact.
Secure evidence before anyone reruns. Test report, logs, screenshots, recording, telemetry. A quick rerun often overwrites the only clue.
Reproduce. Same build, same test data, same environment. Does the failure recur always, sometimes or never?
Classify first. Assign the case to one of the five categories in the table below.
Separate trigger and root cause. Which change made the failure visible, and which condition made it possible?
Build the causal chain. 5 Whys for a linear chain, Ishikawa for several causes.
Define actions. One corrective and one preventive action, each with an owner and a date.
Lock it in. Confirmation testing for the fix, regression testing around it. The fixed defect becomes an automated test.
Which other fields a useful report needs is covered in our defect report template.
How do you classify a failed automated test?
A failed automated test falls into one of five categories: product defect, test defect, environment, test data or flaky test. This split is practitioner practice, not a standard. The syllabus supports it: reported anomalies may turn out to be “false-positive result” or a change request (ISTQB CTFL Syllabus 4.0.1, 2024).
Figure 3: Flaky tests at Google (Google Testing Blog, 2016)
Category | Typical sign | First check | Next step |
|---|---|---|---|
Product defect | Business result differs, reproducible | Reproduce manually on the same build | Defect report to development |
Test defect | Locator or assertion no longer fits an intended change | Compare UI or test script changes | Fix the test, no product ticket |
Environment | Timeout, certificate error, service unreachable | Run the test in a second environment | Ticket to operations or infrastructure |
Test data | Record missing, locked or used up | Check data state before the run | Clean up data, review data rules |
Flaky test | Same code, sometimes green, sometimes red | Repeat several times, check history | Fix the cause or quarantine the test |
At Google, “about 1.5% of all test runs” reported a flaky result, and “Almost 16% of our tests have some level of flakiness associated with them” (Google Testing Blog, 2016). Rerunning until green postpones the analysis.
A flaky test has a root cause of its own. Luo et al. (2014) found async waits, concurrency and test order dependency among the most frequent causes (Luo et al., FSE, 2014). How unstable locators produce flakiness is explained in our article on self-healing locators.
How do you apply 5 Whys and Ishikawa to a failed test?
5 Whys produces a linear causal chain for a single failure, and an Ishikawa diagram collects several possible causes by category. The Lean Enterprise Institute describes 5 Whys as “the practice of asking why repeatedly whenever a problem is encountered” (Lean Enterprise Institute, as of 2026). The method traces back to the Toyota Production System and was popularised by Taiichi Ohno (Card, BMJ Qual Saf, 2017).
Figure 4: Ishikawa diagram adapted to software defects
Example: 5 Whys for a failed payment test
This example is constructed for illustration. A nightly test “Approve transfer” fails at the confirmation step.
Why does the test fail? The confirmation page shows an error instead of the approval.
Why the error? The payment service rejects the order, the test account is locked.
Why is the account locked? A parallel run used the same record shortly before for a lock test.
Why was that possible? The record was not flagged as single-use.
Why was the flag missing? New test cases get no review of their test data.
Category: test data, with the root cause in the process; the product has no defect here. Corrective action: reset the locked test account. Preventive action: flag single-use data and review the test data of new test cases.
What does an Ishikawa diagram for software look like?
The Ishikawa diagram (fishbone diagram) goes back to Kaoru Ishikawa, and ASQ lists it among the seven basic quality tools (ASQ, as of 2026). Its classic “6 M's” come from manufacturing: materials, machinery, methods, measurement, manpower, mother nature. For software defects we use this adaptation:
Product code and configuration
Test code (locator, assertion, wait logic)
Test data
Test environment and infrastructure
Dependencies (third-party systems, APIs, browser versions)
Process (review, deployment, handovers), in place of “manpower”, in line with the blameless approach below
Where are the limits of 5 Whys?
There is little evidence that 5 Whys works. In BMJ Quality & Safety, Alan J. Card argues that the method forces a single linear chain onto complex failures, that the number five is arbitrary and that results depend on the investigator (Card, BMJ Qual Saf, 2017). He attributes its spread to “pedigree, simplicity and pedagogy”.
The example had two causes, the missing flag and the missing review; an Ishikawa diagram shows both.
What belongs in a blameless postmortem?
According to Google SRE, a postmortem is a written record of an incident, its impact, the actions taken, the root cause(s) and the follow-up actions to prevent it from recurring (Google SRE Book, 2016). To be blameless, it “must focus on identifying the contributing causes of the incident without indicting any individual or team”.
A red nightly run needs a short version in the ticket; a production defect with customer impact needs the full one. Either way, list two kinds of action separately:
Corrective action: fixes the specific case, for example a code fix, a cleaned-up record or an updated locator.
Preventive action: stops the same root cause from producing new defects, for example an input check, a review step, a new regression test or monitoring.
Many preventive actions belong earlier in the process, the subject of shift-left testing.
What does DORA require for root causes?
DORA requires EU financial entities to identify, document and address the root causes of ICT-related incidents. Article 17(2) of Regulation (EU) 2022/2554 requires procedures so that “root causes are identified, documented and addressed” (EUR-Lex, DORA, 2022). The regulation has applied since 17 January 2025.
For major incidents, Article 19(4)(c) requires a final report “when the root cause analysis has been completed”. DORA governs incidents in operations, not defect tracking in test. A software defect becomes relevant once it causes an ICT-related incident. Swiss banks follow FINMA Circular 2023/1 on operational risks, including ICT risks; DORA reaches them only through EU business. Our article on audit trail testing shows how to keep that evidence auditable.
What does Autemos contribute to RCA?
Autemos is AI-assisted software for functional test automation across web, mobile, API and desktop. It supplies the evidence for RCA; your team does the analysis itself. Autemos is not a defect tracker. It does not replace Jira or Azure DevOps and sets no severity or priority.
Self-healing logged separately: Every self-healing action appears in the report and can be reviewed, rejected or rolled back. A changed locator shows up as a test-side change, not as a product defect.
Reproducibility: The visual test workflow is the step sequence, and exported Playwright or Appium code reruns outside Autemos.
Evidence: Results land in Autemos reports and Allure; logs and telemetry can go to Splunk or Dynatrace.
Environment and data: Per-environment configuration records where a test ran. Single-use test data is flagged so parallel runs don't reuse it.
Regression test from the ticket: The AI chat drafts a test workflow from a Jira ticket. Every AI-generated step needs team approval before it runs.
Frequently asked questions
What is the difference between root cause analysis and bug fixing?
Bug fixing removes a specific defect; RCA explains why it appeared, so the team can address the cause.
Is every failed test a software defect?
No. Test code, the environment, test data or flakiness can cause a red test too.
How many whys does the 5 Whys method need?
As many as it takes to reach a cause your team can act on. According to Alan J. Card (BMJ Quality & Safety, 2017), the number five is arbitrary. If a failure has several causes, an Ishikawa diagram fits better.
Who should run an RCA?
Development and QA together, plus operations for production defects. Google SRE asks that the review names contributing causes without blaming any individual or team.
Does DORA require banks to document root causes?
For ICT-related incidents at EU financial entities, yes: DORA Article 17(2) requires that root causes are identified, documented and addressed. Defects that only appear in test fall outside it.
Conclusion
For failed tests, root cause analysis starts with secured evidence and an honest first classification: product, test, environment, test data or flakiness. 5 Whys or an Ishikawa diagram pay off after that. Separate trigger and root cause, record corrective and preventive actions apart, and lock in the fix with confirmation and regression testing.
For EU financial entities, DORA Article 17 adds a duty to document root causes of ICT-related incidents. The thinking stays with your team. A tool can keep the evidence complete and stop a self-healing repair from entering the analysis as a product defect.
Want to see how Autemos records test runs, self-healing actions and reports for your analysis? Talk to our team.
