Root Cause Analysis: Find the Mechanism, Not a Scapegoat
A root cause is a changeable mechanism supported by evidence, not merely the last mistake in a timeline.
How to perform root cause analysis
Define the measured gap, preserve a timeline, build several causal hypotheses, and collect evidence that can distinguish them. Call a mechanism a cause only when it predicts observations and a controlled change affects the outcome. Pair the fix with a signal that can detect recurrence.
Define the gap, build more than one cause, and collect evidence that can separate them. A cause is useful only when changing it changes the outcome.
Key relationships
- An observed gap describes what happened without naming a cause.
- A causal hypothesis predicts what else should be true if the cause is real.
- A corrective action changes the mechanism and checks whether the problem returns.
Define the gap precisely
State what happened, where, when, and compared with what expected condition. 'Checkout is slow' is weak. 'The 95th-percentile payment step on mobile Safari rose from 2.1 to 8.4 seconds after the October release' is actionable.
A precise gap narrows the system boundary and tells you what evidence to collect.
Build a causal graph, not a single story
Create branches for people, process, technology, inputs, and environment, but do not assume each branch contributed. Attach observations to edges in the graph. Correlation suggests where to test; it does not complete the causal claim.
Verify the mechanism
A proposed cause should predict something observable. If a new validation rule causes retries, disabling it in a controlled environment should remove the retries. If the outcome does not change, revise the model rather than defending the narrative.
Design a fix with a control
A strong corrective action changes the system and adds a signal that detects recurrence. Training alone is fragile. Validation, safer defaults, automated checks, and clear ownership usually create more durable prevention.
Protect the evidence timeline
Capture logs, samples, configurations, photographs, and witness notes before changing the system. Record time zones and collection methods. A cleaned system may be safer, but it can erase the sequence needed to distinguish initiation from detection and recovery.
Separate what was observed at the time from what was inferred later. A timeline should make uncertainty visible rather than blending reconstruction with fact.
Distinguish trigger, conditions, and safeguards
The trigger starts the event. Enabling conditions allow it to spread. Missing or ineffective safeguards fail to contain it. A retry storm may be triggered by a timeout, amplified by unlimited retries, and allowed to reach customers because a circuit breaker was absent.
Fixing only the trigger may leave the system vulnerable to the next trigger. Strong corrective action addresses the mechanism and improves detection or containment.
Verify that the correction worked
Define a leading measure, an outcome measure, and a balancing measure before rollout. For a queue fix, track queue age, completion time, and error rate. The balancing measure catches a solution that improves speed by reducing quality.
Set a review date and expected range. If the outcome changes without the predicted leading signal, the causal model may be incomplete even when the intervention appears successful.
Write a causal claim that can fail
Use a specific form: under these conditions, mechanism X changes variable Y, which produces outcome Z. Then list the observations expected if the claim is true and the findings that would weaken it. This turns a narrative into a model that can be challenged without turning disagreement into a debate about confidence.
When several causes interact, state the combination. A trigger may be harmless without an enabling condition, while the condition may remain invisible without the trigger. The corrective plan should show which link it changes and why that change reduces recurrence or impact.
Build an evidence ledger before choosing a cause
Consider an illustrative checkout delay that began after a release. The timing makes the release relevant, but timing alone does not identify the mechanism. An evidence ledger separates observations from the claims they support.
| Hypothesis | Prediction | Evidence to collect | Result that weakens it |
|---|---|---|---|
| A new validation rule creates retries | Retry count rises at the validation stage | Stage traces and retry logs by version | Retries remain unchanged when the rule is disabled in a safe test |
| A payment provider slowed | Delay appears after the provider request across versions | Provider latency and request timing | The delay occurs before any provider call |
| Mobile payload size increased | Affected sessions have larger downloads and slower parsing | Payload size, device class, and stage timing | Small and large payloads show the same stage delay |
The ledger makes disconfirmation explicit. A team can stop defending the first plausible narrative and choose the observation that separates the leading explanations. Preserve the event timeline before changing the system, then label every entry as observed, calculated, or inferred.
A correction needs three measures
- A leading measure tied to the proposed mechanism.
- An outcome measure tied to the original gap.
- A balancing measure that catches a harmful tradeoff.
Frame the event before naming a cause
A defensible investigation begins with a measurable gap, which the problem statement guide helps separate from proposed causes and solutions.
Use the Five Whys method only when each link can be supported and branches are preserved.
Apply stage-level evidence and competing explanations to the slow checkout diagnosis exercise.