Hypothesis-Driven Problem Solving: Test Competing Explanations
The best next test is the one that makes plausible explanations predict different observations.
How hypothesis-driven problem solving works
Define the unexplained observation, list several mechanisms that could produce it, and write what else should be observed if each mechanism is true. Choose a low-cost test whose outcomes separate the strongest hypotheses. Update confidence after the result and keep an unknown branch for explanations not yet considered.
List several explanations for the same observation. Choose evidence that would look different under each explanation.
Key relationships
- An observation can support more than one possible explanation.
- A hypothesis predicts a result that evidence can test.
- A discriminating test separates competing explanations with different predictions.
Choose evidence that separates explanations
Imagine that form completion falls on mobile devices. Three explanations fit the same observation: slow loading, confusing validation, or a field that is difficult to use on a small screen. A useful test produces different predictions for each explanation.
| Hypothesis | Distinct prediction | Useful comparison |
|---|---|---|
| Slow loading causes abandonment | Drop-off rises with measured load time before interaction | Fast and slow sessions on the same form version |
| Validation wording causes abandonment | Errors cluster at one rule and repeat after correction attempts | Sessions with and without that validation error |
| One field is hard to use on small screens | Abandonment clusters after focus on that field and varies by viewport or input method | Small and large mobile layouts with the same rule |
A broad mobile versus desktop comparison cannot distinguish these mechanisms because all three predict a mobile gap. Stage timing, error events, and field-level interaction create more information. Instrumentation should be limited to what the decision needs and designed with privacy in mind.
A discriminating test states
- What changes and what remains fixed.
- The outcome expected under each leading explanation.
- A result that would weaken the favored hypothesis.
- The next action for a positive, negative, or unclear result.
Describe the observation without explaining it
Start with what changed, where, when, for whom, and by how much. The statement form completion fell from 62 percent to 41 percent after the layout release is an observation. Users dislike the new form is already a hypothesis.
Preserve comparison groups when available. If the decline appears only on one browser, device, region, or entry path, that pattern constrains the mechanism before any experiment is run.
Generate mechanisms from different lenses
Create explanations involving measurement, inputs, process, interface, capacity, incentives, environment, and chance. Variation across lenses reduces the risk that every hypothesis is a version of the first story someone proposed.
A useful hypothesis names a mechanism. Traffic quality changed predicts different visitor characteristics. A validation defect blocks submission predicts errors at a specific step. Users are confused is too broad until it predicts a behavior or trace.
Create a prediction matrix
List hypotheses as rows and available observations as columns. Mark whether each mechanism predicts an increase, decrease, no change, or an unknown pattern. Evidence that every hypothesis predicts equally is not discriminating and should not drive the next test.
Include evidence against each explanation. Looking only for confirmation lets flexible stories absorb every result. A strong hypothesis risks being wrong in a visible way.
Choose the next test by information value
Estimate how many plausible hypotheses each result could eliminate, then consider cost, delay, reversibility, and harm. A targeted log query may separate browser failure from audience change in minutes. A full redesign may take weeks while changing several variables at once.
When possible, use a test that preserves the system state. Observation should come before intervention when an intervention would erase the evidence needed to understand the problem.
Update without pretending to calculate certainty
After the result, record which predictions matched, which failed, and what new explanations appeared. Confidence can be described as low, medium, or high when reliable probabilities are unavailable. The label should be tied to evidence, not negotiation.
Stop when the remaining uncertainty no longer changes the safe next action, not when the team feels completely certain. Record the residual risk and the signal that would trigger another investigation.
Avoid common test design traps
Changing several variables at once can improve the outcome while leaving the mechanism unknown. Testing only successful cases hides conditions associated with failure. Looking after an incident without a comparison period can mistake normal variation for a signal. Write these threats beside the test plan and reduce the most consequential one.
A negative result may reject the measurement rather than the mechanism if the test could not detect the predicted change. Confirm that the signal, time window, and sample cover the conditions in the hypothesis before updating confidence.
Connect hypotheses to a defined gap
Use the problem statement guide to keep the observation separate from the mechanisms being tested.
When the task concerns a harmful event and recurrence, continue with root cause analysis to verify the mechanism and corrective action.
Compare several explanations that fit incomplete evidence in the locked office lateral-thinking problem.