Conversion rate optimisation has a credibility problem. Teams run tests for weeks, declare winners, implement changes, and see no improvement in actual revenue. The test said the variation won. The business results don’t confirm it. Something is wrong — and it’s almost always the testing methodology, not the idea being tested.
The Statistical Power Problem
Most A/B tests run by in-house teams are underpowered. Statistical significance (typically p < 0.05) tells you the probability that the result occurred by chance, but it doesn't tell you whether the effect is large enough to matter or whether your sample is large enough to detect a real effect. A test that declares a winner at 85% statistical significance with 200 visitors per variant is almost certainly meaningless. Run a proper power analysis before starting any test.
As a rough guide: to detect a 10% improvement in conversion rate with 80% statistical power and a 5% significance threshold, you typically need 1,500–3,000 visitors per variant. Most smaller sites simply can’t run reliable A/B tests on their homepage.
The Hypothesis Problem
A good test hypothesis has three parts: the change, the expected effect, and the reasoning. “We believe changing the CTA button from green to red will increase clicks because our user research shows red is more attention-grabbing for our target demographic” is a hypothesis. “Let’s test a red button” is not. Without a clear hypothesis, you can’t learn anything even from a statistically significant result.
Where to Focus CRO Effort
Rather than running marginal tests across your site, focus CRO effort where the impact is highest:
- Checkout and cart abandonment — high-intent users dropping off. Small improvements compound significantly.
- Lead generation forms — field count, placement, copy, and social proof all meaningfully affect conversion rates.
- Landing pages from paid traffic — high cost-per-click traffic deserves the highest-converting destinations.
- Pricing and plan pages — one of the highest-leverage pages on any SaaS or e-commerce site.
What Actually Moves Conversion Rates
Changing button colours rarely does. What does move conversion rates: reducing form fields, adding credibility signals (reviews, logos, guarantees) closer to the point of conversion, improving page load speed on mobile, clarifying the value proposition above the fold, and adding a compelling primary CTA that isn’t “Submit” or “Learn More”.
The Multi-Variate Alternative
If your traffic volume doesn’t support A/B testing with statistical confidence, consider qualitative CRO methods: user session recordings (Hotjar, FullStory), exit surveys, user interviews, heuristic expert reviews, and heat maps. These generate actionable insights without requiring statistical significance — and they generate hypotheses that make your eventual A/B tests much stronger.
