Think Scientifically
When you are running an A/B test, think of yourself as a scientist conducting an experiment. Strategies described in the Scientific Method apply very accurately in this situation. Before creating the test in Google Analytics or writing any code, ask yourself:
- What problem am I trying to solve? (Observation)
- What changes do I think will improve this? (Hypothesis)
- What effects will my changes have? (Prediction)
- What methods will I use to test my hypothesis? (Experiment)
- How will I know if my changes are successful? (Conclusion)
Once you’ve answered these questions for yourself, you can begin the process of building and tracking your tests.
Be Patient
A/B testing takes time and initial results do not always indicate the final winner. I recommend running a test for no less than two weeks to ensure that any anomalous blips in the data don’t skew your final results. After all, you may find that weekend traffic behaves very differently than weekday traffic and by ending the test too soon you won’t get the whole picture.
Additionally, you may find that after a winner is eventually declared that the increase in gains are rather small. It’s easy to lose heart when seeing this, but remember that if your site is already performing well you won’t see massive changes overnight. Optimization is a process which happens over time, and the goal is to iterate on these changes over time.
Go Big!
As we discussed earlier, it’s important remember that we can iterate on our changes over time. When beginning a new series of tests, think broadly at first then narrow your focus over time.
For example, if you would like to test the layout of a landing page start by testing two different layouts using the same content and imagery. After you’ve found your initial winner, tweak its layout further and test that change compared with the winner.
Don’t Waste Time
Every day is an opportunity to test something new and any day without an active test running is a missed opportunity. While it’s important to avoid simultaneous tests within the same workflow to avoid data confusion, you can test different portions of your site at the same time.
For example, while you are running a test effecting the layout of your checkout page you can also be testing a PPC landing page or contact form.
For more information on user experience planning, read this blog post: The Importance of an Ongoing UX Strategy
Subscribe via RSS
