- First, divide/proportion your traffic, 50/50, 90/10, or 80/20. Note: you can split them randomly, by time of day or by source (Google prospects in Group A and Yahoo prospects in Group B).
- Decide on test stability. What does this mean? If a visitor leaves your site and then comes back, do they see the same test? Choices would be “no” stability, “session” stability, or “cookie” stability. Minimum requirements would be session stability.
- Last, decide what your sample size will be and set up your criteria for success.
To decide the perfect sample size, run a “null” test with your A/B test. (simply an A/A test, where you run the control against itself to determine where the convergence of results line up). You’ll understand the volume of traffic you need to test when the tests converge. How do you determine success? The number of conversions or sales you receive.
To ensure the test is valid, you have to know if there’s enough of a difference between the tests to declare a clear winner. Your results for the winning version should be three times larger, i.e. if A is 15, B should be 45.
Popularity: 1%
