A/B Testing: which versions to compare in a follow up test

Question

I'm having a discussion about running a follow up test after an A/B test that didn't yield a significant result. So we have tested an existing control page (C) with a horizontal layout versus a new variation (V1) with a column layout. After two months of execution, the variation V1 has a revenue that is 2% higher than the control page C, but the chance to beat original is only 58%. So we decided to stop the test, because it seems very unlikely to get a significant result.

Now we want to modify the variation and start another test. My idea is to create a new variation (V2) and test it against the old control page (C): C vs. V2.

But my coworker rather wants to test the better performing V1 variation against another variation: V2 vs. V1.

What is the proper way to handle this? Should we test C vs. V2 — or V2 vs. V1?

Answer 1

The 1st followup should be: What makes the 2% revenue bump in V1?

• Was it caused by a particular element being clicked?
• Is it because a page/link being visited more frequently?

Based on what you've learned from the question, you can apply what you've learned to either your control or V2.

It's important to keep iterating (C > V1 > V2), and it's equally important to analyze the tests so we can learn from them!

Answer 2

Sounds like the change you made on C to create V1 did not have a statistically significant impact on the conversion rate.

Based on how much traffic you have on your site, there are two options for how you can set up your next test:

• If you do not have a lot of traffic to your site, you can compare V2 to either C or V1 (whichever you choose is subjective and can be based on your own preference).
• If you have a lot of traffic to your site (which it sounds like you may not if it took two months to confirm a test wasn't going to be significant) you could try testing all three versions against each other in an A/B/C format to verify the results of your first test. One thing to double check would be to use a Sample Size Calculator to make sure you have a sufficient sample size for your experiment.