When is a Conversion Rate Optimisation test over?
If you answered – “When Google’s website optimiser tool says it is over (with a 99% degree of statistical certainty)” then you may need to think again.
We recently ran a test on a newsletter & subscription sign up process, where the sign up required an email confirmation to complete and receive both products. The original page explained that the user had successfully signed up for the newsletter but then went on in the body of the text to say that they needed to respond to a confirmation email to complete the subscription. The variation made it clear in the headline that this was only Step 1 and made the email confirmation call to action part of the header. The test was set running and, by highlighting that the email needed to be confirmed, the variation came romping home as a winner with 99.1% chance to beat the original and more than 63% observed improvement.
Test after 12 hours (click image for larger version)
However the important point to note was that the variation was prompting people to respond to an email immediately. As you can imagine, those people who read the original page may not have checked their email immediately without prompting but would have checked later on – leaving a lag between the times to conversion between the two pages. Below is the screenshot after 36 hours and, as you can see, the result is no longer so clear.
Test after 36 hours (click image for larger version)
Although the variation is still the better page, with a 26% observed improvement, the difference between the two has been massively reduced. In this instance the variation is still almost certain to be the better page over a longer time but you can see how in closer tests this lag in conversion times could easily produce a false result.
Patience is a vital ingredient in successful testing and, although it is tempting to jump ahead when a result is being declared, it is important to be aware of elements that could skew figures and make sure you have given the test every opportunity to return a valid result.
If you have enjoyed this post you can subscribe to the rss feed to read more about how you can monitor and protect your brand online
Comment link
What’s it Worth to Ya? | Distilled blog on Fri (25 Sep) @ 2:18 pm
[...] to people who have no interest in your business. We are currently doing a lot of work to increase conversion rate optimisation for our clients, and often this means ensuring that all of our marketing efforts are going to where [...]