PPC lesson from Kenny Rogers and some advanced maths
You gotta know when to hold ‘em, know when to fold ‘em, know when to walk away, know when to run Kenny Rogers
Sometimes, when you are running a PPC campaign, you find a keyword, ad group (or even a campaign!) that has low volume. So low that you can go a while without any conversions. When you find yourself in this situation, it’s important to know (as Kenny would say) when to fold ‘em.
The way I have approached this problem, is to select a conversion rate (whether it be a target, the conversion rate of other elements of the campaign or a historical conversion rate) and then pose the question as:
How many clicks have to elapse without a conversion before we are sure (with 95% confidence) that the conversion rate is lower than our estimated conversion rate?
The following table shows, for sample conversion rates, how many clicks have to go by without a conversion before you can be pretty sure the actual conversion rate is lower than the target:
| Historic conversion rate | Number of clicks without conversion that should worry you (95% level) |
|---|---|
| 0.1% | 2995 |
| 0.5% | 598 |
| 1.0% | 299 |
| 1.5% | 199 |
| 2.0% | 149 |
| 2.5% | 119 |
| 3.0% | 99 |
| 5.0% | 59 |
| 10.0% | 29 |
Why you need advanced maths to do PPC
You don’t actually need to have a degree in probability and statistics to manage a PPC campaign effectively. Creativity and an attention to detail are probably the greatest requirements. Calls like the one that is the subject of this post can often be made by ‘gut feel’ without needing to know exact probabilities.
Having said that, gut feel doesn’t scale. If you are running dozens of campaigns, with hundreds of ad groups and many thousands of keywords, you need to be able to automate some elements of the process. A big part of where we add value for our clients is by combining industry knowledge, creativity and that attention to detail with our mathematical approach. We build tools that leverage our maths degrees (I knew there was a reason for sitting through advanced probability and stochastic modeling) into scalable solutions that help us get deep into the heart of what’s going on with a campaign, no matter what size it is.
The maths bit - warning probability ahead!
(unless you just want to see us doing some clever stuff!)
So, the question can be reframed as:
How many clicks have to occur without a conversion before we are 95% sure that the conversion rate is less than p?
The generalised case (I have n conversions in C clicks and a supposed conversion rate of p, should I be worried?) is harder. We have a tool in development that will do this kind of statistical analysis, but it involves beta functions and I can’t do it in my head…!
The basic case, however, is pretty straight-forward undergraduate statistics.
Let W be the random variable ‘number of clicks we have to wait before we get a conversion’ (on whatever subset of the account we are watching). Then W is a random variable known as the waiting time of a sequence of Bernoulli trials. It turns out it is a geometric variable with mass function f(k) = p(1-p)^(k-1) (integer k) (*).
Let p be the probability a click converts => probability a given click doesn’t convert = 1-p.
Then P(W > k) = (1-p)^k (essentially, the probability of having to wait longer than k = probability none of the first k convert).
So, if we want to be confident at the 95% level that there is something wrong (i.e. we should have already had a conversion), we need to find the k such that:
0.05 > P(W > k) (i.e. 5% chance we wouldn’t have had a conversion within k clicks)
=> 0.05 > (1-p)^k
=> log(0.05) > k*log(1-p)
=> k > log(0.05) / log(1-p) (reverse inequality since log(1-p) is negative)
To be 99% certain, we’d need to see k > log(0.01) / log(1-p) clicks and no conversions.
Note that you can’t generalise this by simply looking for gaps of k clicks when there are no conversions over a larger number of clicks including a number of successful conversions. This is because, while 95% seems like a high number, it does mean that out of every 20 sets of k clicks, there will be one with no conversions in it. So as soon as you start looking at multiple sets of k clicks, the results no longer hold.
It’s a good quick ‘n’ dirty way of working out statistical significance on small sample sizes, however.
* Probability and Random Processes, Grimmett and Stirzaker
If you have enjoyed this post you can subscribe to the rss feed to read more about how you can monitor and protect your brand online
Comment link
Dr. Pete on Mon (12 Nov) @ 10:38 pm
“You don’t actually need to have a degree in probability and statistics to manage a PPC campaign effectively.”
Shhh… stop telling people that.
Some nice, practical math, Will. Doing stats on low-volume ads, pages, etc., is tricky, but it can be done, and doing it can often keep you from continuing down a bad path (or wasting precious data that could be used on better ads or landing pages).
Comment link
Will Critchlow on Tue (13 Nov) @ 7:37 am
It’s alright, Pete, I go on to point out why it’s handy to have one
Low-volume stats was never my strong-point (T-tests etc.) but basic probability, I can do! Good to have you commenting here…
Comment link
Jay on Wed (14 Nov) @ 11:39 pm
Can you comment on the negative binomial distribution and why both conceptually and theoretically it’s suitable for analysis of this type.
Why not a t-test?
Interesting post!
Comment link
Duncan Morris on Thu (15 Nov) @ 9:37 am
Hi Jay.
I’m sure Will will have a comment for you, but he is currently on holiday for a few days. Ill get him to respond when he is back
Comment link
Jay on Thu (15 Nov) @ 2:37 pm
Thanks, Duncan - Maybe I can get your opinion on this while Will is away.
Assume you have a list of keywords and some historical data. For the conversion rate you’re trying to to reject, is it more prudent to use the conversion rate among only keywords with conversions, or the conversion rate list wide, which is obviously diluted my keywords which don’t convert, although they may accrue as many as 3 or 4 clicks.
Further, the criteria for a negative binomial distribution:
1) user intent varies
2) broad match increases the variability of user intent
3) if your data set uses multiple landing pages and your ads are displayed in multiple regions on the nation/world - the P(conversion) is anything but constant.
Now, global warming is one thing…but pseudo science applied to pay per click optimization is a different case entirely.
Thanks,
Jay
Comment link
Tom Critchlow on Fri (16 Nov) @ 3:38 pm
Hi Jay,
While Will is certainly better at maths than I am - hopefully I can answer some of your questions while he’s away!
Firstly, with regards the individual keyphrases within an ad group - absolutely yes for the above test you should include the data from all your keyphrases. Ruling out the non-converting keyphrases will break the test we’ve outlined above. Taking an overall conversion rate for an ad group and identifying keyphrases which are likely to be poor performers would certainly be a useful test but that’s not what this does.
Regarding the use of the negative binomial distribution rather than a t-test, I think Will went down this route because it allows you to use quick and dirty maths to quickly get a feel for statistical significance. As Will mentions, this method doesn’t scale well and shouldn’t be used for comparing multiple ad groups together. That’s a different test entirely!
Regarding the fluctuation in user intent affecting P(conversion) - I think we’re making the assumption that each traffic group has a constant conversion rate P (note that P(conversion) historically doesn’t need to be the same as P(conversion) after we’ve made changes - in fact this is the very thing we’re testing).
While this obviously isn’t true (for example, the same keyphrase may convert better in the run up to christmas compared with summer) it holds true enough, specifically for small sample sizes.
It’s sad how much maths I’ve forgotten since my uni days
Anyway - I’m sure Will can weigh in with a more authoritative comment when he’s back.
Comment link
Duncan Morris on Fri (16 Nov) @ 5:14 pm
Cheers Tom..
Note to self : Never let Will write a post about maths I don’t understand and then go on holiday.
Comment link
Will Critchlow on Tue (20 Nov) @ 3:22 pm
I’m back!
I think Tom’s answer is pretty much what I would have said (you’re too hard on yourself, Tom). In particular, your assessment of the effect of seasonality is spot-on - that is the only thing that isn’t taken into account in this analysis in my opinion (either in the analysis itself or the underlying assumptions). If you stop getting conversions - it could just be the time of year and this kind of simple analysis is never going to be able to catch that.
Jay - you say:
“probability of a positive outcome is constant from trial to trial (I don’t think this is the case at all for a number of reasons)
1) user intent varies
2) broad match increases the variability of user intent
3) if your data set uses multiple landing pages and your ads are displayed in multiple regions on the nation/world - the P(conversion) is anything but constant.”
I think that while all 3 points should be considered, it doesn’t mean that the probability of a positive outcome isn’t constant from trial to trial (Tom’s seasonality point notwithstanding). If you don’t know anything about the next visitor (which ad they are being served, which landing page they are going to get, where in the world they are) and the proportion of the time you are serving up each option is unchanged, then assuming constant P(conversion) is valid.
If you throw new landing pages into the mix, or change anything else, then that is exactly what this test is designed for - testing to see if the new conversion rate is significantly worse than the old one.
Since you can do an exact negative binomial calculation, you don’t need any statistical tests.
Comment link
Will Critchlow on Tue (20 Nov) @ 3:27 pm
PS - sorry - after writing that, I realised that I wasn’t quite done:
I think that negative binomial is valid (for the reasons outlined above) - assuming we allow ourselves to discount seasonality.
In addition, a t-test would only ever be an approximation - it is designed for comparing two small samples from normal distributions. The number of conversions is never normally distributed (though it can be approximated by a normal distribution) - since we can do the exact calculation in this case with the negative binomial, we don’t need a t-test.