LumenMath: The Rule of Three, or Should I Keep This Audience or Platform Running?
Update 2/26/20: We made a companion calculator, which you can find here.
Scenario: you have three audiences running, one is doing well, the second isn’t, and the third just doesn’t have enough data. You keep the first one running and shut the second one off. But what about that third one?
In other words, what do you do when a target group (be it domain, platform, audience, etc.) has no performance? And how can you tell if you’ve given it enough chance to perform without having any performance metrics?
One of the most difficult parts of running digital ad campaigns can be trying to identify good targets. Whether you optimize by domain, platform or audience, picking relevant inventory can be tricky.
While target groups have different strategies and rules, they share this common problem during performance optimization.
We can make this a little more concrete with an example:
Imagine you’re running a campaign with a goal of a $25 CPA (Cost Per Acquisition — I’ll use “acquisition” and ”conversion” interchangeably) on 3 platforms.
You begin running ads and check in on them after a week or so. Platform A has spent $15 and driven one conversion. It’s currently sitting at a $15 CPA so it’s safe to say you’d want to keep that platform running.
Platform B has spent $250 and driven one conversion. It’s CPA is currently $250 so we probably want to stop running on that platform since it’s 10x our goal.
Platform C has spent $25, but has no conversions. Should you keep it running or stop spending?
Your Title Goes Here
In real conditions there’s an interplay between the platforms and an experienced ad ops specialist will take that into account when making optimizations, but this is just a simple example.
The first two platforms are pretty obvious: we can calculate the CPA of those and compare them to our goal ($25 CPA). The third is not so obvious.
CPA is supposed to be “spend/conversions,” but our conversions are zero for Platform C. As most of you probably remember from school, dividing by zero is considered a “party-foul” in math.
This brings us to the heart of the issue: how can we tell if Platform C is worth putting money into?
We can utilize a clever mathematical rule of thumb to figure this out.
The “Rule of Three”
In statistics, the “Rule of Three” states the probability of an event occurring after n observations without witnessing the event is bounded by 3/n with 95% confidence.
In other words, you can be 95% confident that the probability of something happening is (at most) 3/(number of times you’ve tried).
Your Title Goes Here
Three is an approximation. The true value here is –ln(0.05), which equals 2.99573227. But three is close enough for our purposes.
A word of caution: this rule cannot be applied indiscriminately.
Mainly because it assumes the observations are Bernoulli trials (which are way outside the scope of this article) and there is a sufficiently high number of observations (more than 30). However, we can apply it to our example above.
If we treat each impression as an observation, we can estimate the probability of an impression leading to a conversion using the Rule of Three.
Say our Platform C has spent $25 for 5000 impressions with no conversions. If each impression is an observation, then the Rule of Three tells us the maximum probability of an impression leading to a conversion is about 3/5000 = ~ 0.0006.
That’s a very small number, but driving conversions is tough. It’s not uncommon for conversions to take thousands of impressions.
The bigger question is, how does this information help us?
Why the Rule of Three is helpful
We need to dissect what this value really means. 3/5000 represents a probability — it’s an estimate of the probability of an impression leading to a conversion.
We can invert that probability (5000/3) to figure out how many impressions need to be delivered before we can expect a conversion. In this case that’s about 1667 impressions.
We know from what we’ve already spent ($25 for 5000 impressions) that our CPM (Cost Per Thousand Impressions) is $5.
If we estimate that we need 1667 to drive a conversion and we pay $5 per 1000 impressions, we may still be able to achieve a CPA of $8.34 ($5 * 1667 / 1000).
Even though we’ve spent $25 and received no conversions there’s still a chance we could get CPAs below $9!
It may seem counterintuitive, but we’ve delivered so few impressions (relatively) that we’re not ready to give up on this platform.
Knowing that we still have room to spend is fine, but how much more should we spend? Can we figure that out ahead of time?
We certainly can. Put your math hat on.
Let’s take the Rule of Three even further
Let’s turn the equation around. If CPA = (CPM / 1000) * (1/p) where p is our probability of an impression leading to a conversion, then we can set our CPA to our target and solve for p.
We know from the previous step that p = 3/n where n is the number of impressions served without a conversion.
Therefore, CPA = (CPM / 1000) * (n/3).
If we set CPA to our target and plug in our value of CPM ($5) we see:
25 = (5 / 1000) * (n/3)
25 = 5n / 3000
75000 = 5n
n = 15000
According to this formula we’ve only delivered 1/3 of the necessary impressions (5000 delivered out of 15000) to be 95% confident our CPA won’t hit target ($25).
But wait! There’s more. We can extend this further to create a general rule that can be applied to any target group and any KPI:
CPA = (CPM / 1000) * (n/3)
CPA = (CPM * n)/3000
3000 * CPA = CPM * n
n = (3000 * CPA) / CPM
Notice that by our definition of CPM,
Total Cost = CPM * n / 1000
Total Cost = CPM * [(3000 * CPA) / CPM] / 1000
Total Cost = 3 * CPA
Finally, we arrive at LumenAd’s Rule of Three. After you spend three times the KPI value, you can be 95% confident that the true value of that target group will be too high to continue.
However, that is for 95% confidence, which may be a higher level than you need. Which introduces a whole other can of worms. Let’s keep that in the Nuance Note.
Your Title Goes Here
Obviously, higher confidence is nice, but if you have a very expensive KPI, you might not have the luxury of spending 3x that value on each target group.
Maybe you’re fine with ~86%, in which case you can use the same approach to arrive at a Rule of Two (2 * KPI value).
You can even go down to ~63% confidence and you only have to spend 1 * KPI value. In this case, you could turn off the Platform C from our example.
So, the next time you’re stressing about whether to keep a platform or an audience running just remember the Rule of Three.
Hope this helps!
If you’ve used, known, or at least heard about Facebook's Campaign Budget Optimization (CBO) feature, chances are it’s made you (and your department) a little uneasy. Rest assured, I come delivering...
Updated: 4/24/2020 It’s April, and working from home and sheltering in place is now the norm for most Americans. (Thank you essential workforce for all that you are doing right now!) As we wrap our...
At this point, just about every CMO and agency executive is asking themselves what impact the COVID-19 health crisis will have on their respective industries. The reality is we’re most certainly...