menu close
menu close
back to Insight Page

The Statistics of Automated Stop Loss: Part Two

Written by: Mike Chalson, Senior Optimization Engineer

In part one of this series we introduced a realistic debate between performance marketer Prudence Marshal and her excitable colleague Bob Smith. The debate centered on the appropriate settings of the Stop Loss feature in their ad automation software. To understand the arguments, we conducted some simple statistics experiments and demonstrated the relationship between expected variance in conversion rate data and false alarms, by which the Stop Loss algorithms could end up pausing a perfectly good ad.

Why false alarms can kill your ad spend

The graphs in part one of this series showed the hourly conversion rate and CPA data for a single simulated day. In order to estimate the probability of any given day experiencing a false alarm in one of its hours and pausing the $10 CPA ad, let’s run the same simulation 1000 times, representing 1000 independent trials of our simulated day. For this experiment we will define a false alarm occurred for the day any time the daily cumulative CPA exceeds the given threshold in any hour of the day.

In the first test of 1000 trials, a false alarm occurred 891 times at the $10 CPA threshold and only 44 times at the $13 CPA threshold, representing a probability of false alarm of 89% and 4.4%, respectively. For good measure (and because it’s easy) let’s rerun the 1000 trials several times and test several other threshold levels. The following curve summarizes the results:

Stop Loss 2

According to the experimental results, Bob’s recommended $10 CPA threshold has an 89.3% chance of incorrectly pausing a perfectly good ad in any given day. Prudence’s recommended $13 CPA threshold has only a 4.1% chance of doing so. It’s pretty clear that Bob’s recommended Stop Loss settings will interfere with desired spend delivery. To translate these probabilities into terms that should make the impact even more clear: for every 10 ads that really have an acceptable average CPA of $10, Bob’s threshold is expected to pause 9 of them in any given day. With Prudence’s settings, only 4 out of every 100 such ads should get paused in any given day. The later result is a much more acceptable price to pay for the protection provided by the Stop Loss feature.

To gain additional insight about the tradeoffs among stop loss false alarm rate, cost per click, conversion rate, click volume, and stop loss threshold, explore the following visual calculator:

Always be optimizing

This discussion was intended to build some general intuition about the statistical concepts underlying automated Stop Loss algorithms for digital marketing. The next level of discussion would address enhancements to the pausing logic – such as never pausing an ad with less than 3000 clicks – and would also address the other half of the false alarm tradeoff, which is the probability of correctly pausing a truly bad ad (i.e. probability of detection). Fortunately, these more advanced optimizations are handled automatically in Nanigans advertising automation software. So, when you launch your next campaign, use the automated Stop Loss feature…with confidence.

next post

But wait, there's more

Join Our Newsletter