menu close
menu close
back to Insight Page

Why Every Digital Advertiser Needs to Test Like a Scientist

Written by: Claude Denton, Co-Founder and CTO

Advertisers have long been nicknamed “mad men.” But in the data-driven digital age, perhaps “mad scientists” is a better moniker.

There is indeed a strong parallel between scientific testing and advertising testing. When doctors and medical researchers look to evaluate the effectiveness of a new treatment, a randomized controlled trial is the gold standard of testing. It splits study participants into two randomly selected groups, giving those in one group the experimental treatment while withholding it from the other.

The scientific validity of randomized controlled trials, or RCTs, is unmatched. Luckily for data-driven marketers, the RCT methodology applies just as well to testing in digital advertising as it does in medicine. However, very few advertisers today are actively using RCTs to evaluate the actual impact of their ad campaigns.

It’s time for a change.

Incrementality is the true measure of ad effectiveness, and the best way to determine incremental lift is by running a randomized controlled trial. New tools are making it simpler and less costly to run these tests, and digital leaders are recognizing that the best path to long-term growth is to align advertising practices with the true business goals of marketers.

Why randomized controlled trials haven’t caught on in advertising

Randomized controlled trials are the ideal form of performance testing, but they aren’t always an easy option for marketers. As Florian Zettelmeyer, a marketing professor at the Kellogg School of Management at Northwestern University, explains: “RCTs are hard to run. They can be expensive to run. They can take a lot of coordination to run.”

However, if you think your existing testing strategy gives you a “good enough” assessment of your ad performance, think again. In a recent study by the Kellogg School and Facebook, researchers analyzed true advertising lift as judged by expertly managed RCTs compared to lift calculated using common alternative testing strategies. The results were lousy for the alternative tests. “The degree of variation was stunning,” according to Zettelmeyer, as they dramatically inflated the true incremental lift. In one case, lift was reported at 416%, while the RCT proved it was actually closer to 77%.

The more muted results are one reason that ad agencies don’t advocate more for RCTs. It’s not great for business to point out that the ads aren’t working as well as a brand had thought. And the time required to properly manage RCTs may not directly add to the bottom line. With a lack of turnkey RCT solutions on the market for advertisers, many end up settling for simpler testing strategies, despite the fact they can often produce wildly inaccurate results.

Unlocking the benefits of RCTs for all advertisers

Despite the challenges, randomized controlled trials are catching on in advertising. The biggest vote of confidence for RCTs came in May, when Facebook introduced Conversion Lift, a metrics platform based on RCT methodology.

Because Facebook’s users are signed in across both desktop and mobile devices, the company has a solid deterministic framework to understand which ads drove conversions and why. And the Conversion Lift technology makes direct use of RCTs in combination with that cross-device data to determine true incremental ROI.

For advertisers looking to do randomized control trials, holding out 10-20% of your audience is a good place to start.

As marketers gradually realize the benefits of RCTs, the question arises of how much of a given audience should be in the control group (with no ad exposure) versus the audience exposed to ads.

The precise split will vary across companies and markets, but holding out 10-20% of your audience is a good place to start. For the largest advertisers online, control groups of as low as 3% of the entire audience may be enough for statistically significant lift testing.

Another factor challenging marketers who are exploring RCTs is how often to run them. Past methodologies may have relied on turning ads off entirely for a period of time, then turning them back on and analyzing the lift. However, there are simply too many external factors (seasonality, consumer behavior, etc.) to make this a valid test.

Thankfully, new technologies are emerging that enable always-on RCTs to provide the most accurate real-time data on true incremental lift from advertising. Nanigans, for example, utilizes RCTs to not only measure incrementality, but also to optimize ads for higher incremental revenue across channels.

With an infinite number of variables playing into a consumer’s real-world media consumption and buying behavior, even testing with RTCs is not a marketing panacea. But for data-driven marketers that have been relying for years on deeply flawed tests and attribution rules to make multi-million dollar decisions affecting their bottom line, RCTs can make all the difference.

The Online Retailer’s Creative Playbook for Profit-Driven Holiday Advertising

Learn creative tips and examples from real-world ecommerce advertisers that will accelerate your customer growth and sales during the competitive holiday season.
Download the eBook

next post

But wait, there's more

Join Our Newsletter