Feeling like your marketing budget is vanishing without a trace? You’re adjusting campaigns and exploring new avenues, yet still unsure what’s truly effective. Incrementality testing offers a solution, revealing the genuine impact of your marketing investments.
helps you determine if your marketing spend drives the impact you intend. It’s common for professionals to contemplate this privately, possibly due to discomfort with peer discussions or social media posts.
Table Of Contents:
- The Basics of Incrementality Testing
- Why Bother with Incrementality Testing?
- Incrementality Measurement vs. Multi-Touch Attribution (MTA)
- Setting Up Your Incrementality Test, a Step-by-Step Guide
- Common Approaches of Conducting an Incremental Test
- When Results Challenge the Norm
- What Incrementality Testing Can (and Can’t) Do
- Leveraging Geo Experiments and Incrementality Testing
- Conclusion
The Basics of Incrementality Testing
Incrementality testing is a method for discovering the true value of your marketing efforts. It goes beyond examining the final ad someone clicked before purchasing.
It’s about measuring extra sales or sign-ups achieved because of your marketing. Incrementality testing takes into account whether customers would have converted regardless of your campaigns.
Imagine launching ads on a new platform and witnessing a sales jump. But some sales might have happened anyway. Incrementality testing isolates sales truly caused by those ads, using “but, for…” reasoning: Would the actions have occurred but for that specific ad interaction?
Why Bother with Incrementality Testing?
In today’s marketing, clicks are down, and costs are up, especially for pre-profit or non-funded startups. Understanding ROI, including incremental return on ad spend (iROAS), is crucial. Knowing ROI helps cut spending on ineffective activities and increase investment in those that drive results.
Forrester’s report indicates that incrementality testing can boost marketing ROI by 30%. A solid testing framework can lead to substantial efficiencies and decreased risk.
Here’s how to calculate incrementality testing clarifies things:
- Stop Wasting Money: Identify marketing spend areas not delivering expected ROI. This is like trimming fat to allocate funds to what truly boosts conversion rates.
- Smarter Budget Choices: Guides focus on improved results in marketing budget allocation. This is particularly relevant given the rise of remote work.
- See the Big Picture: It highlights the value of less direct marketing.
- Reliable Insights in a Cookie-less World: Traditional tracking is fading, but incrementality testing offers trustworthy data.
Incrementality Measurement vs. Multi-Touch Attribution (MTA)
Multi-touch attribution (MTA) assigns credit for a conversion across various customer touchpoints. This could include a display ad click, a social media interaction, and a paid search ad, each receiving partial credit.
Incrementality testing, however, gauges the additional impact. It reveals the overall effect of marketing activities compared to what would’ve occurred organically. Consider it a test group versus a control group scenario to measure incrementality.
How They are Different
Key distinctions to remember:
- Attribution is a Guessing Game: Even top MTA models make assumptions about each touchpoint’s influence. This might inaccurately credit certain activities or user acquisition/engagement stages.
- Incrementality Focuses on Cause: Did your campaign cause the sales, or would they have happened anyway? Insights enable marketers to prioritize budget allocation more predictably.
- Less Cookie-Reliant: As cookies decline, MTA becomes harder. Incrementality testing relies less on individual tracking and offers a valuable alternative in media measurement.
Setting Up Your Incrementality Test, a Step-by-Step Guide
Ready to implement your incrementality calculation testing process? Here’s a step-by-step guide:
Step 1: Select Your Goal: Determine your KPI, whether it’s increased website sign-ups, app downloads, or revenue. This ensures clear tracking and objective alignment.
Step 2: Choose the Thing You Want to Test:
This could be a focused advertising campaign, bid adjustments, or changes to brand creative assets. Be precise about the isolated variable to keep your testing budget focused.
Step 3: Find Your Groups:
Test Group: Exposed to new or modified marketing.
Control Group: Not exposed, serving as a comparison.
Group division depends on the testing context. For targeted advertising, such as paid social media or ad platforms, geographic segmentation is an option.
- Option #1: Geo-Testing: Split your audience by region, applying full marketing to some, while keeping others unchanged.
- Option #2: Use an “Intent-to-Treat”: Some platforms use algorithms to decide ad auction winners. Selecting “intent to see” uses those who *nearly* saw your ad as a comparison.
Step 4: Let the Testing Go: Allow sufficient time for statistically significant data collection, sometimes months. Observed results may shift with altered timeframes.
Step 5: Determine the Resulting Data: Analyze the numbers.
Compare KPIs between test and control groups to determine lift. If the test group performs significantly better, that’s incrementality.
Breaking Down The Math
Here’s the calculation:
(Test Group Conversion Rate – Control Group Conversion Rate) / (Test Group Conversion Rate) = Incrementality
For example, consider an ecommerce platform:
Control group: 0.4% conversions; Test group: 1.3%.
Calculation: (1.3% – 0.4%) / (1.3%) = 69.23% incrementality in sales/engagement. This indicates a lift compared to not running the ad at all.
Common Approaches of Conducting an Incremental Test
We’ve covered the core math; now let’s explore specific incrementality statistical significance testing approaches.
A/B Testing
A/B testing compares two marketing versions, such as variations in brand copy or ad graphics. Clear A/B variations and testing parameters guide advertisers in choosing which to leverage or refine.
Multi-Variate Testing (MVT)
MVT tests numerous combinations to identify the optimal element mix, revealing what drives lift. More combinations require larger audiences and/or longer durations for informed decisions.
Holdout Groups
Here, users are unexposed to the campaign from the Holdout (control group). If ads are displayed widely, a portion can be held out for comparison. Thoughtfully segment potential target audience members for accurate comparisons.
Geo-Match Market Testing
This segments campaigns across markets with Test and Control combinations. For instance, Seattle might see test campaigns while Tacoma doesn’t, but similar demographics are crucial. Geo-testing relies on multiple parameters, requiring consideration of short and long-term factors.
When Results Challenge the Norm
Incrementality tests might contradict other reporting, showing previous understandings were inaccurate. Embrace this; it improves future business growth.
When Digital Attribution and Testing Seem to Disagree: Your MTA model might heavily credit a channel. MTA measures all sales linked to ad exposure, potentially including users shifting from other channels—without increasing the customer base.
Testing can reveal some conversions would occur naturally. Testing might show lower ROI on a channel than usual reporting; this is valuable intel. Marketing attribution may make you look at data differently.
What Incrementality Testing Can (and Can’t) Do
It’s vital for understanding cause-and-effect in marketing. This test offers visibility that MTA tools and tracking software alone cannot.
For example, last-click attribution might attribute a sale to a paid search ad. An incrementality test could reveal those customers would have found you organically later.
This doesn’t devalue search. Just understand the paid search results’ “incremental” conversion value.
Leveraging Geo Experiments and Incrementality Testing
As previously mentioned, geography-based experimentation offers insights. Ensuring regions have similar characteristics before running ads and allowing ample time is critical. In ecommerce, this could mean running Facebook ads to half the U.S., leaving the rest as a comparison. Therefore retail media benefits significantly.
Several marketing platforms, including Meta with its recent incrementality-optimized updates in early 2024, utilize this approach, providing insights to take advantage of. There are measurement solutions available too.
While theoretically sound, this assumes a test/control audience split. Geographic factors like weather in California versus Seattle can affect sales for retailers with physical stores. Make sure those are controlled when running incremental lift tests. It would likely be much harder to use cities within different parts of the world.
New Zealand will be very different than Toronto, Canada. Comparing similar demographic groups aids proper lift evaluation. Factor in weather, seasonality, cultural, and linguistic differences to avoid testing for unrelated variables.
Conclusion
Incrementality testing, while seemingly complex, isn’t limited to large corporations. By correctly implementing test results, marketers can make informed advertising decisions with a statistically significant confidence level.
Embracing testing prepares marketers for a cookie-less future with enhanced privacy settings. The test results provide visibility beyond “return on ad spend” to iROAS (incremental “return on ad spend”), which can greatly benefit cross-channel marketing.
