In my last few posts, I touched upon the following points (link to all in comments):
– Why you should not use experiments to calibrate your MMM model.
– Why you can’t use Geo experiments to fix priors of your MMM model.
– Why you should instead use experiments to only validate your MMM model.
So all this begs the question – How should one ideally validate MMM models?
📌 What about hold out sample test?
Some would opine that a simple hold out sample test could also help you validate the MMM model.
Well not really.
At best, hold out sample test may only prove the predictive ability of your MMM. MMM is just not about only prediction.
Fundamentally it is about inference and causal inference at that.
To really layer in causal inference, you would need causal experiments.
📌 General experimentation can’t validate MMM fully.
Experimentation that are univariable in nature only, measure only the effect of that variable on the KPI. But MMM is about measuring marketing effectiveness.
Marketing effectiveness is a multivariable problem. It talks about the combined effect of all the variables on the KPI and not just effect of one variable is silo.
📌 DID – The ideal way to validate your model.
At Aryma Labs, we recognize the problem of Experimentation not fully able to validate MMM. Therefore we used a quasi causal experiment that kind of mimics the marketing effectiveness process that MMM does.
For this, our experiment should not be just univariable (e.g. just making some change in Instagram ads spends in test market and the keeping it same in control).
Our experiment needs to (as much as possible) include all the effects of the independent variables on the KPI.
So here is how we re adapted our DID Experimentation:
✅ Through ROI analysis of MMM, we figured out the top 4-6 variables that affected nearly 70-80% of the change in KPI. We called them ‘heroes’
✅ We implemented MMM based increased spends strategy for these hero variables in Market A (test market) while in Market B (control) we continued the ‘as is’ strategy (the one before doing MMM).
Thus through DID, we could prove the following:
🎯 MMM identified top drivers of KPI were indeed the top drivers in reality.
🎯 MMM suggested strategy conclusively worked in driving incremental sales.
🎯 Given that MMM is a marketing effectiveness problem (that is inherently multivariable), we believe DID is the only approach which can test or validate MMM fully.
Link to the DID whitepaper in resources.
Resources:
Proving efficacy of MMM through DID :
https://www.arymalabs.com/proving-efficacy-of-mmm-through-difference-in-difference-did/
Use Experimentation to validate your MMM models, not calibrate it.
https://www.linkedin.com/posts/venkat-raman-analytics_marketingmixmodeling-experimentation-statistics-activity-7155901631820177408-tCs4?utm_source=share&utm_medium=member_desktop
Why you shouldn’t use Geo tests to fix priors in Bayesian MMM.
https://www.linkedin.com/posts/venkat-raman-analytics_marketingmixmodeling-statistics-marketingattribution-activity-7152885714853023746-3C9S?utm_source=share&utm_medium=member_desktop
Why Experimentation is not a substitute for Marketing Mix Modeling (MMM)
https://www.linkedin.com/posts/venkat-raman-analytics_marketingmixmodeling-statistics-experimentation-activity-7155179383505207296-Obnf?utm_source=share&utm_medium=member_desktop
