I come across a lot of literature and talks on the internet that one should or can calibrate their MMM models through Experimentation.
I disagree.
Why?
Because Calibration and Validation are entirely different things in statistics.
ICYMI we wrote a detailed article on this subject (link in resources).
But a TL;DR version is:
Calibration is a process where you try to improve the model fit by tweaking various knobs and levers. There are metrics that tell you how well you have calibrated your model. The primary goal of calibration eventually is to reach a ‘Final Model’.
Validation on the other hand is a way to test your ‘finalized model’. Simple validation test include testing the model on ‘out of sample data’. But such hold out tests generally provide proof for only prediction accuracy.
To tests models like MMM which are more about inference rather than predictions, one would need comprehensive causal experiments.
📌 The problem with calibrating your MMM model through Experimentation.
â—¾ The time period
Generally MMMs are built on a lengthier historical time period say 2-3 yrs time frame. They also reflect long term effects of marketing inputs on the KPI.
Experiments on the other hand are more current and reflect a short term effect of marketing inputs on the KPI.
Using the information of short time span experiments to make changes to MMM (that are about long term effects) is fundamentally wrong.
By doing so, you will not only bias the model but also put the inference of the model as a whole in jeopardy.
📌 Can Experimentation totally invalidate MMM?
Even though I have written that one should use Experimentation to validate the model, there are certain limitations.
It depends on what aspect of MMM are you validating. I have seen people trying to invalidate MMMs just because the ROI numbers of MMM don’t correspond with the Experimentation.
In all probability it won’t.
Why?
Because again the time period issue.
An ROI gotten through a modeling process of ingesting 3 years of data will rarely be equal to the ROI from a experiment that is run for say 1 week or 1 month. What we get from MMM is an average ROI across the 3 years. What we get from an experiment is an ROI reflective of that current time period.
So, Experimentation can’t invalidate MMM altogether.
📌 So what can Experimentation validate in MMM?
Experimentation can tell you whether the drivers of the KPI as identified by your MMM are genuinely the drivers of your KPI or not.
&
It can quantify the effect of changes that you would make in the market because of your MMM model suggestions.
Experimentation techniques like DID can help in this regard. We regularly use DID to validate MMM for our clients.
Our detailed whitepaper (links in resources) explain this in detail.
Resources:
Validating MMM the right way: https://www.linkedin.com/posts/venkat-raman-analytics_marketingmixmodeling-statistics-linearregression-activity-7132601614355374082-lWLK?utm_source=share&utm_medium=member_desktop
DID whitepaper: https://www.arymalabs.com/proving-efficacy-of-mmm-through-difference-in-difference-did/
