I often come across posts where people overtly prescribe out of sample error and incrementality testing as the only way to validate the MMM model.
They often do so by belittling the importance of goodness of fit.
I am sorry to say, but people who say “R squared value and statistical significance measures (e.g. p-values) don’t help in validating the MMM model” simply don’t understand statistics and as an extension don’t understand MMM either.
If your vendor is trying to water down statistical significance measures, then you should be really wary of them.
MMM models do need to be validated on out of sample data and incrementality testing (with causal framework).
But that is Path B. One can’t reach path B without traversing Path A. That Path A is ‘Goodness of fit’.
As a MMM modeler, your first job is to make sure you have a good model fit.
Talking about model fit, lot of people blindly discard R squared value much like people callously say “correlation is not causation”.
Granted R squared value is prone to inflation as you add more variables (anyhow to mitigate this you also have adj R squared value).
But at its core, it answers one key question that should be the holy grail for MMM modelers.
That question is “how much of the variation in KPI is accounted for by the independent variables in the model”.
In MMM, the goal is attribution. But before you get to attribution, you need to make sure that you have all the right candidates that played a crucial role in impacting the KPI.
R squared value provides that clue.
Further one can always strike a balance by relying on information theoretic approaches like AIC that gives you information on parsimony of the model.
So together with R squared value, AIC and other significance measures one can gauge the goodness of fit of the model.
Another excellent way to measure goodness of fit is through Cross validation. The beauty of CV is that you won’t be wasting precious data by doing train-test split.
In case you are wondering why I am saying ‘wasting data’, check our post ‘Don’t train-test split your data in MMM’. Link in resources.
In summary – One can’t magically start validating the model through out of sample and incrementality. You need to first get right the Goodness of fit.
Goodness of fit provides you crucial information on Multicollinearity and Endogeneity. Both are kryptonite for MMM.
Please remember – The path to validating the MMM model starts with Goodness of Fit.
