As most of you know, one of the interesting application of linear regression is Marketing Mix Modeling (MMM).
Even though additional bells and whistles are added in MMM over and above what a traditional linear regression entails, the core of MMM is still linear regression (or its many variants).
Given this background, it becomes imperative that you diagnose your MMM model properly.
Many misread the diagnostic plots and metrics in Linear Regression. And as a result this extends to wrong interpretation of the MMM model too.
So lets delve into what people get wrong about the F test.
But before that, let me give you a background of what F test does in the context of Linear Regression.
The F test in Linear Regression essentially tests for ‘goodness of fit’.
It has the following hypothesis:
▪ Null: The intercept only model (no independent variables) fits the data as good as your full model( all your independent variables included).
▪ Alternate : The intercept only model fits the data poorer than your your full model.
Now you run the test and get a very low p value.
Should you celebrate that your full model has provided a better fit?
No, not so fast.
The F test in Linear Regression talks more about the combined effect of all the independent variables rather than effect of individual variables.
Technically, it is telling that the coefficients of all the Independent variables are not zero. But what if you decide to run individual t tests on them?
You would sometimes be surprised to see that while the F test provides you a significant result, the individual t tests could showcase that the variables are not significant. In a way this is Simpson’s paradox (If you want to learn more about it, See link in comments).
📌 How does all this affect MMM?
Through MMM, our ultimate goal is attribution and more specifically we are looking at attribution of each individual marketing variables towards the KPI (e.g. sales, brand equity, CAC etc.)
As mentioned above, The F test in Linear Regression talks more about the combined effect of all the independent variables rather than effect of individual variables.
F test at this stage is not telling you much. It simply kinda says “you are directionally correct”. It does not provide the specifics that you desire.
Therefore one needs to go further and explore post hoc test as well.
