One of the hallmarks of Frequentist philosophy is the adoption of Type 1 error rate control.
Type 1 error is about false positives.
In MMM, one has to be more wary about type 1 error than Type 2 errors.
Why?
📌 Because the implication of falsely attributing a media/marketing channel for the change in KPI means real dollars gets invested in those media that in reality was not instrumental in driving the KPI !!
You lose your hard earned dollars through wrong attribution.
The saying of John Wannamaker “Half the money I spend on advertising is wasted; the trouble is I don’t know which half” becomes ironically true.
It is ironic because MMM is supposed to solve this very problem.
Does Frequentist MMM or Bayesian MMM comes equipped at identifying Type 1 errors?
Frequentist MMM ✅
Bayesian MMM ❎
How?
📌 Frequentist MMM
Because Frequentist MMM is built using Frequentist principles, it inherently contains type 1 error control.
In my last post I mentioned, how Confidence Intervals in a way informs you about the construct of the MMM model and its reliability.
It does so because, if you have 95% CI, it tells you if one ran the same statistical test 90 times or 95 times by taking different samples and constructed a confidence interval each time, would they find the parameter of interest in those intervals each time.
This performance guarantee is only made by Frequentist MMM.
📌 Bayesian MMM ?
Well, Bayesians don’t have the concept of type 1 error control. Performance guarantee through long run experiments is antithetical to their core philosophy.
So if you are running Bayesian MMM, you don’t have any guard rails against ‘incorrect attribution’.
📌 What about Incrementality testing?
Incrementality testing has kind of become a buzz word. Every MMM vendor would use this. But if you are doing incrementality testing, it all the more becomes important that you choose Frequentist framework. Why? Again the same reason that it provides Type 1 error control.
You don’t want to falsely conclude an incremental effect of a marketing variable (when in reality there was no effect) in your incrementality tests.
So again the Frequentist framework comes to the rescue.
For the statistics nerds – One of Daniel Laken’s excellent blog (link in resources) talks about why Type 1 error is more important than Type 2. The post specifically hones on the point of caring about evidence.
Since in MMM, we care about finding evidence of attribution, Type 1 error control is very important.
📌 In summary: If you want performance guarantees and also Type 1 error control, choose Frequentist MMM. It is simple, elegant and accurate in comparison to Bayesian MMM.
Resources:
Daniel Laken’s post on Type 1 error:
https://daniellakens.blogspot.com/2016/12/why-type-1-errors-are-more-important.html?m=1
Bayesian uncertainty ≠ Frequentist uncertainty
https://www.linkedin.com/posts/venkat-raman-analytics_marketingmixmodeling-statistics-martech-activity-7150376942133776384-DXCG?utm_source=share&utm_medium=member_desktop
Adopting MMM for the first time ? Use Frequentist MMM.
https://www.linkedin.com/posts/venkat-raman-analytics_marketingmixmodeling-marketingattribution-activity-7148928932862472192-ozrW?utm_source=share&utm_medium=member_desktop
