Are you using Geo tests to fix priors in Bayesian MMM? You might want to rethink that.
Priors are subjective in nature, making it difficult to get a consensus on what to set, especially in a domain like MMM where you have multiple stakeholders.
While some MMM vendors try to add objectivity by using Geo tests, A/B tests, or incrementality tests, this method has severe limitations and statistical problems.
Let me explain both:
📌 Statistical problems:
1. Geo tests don’t give you attribution coefficients.
In MMM, the goal is attribution. How much of the change in KPI can be attributed to the marketing/media variables. Geo tests on the other hand don’t give you attribution coefficients.
They only vaguely tell you which media variables were instrumental in driving incremental lift. It is vague because causality mechanisms are not in place.
Geo tests are better carried out by implementing Difference in Difference (DID) (see our whitepaper – link in resources).
So through Geo tests, you kind of have a hunch that some marketing variables worked but you don’t have the crucial answer – by how much. This ‘how much’ is crucial if you want to use it in your priors. But lo and behold we don’t have it. So again MMM vendors resort to subjective priors on the variables they got through experiments.
2. The behavior of a variable in a Geo test won’t be the same in MMM.
Let’s say one ran ad campaign to test a medium through geo lift. Let’s also assume that one got conclusive proof that the medium works.
Now if you put this medium in your MMM and correspondingly set the priors for them, you will inevitably misspecify your model.
Why?
Because you are not taking into account the interaction effect the variable could have wrt to other variables.
Your experiments may be univariable but MMM is not.
📌 Limitations:
1. The cold start problem.
If you have a new client who has never adopted MMM or done geo tests. You are only left with option of subjective priors.
2. Geo lift experiments are hard to do.
To do Geo lift study in context of MMM, one has to first identify the ‘identical sister market’. In most cases, this is very hard to do because markets are different, the demographics could be different, how people respond could be different. Even if you ensure identical marketing/media spend pattern, the outcomes may not be same.
And the demarcation of ‘identical’ is more about the response of the two regions/geography not just the inputs.
Summary:
Defining priors in MMM through Geo tests is neither accurate nor easy.
Resources:
DID whitepaper – https://www.arymalabs.com/proving-efficacy-of-mmm-through-difference-in-difference-did/
