A lot of people blame Confidence Intervals and its ‘unintuitive’ nature for switching to Bayesian side of things.
But if you are in Marketing Mix modeling domain, frequentist concept of confidence interval makes more sense.
Before I elaborate, let me provide a quick recap of what exactly is Confidence Interval.
📌 What is confidence Interval?
Lets say we are talking about the popular 95% CI.
The definition of it would be – If one ran the same statistical test taking different samples and constructed a confidence interval each time,
then in 95% of the cases, the confidence interval so constructed for that sample will contain the true parameter.
Now that we have got the correct definition, lets answer the question of ‘Confidence’ in confidence Interval.
To understand what the word ‘confidence’ signifies, we need to first consider another word – ‘Coverage’.
📌 Confidence Interval is all about coverage.
That is, if one ran the same statistical test 90 times or 95 times by taking different samples and constructed a confidence interval each time,
would they find the parameter of interest in those intervals each time.
📌 What Confidence Interval tells you about MMM that Credible Interval does not?
I like to think of MMM as an apparatus that is trying to capture the true Marketing ROI. The data that goes into modeling is a sample from the marketing reality (population).
This data contains the true marketing ROI and your MMM is the experiment that either captures the true marketing ROI or not.
Now as a client, you would want to know how well this apparatus (MMM) is constructed and does it capture the true marketing ROI most of the time?
Confidence Intervals thus in a way informs you about the construct of the MMM model and its reliability.
📌 Now what about credible intervals?
Well credible interval tells you the probability with which the parameter of interest lies in the particular interval. Now this does not tell us anything about the build quality of the MMM process or design of experiment in general.
▪ In summary: Confidence Intervals are a barometer of your MMM process itself. So, if you are in the MMM domain, do not shun Confidence Intervals. Instead, embrace them as a critical tool for evaluating your model’s reliability.
