Calibration is fast, Validation is slow; Calibration does not require sacrifice, Validation requires sacrifice

I have written in my previous posts on why one can only validate (partially) MMMs and not calibrate it through experiments.

Let’s quickly recap what is calibration and validation.

Calibration is a process where you try to improve the model fit by tweaking various knobs and levers. There are metrics that tell you how well you have calibrated your model.
The primary goal of calibration eventually is to reach a ‘Final Model’.

Validation on the other hand is a way to test your ‘finalized model’.
Simple validation test include testing the model on ‘out of sample data’. But such hold out tests generally provide proof for only prediction accuracy.

📌 Calibration is fast, validation is slow

One key reason why we can’t calibrate our models with experiments is because experiments are slow. It could take anywhere between 1 week to 3 months for the result of experiments to trickle in. And I am speaking only in context of MMM. Your analysts can’t wait for weeks or months just to tune some parameters in the model.

That would be a huge waste of time & resource. Not to mention, the whole MMM project delivery will get delayed.

Calibration is supposed to be a quick process to get to a good model. Generally Goodness of fit measures like R squared values, P value, Standard Error, Cross validation and within sample MAPE/MAE/RMSE inform you how well you have fit the model.

All these measures are available at a quick cadence as and when the model is fit.

Therefore one of the hallmark of calibration is also quick iterations.

📌 Calibration does not require sacrifice, validation requires sacrifice

Validation of MMM models also requires sacrifice. What kind of sacrifice?

Well, MMM models can only be validated through test/control mechanism. This could be in the form of geo testing or Difference in Difference (we highly recommend this).

The problem with validating MMM models through test/control mechanism is that it calls for sacrifice.

One should be willing to forgo any positive incrementality in the control market during the period of the experiment as the MMM suggestions are not implemented.

It is for this reason we internally call control market the ‘sacrificial lamb’. For the greater good of proving out MMM’s efficacy causally, this sacrifice is required.

📌 To summarize:

First marketers should not conflate calibration and Validation.

Second, I am not denouncing experiments or validation of MMM. They are very important and we carry out tons of experimentation.

It is just that sometimes when experimentation is infeasible (due to costs or other reasons), a well calibrated model can be considered as good as a validated model.

Anyhow only a well calibrated model can be put out for validation. So calibration is first step to achieve validation.

Resources:

https://www.arymalabs.com/calibration-vs-validation-in-mmm/