After a lot of hard work you have built a great MMM model. The model is now powering saturation / reach frequency curves, scenario planning and Budget optimization.
The client is happy using all of these.
But the client asks you the following question –
“How long can I continue to use the MMM models, When should we update them ?”
At Aryma Labs, we liken MMM models to movie making. Just like a series of snapshots is put together to make a movie, a series of model updates gives you the ‘motion picture’ of your market.
So it is pretty clear that frequent model updates are required. But how to know exactly when to update the model?
The answer lies in Population Stability Index.
𝐏𝐨𝐩𝐮𝐥𝐚𝐭𝐢𝐨𝐧 𝐒𝐭𝐚𝐛𝐢𝐥𝐢𝐭𝐲 𝐈𝐧𝐝𝐞𝐱
Population Stability Index (PSI) is a statistic that tells you how much your population (data) has shifted over time or between any interval of time. An excellent paper on PSI by Dr. Bilal Yurdakul can be found in the resources.
PSI is a close cousin to KL Divergence. It is a symmetrised KL Divergence. In my last post (link in resources), I wrote about how we use KL divergence as a MMM calibration metric.
𝐏𝐒𝐈 𝐚𝐧𝐝 𝐊𝐋 𝐃𝐢𝐯𝐞𝐫𝐠𝐞𝐧𝐜𝐞 𝐚𝐧𝐬𝐰𝐞𝐫 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐭 𝐪𝐮𝐞𝐬𝐭𝐢𝐨𝐧𝐬.
PSI informs you about data shift.
KL Divergence informs you about model shift.
𝐇𝐨𝐰 𝐰𝐞 𝐥𝐞𝐯𝐞𝐫𝐚𝐠𝐞 𝐏𝐒𝐈 𝐢𝐧 𝐌𝐌𝐌
So once the MMM model is built and put in production (we do this in our product ArymaEdge), we compare the data on which the model was built with the latest data given by the client.
The cadence of providing new data depends on the client. It could be weekly or monthly.
We then compute the PSI between the old data and the new data.
There are some thumb rules as follows:
▪ PSI <0.10 : Little shift, no action required
▪ PSI between 0.10 – 0.25 : Moderate shift. Investigate data shift, investigate variable shift through CSI (Characteristic Stability Index)
▪ PSI > 0.25: Significant shift. Rebuild model
So using these rules, we take a decision on whether the MMM model needs to be updated or not.
𝐏𝐒𝐈 𝐚𝐬 𝐌𝐌𝐌 𝐫𝐞𝐥𝐞𝐯𝐚𝐧𝐜𝐲 𝐦𝐞𝐭𝐫𝐢𝐜
Marketing environment is always dynamic. One must make sure that the business decisions being made are relevant to current circumstances. PSI helps in evaluating the relevancy of your MMM model.
𝐈𝐧 𝐬𝐮𝐦𝐦𝐚𝐫𝐲:
Use KL Divergence to assess model fit.
Use PSI to assess MMM model relevancy.
P.S: Image credit in resources.
Resources:
KL Divergence as MMM Calibration Metric
https://www.linkedin.com/posts/venkat-raman-analytics_marketingmixmodeling-statistics-activity-7174277933312724992-X98t?utm_source=share&utm_medium=member_desktop
How often should you update your MMM model?
https://open.substack.com/pub/arymalabs/p/why-should-you-update-your-mmm-model?r=2p7455&utm_campaign=post&utm_medium=web
PSI Resources –
Dr. Bilal’s paper: https://scholarworks.wmich.edu/cgi/viewcontent.cgi?article=4249&context=dissertations
Image credit: https://mwburke.github.io/data%20science/2018/04/29/population-stability-index.html
