In my last post (ICYMI link in resources), I talked about the similarities and differences between Robyn’s Business metric Decomp RSSD and Bayesian Priors.
Decomp RSSD is a good metric but at the same time it also runs the risk of confining the user to historical spends as yardstick.
Given this limitation, the question arises – How best to use Decomp RSSD.
Here is how we use it at Aryma Labs.
We rely on our own methods to fit the MMM models and we don’t try to optimize on Decomp RSSD.
But we just use the Decomp RSSD metric in following cases:
1) For Immediate model updates
If you have a project that requires model updating at a very quick cadence (say weekly – like Gaming industry/ D2c Industry) then as a first pass it is good to see if your updated model’s Decomp is as low as possible compared to the previous model. Given this short time period it is unlikely that a drastic change in spends is required.
Of course one of the big assumption here is that your original model is accurate and has captured the ground reality. This can be tested via DID experiments (check link in resources).
Also the other big assumption is that your market reality has not changed either. As they say sometimes “there are decades where nothing happens, and there are weeks where decades”.
2) For Budget optimization exercises
If you are tasked with a scenario of finding the most optimal budget allocation under the constraint of ‘Keeping budget constant’, your optimizer will give you multiple options.
But assuming you have figured out one mix that is ideal, but for some reason that is not implementable, then you could choose another mix that has the least Decomp RSSD with respect to the mix you had identified earlier.
▪ The pre check
So we saw how Decomp RSSD can be used effectively. But is there a way to know when to apply Decomp. Yes, through Population Stability Index (PSI). PSI tells you how much your population (data) has shifted over time.
If PSI is high, chances are your new model will be quite different from the previous model. Hence the marketing strategies derived from older models won’t apply to the new.
Therefore optimizing a model on Decomp will be a futile exercise.
▪ Can we validate Decomp RSSD ?
Decomp RSSD is tricky to validate but we at Aryma Labs validate it in a indirect way. We use KL Divergence as a metric to gauge how biased MMM models are.
Model with low Decomp does not mean it is less biased. Bias indicates how much we have veered off from the ground truth. KL Divergence will tell you if your model with low Decomp is indeed less biased or not. This way we can validate a metric by another metric.
In summary:
▪ Before applying Decomp make sure your previous model reflects ground reality.
▪ Make sure you carry out precheck through PSI to test feasibility of application of Decomp.
▪ Make sure to validate Decomp RSSD itself through KL Divergence.
Resources:
Similarities between Decomp RSSD and Bayesian Priors in MMM:
https://www.linkedin.com/posts/venkat-raman-analytics_marketingmixmodeling-statistics-activity-7177914630449631232-IqeO?utm_source=share&utm_medium=member_desktop
DID Experiments:
https://www.arymalabs.com/proving-efficacy-of-mmm-through-difference-in-difference-did/
Using PSI to gauge relevancy of MMM models:
https://open.substack.com/pub/arymalabs/p/assessing-relevancy-of-marketing?r=2p7455&utm_campaign=post&utm_medium=web
KL Divergence as MMM calibration metric:
https://open.substack.com/pub/arymalabs/p/kl-divergence-as-marketing-mix-model?r=2p7455&utm_campaign=post&utm_medium=web
