At Aryma Labs we constantly try to push the frontier of what’s possible in Marketing Measurement and Attribution.
In this regard, we have been working on a very interesting problem for a long time.
What is this problem?
Well, basically your MMM model should not be just good at prediction but should also have good goodness of fit (inference) or vice versa.
The question then arises, can we have a model that is good at both?
We hence decided to choose models based on 3 metrics – R squared value (provides goodness of fit), Decomp RSSD ( provides business goodness of fit – see link in resources) and finally MAPE (helps gauge prediction accuracy).
Getting a model that is good on all three metrics actually becomes a multi-objective optimization problem.
We started coding this from scratch but then realized that open source MMM library Robyn already has elements accomplishing the same.
We are not affiliated to Meta in any form. But credit where it is due, among the open source MMM libraries, Robyn seems to have all the ingredients to specify a good MMM model.
So here is what we did.
📌 Change objective function criteria
Currently Robyn optimizes a model on Decomp RSSD and NRMSE. It will give you a list of model that are minimized on both fronts.
As stated above, our philosophy is that a MMM model needs to be good at both inference as well as prediction.
We hence tweaked the Robyn code to incorporate R squared value and MAPE in addition to existing Decomp RSSD.
📌 Pareto Optimality
Robyn leverages Pareto optimality to optimize the multi objective function.
Pareto Optimality is a very interesting concept that has already been used in fields like economics, resource allocation etc.
Basically it is about reaching a stage (pareto optimality) where no one agent can benefit without causing a decline in others. So it is in essence trying to achieve a equilibrium where all agents benefits.
In our case we want a model that is good on all three objective functions of Decomp RSSD, MAPE and R squared.
We hence used Pareto Optimality.
📌 Blessing of Dimensionality
You might have heard about curse of Dimensionality (check link to my post). But adding dimensionality (like in our case of 3 objective functions) could be beneficial as well.
We noticed faster model convergence and not to mention one can have more than 10K models without having to worry much about distance collapse. We are still verifying these emergent properties (will write a detailed whitepaper or paper on this soon).
📌 About the plot:
The plot depicts 2048 models optimized on 3 objective functions. The black dots are the models that relatively good on all 3 metrics. This frontier where you see black dots is called pareto optimality frontier.
📌 In summary:
We are excited to having cracked this problem. We believe this breakthrough will help us provide MMM models that have good GOF, Business fit and Predictive properties.
Resources:
Why Decomp RSSD is business fit metric :
https://www.linkedin.com/posts/ridhima-kumar7_marketingmixmodeling-statistics-marketingattribution-activity-7134875158913196032-mudI?utm_source=share&utm_medium=member_desktop
Curse of Dimensionality :
https://www.linkedin.com/posts/aryma-labs_datascientists-artificialintelligence-machinelearning-activity-6825854742154084352-KiHM?utm_source=share&utm_medium=member_desktop
Robyn code references :
https://facebookexperimental.github.io/Robyn/docs/analysts-guide-to-MMM/
