When working on real-world machine learning problems, you usually don’t have the luxury of clean datasets, and your goal is a business outcome, not a perfect model. One of the important tradeoffs you have to consider is ‘perfect vs good enough’ data. I experienced this firsthand when I was working with a retail chain to build an inventory demand forecasting system. The goal was to reduce overstock costs, which were about $2M annually. The data science team set a technical target: a MAPE (Mean Absolute Percentage Error) of 5% or less.
But here’s the thing: the 8-month timeline was a huge risk. So while the main data science team focused on the perfect model, as a Product Manager, I looked for the worst model that could still be more valuable than the current forecasting process.
We analyzed the manual ordering process and realized that a model with a 25% MAPE would be a great win. In fact, even a 30% or 40% MAPE would likely be good enough to start delivering value by outperforming manual forecasts. This insight gave us the justification to launch a faster, more pragmatic parallel effort.
Within two weeks, using only minimally cleaned data, we trained a simple baseline model with a 22% MAPE. It wasn’t pretty, but it was much better than the status quo. We deployed this imperfect system to 5 pilot stores and started saving the company real money in under a month while the ‘perfect’ model was still being built.
The key lessons for applied ML products are: your job is to solve business problems, not just optimize metrics. Always ask ‘why?’ What is the business value of improving this model from 20% MAPE to 15%? Is it worth three months of work? Embrace iteration and feedback loops. The fastest way to a great model is often to ship a good-enough model and learn from its real-world performance. A live model is the best source of training data. And finally, work closely with subject matter experts. Sometimes, they can give you insights that can improve your models while saving you months of work.
In the end, our iteratively improved model significantly outperformed the ‘perfect’ 6% MAPE system in reducing actual business costs. So, the next time you’re tempted to chase perfection, remember that good enough can be, well, good enough.