This post was originally published by xiao z at Towards Data Science
It’s been shared that 87% of data science projects don’t make it into production. The problem is well documented (see here, here and here).
In my experience building analytics products at Best Buy Canada, applied data science projects rarely fail because of the science. They fail because the model couldn’t be integrated into existing systems and business operations. We have had sound models showing good accuracy, even with proofs of concept demonstrating value, yet still fail to get deployed into production. The challenge isn’t POCs, its scaling.
Specifically, a gap between data scientists developing the model and engineers implementing it into systems is a recurring challenge. Data scientists tend to develop their ML models in a siloed environment away from day-to-day firefighting and even customer interactions. Playing with data in “labs.” This means we miss out on domain knowledge and context on key problems and miss out on the chance to see eye to eye with engineers and business users that will integrate, support, and use the models.
There’s little chance to feel user pain points in a lab. In turn, chances for data scientists to solve the wrong problem, develop solutions that are difficult to implement, or fail to add value in the eye of users are high. Lab work is by design hard to deploy into production.
The problem is compounded by the fact that touching million dollar money-making production systems are high risk. The code is likely a mix of founder’s code with patches bolted on top, integrated with a dozen of other systems. There are a lot of unknowns. So integrating model outputs too quickly and we run the risk of causing unforeseen issues. Do it slowly and other priorities may take over. We need to balance the need to show value of ML models with the need to evolve software features and maintain up time.
Some fellow colleagues mentioned that bringing a model into production is a case of creating an API. APIs are great delivery vehicles, but it’s a bit simplistic to think that they solve all production challenges. What if the API goes down on production software that makes $10M/hour? Is the data scientist going to drop everything at the lab and work on support with the engineering team? Who’s responsible to design experiments to scale iteratively? Does the current dev environment even allow testing of ML models if there’s limited data there? How do we monitor unforeseen results (Think Amazon’s AI recruiting tool biasing against women). We data scientists need to own up to these challenges hand in hand with production software teams, and we can’t do that well in a lab.
So we, at Best Buy Canada, don’t do labs anymore.
Most attempts at deploying data science products into production focus on tools that make integration easier, or process improvements that are somewhat confusing (case to let consultants take the lead). They, however, fail to close the structural gap: The fact that data scientists develop algorithms in a lab away from production environment. In turn, the outcome of failing to deploy into production remains unchanged. (Structure -> Process -> Outcome is a well-established quality improvement framework in healthcare that we can learn from.)
To structurally improve our chances, our data scientists now work hand in hand with tech product teams.
We’ve embedded data scientists into the respective product team that owns a given the technology we want to disrupt. We develop right in production. The real world. This aligns with one of our core values of playing as one team, and winning or losing as one team. We believe developing a model in a lab to be passed off to the engineering team is like a quarterback throwing a pass to the end zone with nobody ready to catch the ball. We want to make sure the team is ready.
For example, to build a model to optimize inventory allocation, our analytics team embed with the inventory technology team. We eat what they eat, look at the same systems and code they look at, and most importantly, are accountable to the same metrics and leaders they are. Trust is organically built between data scientists and software engineers that traditionally came from two different worlds.
This structure forces us to ask tough questions from the get-go. Do we even have resources to deliver the solution this year? What’s the value of this ML work against other product features? Who will help support the system in production? What experiments and tests need to run before we scale?
One critical question that aligns everyone’s expectations is: What do we have Capacity to do? Acknowledging what we can realistically deploy and support in production avoids developing things that can’t be integrated. For example, if there’s no resources to pull the required data into a data lake and create an automated pipeline, perhaps a better first step is to do a descriptive analysis and implement the static dataset until the data engineering team is ready to support our initiative (sometimes results from descriptive analyses are just as good).
Another key element of the embedded model is to let the product manager responsible for production software set development priorities for ML models as well. They may not have machine learning or analytics expertise; They may need assistance from an analytics technical product manager to solution and roadmap together. But they need to be the leader and the face of the initiative. Deployment into production is naturally prioritized if product managers actively ask for analytics solutions to their problems, and if they’re the ones selling it to users. This benefits data science teams as well: Instead of selling results, pitching to execs on the value of the new ML model, they can focus on delivering value: i.e. gather data, test models, refine, test, and scale.
Applied data scientists exist to solve problems. Not just do cool science. Being embedded allows us to feel business pain points directly, and design solutions that production engineers and business users are excited to implement, together.
Getting the Benefits of Lab Work in Production Teams
Developing models in a lab does however have some benefits. Here are three that we are working to replicate into our embedded structure.
#1: System Thinking: Labs allow room for more system thinking. My favorite talk on System Thinking by Dr. Ackoff stresses the importance of making sure we improve how parts interact together to improve the system as a whole.
Being embedded in a team has the danger of only focusing on what matters to that team, one small part of the system. Just improving last-mile delivery speed alone may not yield to faster fulfillment if middle-mile or inventory challenges are not addressed.
To avoid tunnel vision, we’ve found success in allowing proper time for holistic problem research before embedding into a team. This allows us to talk to anyone and everyone that touches an area of the business, probe their challenges, and make a holistic evaluation of where the problems and opportunities lie. It gives room to be curious and explore before focusing on one part. It also ensures we embed in an area of the business where we forecast making the biggest impact.
#2: Ideal Design: It also happens that Ackoff has a book on Ideal Design, which centers on the concept of solving the root problems (what will kill your business if you do nothing and why why why…). Not letting today’s solutions and constraints limit our imagination. In other words, disrupt, don’t tweak.
By being embedded and only working on what we can do this year, we run the risk of just tweaking and not disrupting. Make the current inventory allocation run 10% better, not blow it apart and get it 100% better for example.
In that regard, we start the solution design process from the ideal standpoint. Set a north star. Only when the end goal is clear do we acknowledge the team’s capability and slice out Version 1. Success translates into deploying into production incrementally, then allowing every new version to bring us closer to the north star. This is a win-win as we can test our model in the real world quickly, while business teams see business results right away.
#3: Always be Innovating: The biggest benefit of working in a lab is having the room to design new tools without being bogged down in supporting existing systems. Give room to explore, test, iterate.
Under the embedded model, what is deployed into production also needs to be supported by data scientists. That takes time and mind away from improving the model: exploring new analyses, new data features, new algorithms, etc.
We’re still formulating a best-practice and would honestly love comments on this, but something that has worked to a degree is to dedicate entire sprints (i.e. 2 weeks) to just innovation and neglect support work for that period. This prevents context switching and distractions. And we all know that trying to keep support under 30% per sprint has a tendency to turn into the opposite.
Increase the Odds of Success by Working Together
Crossing our fingers and hoping that our model makes it into production is not good enough. We need to seriously work at this. Turn the odds in our favor.
Reducing failure rate from 87% to even a 50/50 coin toss will need a combination of improvements on team structure, processes, and tools. Data science teams can do much better than startup ventures with a 1 in 10 chance of success. The product-market fit can exist on day 1 by working hand-in-hand with production software and business teams to build solutions they want, and more importantly, to specifications they can deploy.
This post was originally published by xiao z at Towards Data Science