Why AI models rarely make it to Production

Why AI models rarely make it to Production

How an MLOps mindset can revitalize your AI projects

Joint post with Tony Paikeday from NVIDIA.

Problem: ML models are hard to ship

After making artificial intelligence investments, many teams have found the ROI to be elusive. Often, great experiments fail to become great products. That pain is summed up nicely in this tweet:

After all the sweat equity invested in developing models with high predictive accuracy, it’s painful to see businesses fall into “model debt” with un-deployed or under-deployed models. More often than not, what’s missing is an “ MLOps ” strategy that unifies machine learning development and IT best practices. Let’s look at why.

Why is getting ML models into production so difficult?

1. Teams don’t implement the same DevOps rigor for data science projects as they do for more traditional dev projects. Like other production products, production AI pipelines should have monitoring, versioning, scalability, & repeatability. Organizations should budget for IT teams to implement this same level of resilience and repeatability for AI projects.
2. AI models are not like conventional software. Model performance can degrade rapidly after the code has shipped. Continuous monitoring and retraining are required, and the related tools are still largely immature — due largely to the fact that best practices are still constantly evolving. Organizations should at least plan for continued data scientist involvement even after models are in production.

How to plan for success

① People: Bring IT and data science teams together

A lot of AI work is opaque — not everybody can easily inspect or understand it. The people who build it are basically considered artisans. So, organizations often perceive that the responsibility for these projects lies solely with data science teams. This is short-sighted.

AI projects need to be integrated into the enterprise IT operation. AI developers often aren’t–and probably shouldn’t–be experts in how to build sturdy, scalable platforms.

And it’s a two-way street: both sides of the house need to plan for each other’s work. Do your data scientists build robust models? Do they have a scalable design for inference? Can your IT team support ad-hoc infrastructure requests? Do they provide self-service environments to developers? Maybe you need an “AI Engineer” role?

② Process: Implement a Design → Test → Production process for AI projects

Today, many companies haven’t implemented lifecycle management for their AI projects. Like with any product, having a defined release cycle standardizes the process. Having a little bit of process actually makes it easier for individual projects to get out the door since they don’t have to reinvent the wheel each time.

Expect for the process to evolve and grow — that’s a good sign that you’re using the system. But you have to start somewhere.

③ Technology: Plan for rapidly evolving technology

For traditional software development workflows, modern devops tools are constantly evolving. But for AI development, the tools are evolving even more rapidly as the application ecosystem rushes fill tooling gaps. Planning for evolving MLOps applications requires flexible infrastructure and a willingness to keep researching the latest schedulers, versioning tools, experiment trackers, ….

Prioritize Production instead of Experimentation

The goal of AI is to use models to impact the business, not just to run experiments for experiments’ sake. So by prioritizing production over experimentation, we’re not saying don’t do necessary exploratory work — we’re saying that all work, including exploration, should have the end result in mind.

The fastest way to bridge the gap between experimentation and production is to bridge the gap between AI teams and IT teams. Acknowledge the lift required beyond model development. Commit to implementing the same DevOps rigor for data science projects as you do for traditional dev projects.