ModelOps (model operations) is a holistic approach to building analytics models that can quickly progress from the lab to production. An important focus of ModelOps is to automate the deployment, monitoring, governance and continuous improvement of data analytics models running 24/7 within the enterprise.
ModelOps is a DevOps variation. While DevOps focuses on application development, ModelOps is designed for analytics. ModelOps -- which mirrors DevOps practices for ensuring IT compliance, security and manageability -- is crucial for predictive analytics at scale and enables the continuous delivery as well as smooth and efficient development and deployment of models.
Although still an emerging concept, a ModelOps approach typically involves culture, processes and technology to help organizations derive maximum value from analytics. With the rapid adoption of artificial intelligence (AI) and machine learning, analytical assets and models are multiplying at a fast pace. Although many organizations acknowledge the growing importance of a data-driven culture in their continuing digital transformation, applying changes that the data suggests is challenging.
Organizations often struggle with the final stages of "implementing, operationalizing and putting analytics to work," according to IDC. As model development becomes more prevalent for solving business problems, deployment and governance often are the last hurdle. In fact, IDC estimates that just 35 percent of organizations say that analytical models are fully deployed in production.
How does ModelOps work?
The ModelOps team helps foster communication between data scientists, data engineers, application owners and infrastructure owners and coordinates proper handoffs and execution so that models can advance to the so-called "last mile." Workflow automation, version management, promotions, compute resource management, monitoring and scaling and tuning are among ModelOps responsibilities.
Monitoring the effectiveness and performance of a ModelOps program is crucial. ModelOps involves development, testing, deployment and monitoring -- all of which can be effective only if it's making progress in providing the scale and accuracy an organization needs.
As the models degrade, they need to be adjusted and redeployed. Performance parameters include the following:
- Set up and track accuracy goals for models through development, validation and deployment.
- Identify business metrics affected by the model in operation. Determine if the model is having the intended effects.
- Track metrics such as data size and frequency of update, locations, categories and types. These metrics can help determine if model performance problems are a result of changes in the data and its sources.
- Monitor how much computing resources or memory models consume.
Elements of a ModelOps approach
ModelOps aims to stimulate dynamic collaboration and improved productivity for analytics and IT operations teams, no matter which analytic language is used, which data is accessed, or where the model will be deployed.
To bring analytics from the lab into use, a ModelOps approach includes the following, according to specialists:
- Accessing data from a trusted source and maintaining privacy and security standards.
- Avoiding rework by keeping a deployment scenario in mind when creating models.
- Retaining data lineage and track-back information for governance and audit compliance.
- Accomplishing ModelOps rapidly while ensuring close collaboration between data scientists and IT.
- Focusing on monitoring when deploying models so that analysts can monitor and retrain models as they degrade.
Benefits of ModelOps
Although not yet widely used, ModelOps can help companies that face increasing challenges in scaling their analytics to move models from the data science lab into IT production. By providing regular updates and deployments as these models are managed, scaled, monitored and retrained to push them into production, ModelOps has the potential to help companies maximize analytics' predictive capabilities and provide significant time and money savings.
McKinsey estimates that the total annual value generated by analytics and AI is between $9.5 trillion and $15.4 trillion. However, a large portion of this potential value could be lost if analytical models aren't pushed into production.
Challenges of ModelOps
Due to inefficiencies that slow down the process, many analytics models never make it to "the last mile." ModelOp, a company that specializes in ModelOps, cites the following technical challenges organizations face in deploying a model into production and offers recommendations:
- The analytics model must be compatible from the creation environment to the production environment. An agnostic scoring engine designed to take models created in any language and deploy them into production can help address the challenge of model compatibility across the analytics lifecycle.
- The model must be portable. Docker and other container technologies can help solve the application portability challenge by capturing the environmental dependencies for the analytic workload, providing a portable image.
- Monolithic and locked-in platforms may limit what organizations can do or offer services companies don't need. However, containerization technologies can help organizations to use native microservice software to address changing needs and limit service failures to isolated components.
- As the model progresses to production, it is typically exposed to larger volumes of data and data transport modes. The application and IT team will need tools for monitoring and solving performance and scalability challenges. Adopting a consistent, microservices-based approach to production analytics can help solve scalability challenges.