We see continuous delivery for machine learning (CD4ML) as a good default starting point for any ML solution that is being deployed into production. Many organizations are becoming more reliant on ML solutions for both customer offerings and internal operations so it makes sound business sense to apply the lessons and good practice captured by continuous delivery (CD) to ML solutions.
About a decade ago we introduced continuous delivery (CD), our default way to deliver software solutions. Today's solutions increasingly include machine-learning models and we find them no exception in adopting continuous delivery practices. We call this continuous delivery for machine learning (CD4ML). Although the principles of CD remain the same, the practices and tools to implement the end-to-end process of training, testing, deploying and monitoring models require some modifications. For example: version control must not only include code but also the data, the models and its parameters; the testing pyramid extends to include model bias, fairness and data and feature validation; the deployment process must consider how to promote and evaluate the performance of new models against current champion models. While the industry is celebrating the new buzzword of MLOps, we feel CD4ML is our holistic approach to implement an end-to-end process to reliably release and continuously improve machine-learning models, from idea to production.
Applying machine learning to make the business applications and services intelligent is more than just training models and serving them. It requires implementing end-to-end and continuously repeatable cycles of training, testing, deploying, monitoring and operating the models. Continuous delivery for machine learning (CD4ML) is a technique that enables reliable end-to-end cycles of development, deploying and monitoring machine learning models. The underpinning technology stack to enable CD4ML includes tooling for accessing and discovering data, version control of artefacts (such as data, model and code), continuous delivery pipelines, automated environment provisioning for various deployments and experiments, model performance assessment and tracking, and model operational observability. Companies can choose their own tool set depending on their existing tech stack. CD4ML emphasizes automation and removing manual handoffs. CD4ML is our de facto approach for developing ML models.
With an increased popularity of ML-based applications, and the technical complexity involved in building them, our teams rely heavily on continuous delivery for machine learning (CD4ML) to deliver such applications safely, quickly and in a sustainable manner. CD4ML is the discipline of bringing CD principles and practices to ML applications. It removes long cycle times between training models and deploying them to production. CD4ML removes manual handoffs between different teams, data engineers, data scientists and ML engineers in the end-to-end process of build and deployment of a model served by an application. Using CD4ML, our teams have successfully implemented the automated versioning, testing and deployment of all components of ML-based applications: data, model and code.
Continuous delivery for machine learning (CD4ML) apply continuous delivery practices to developing machine learning models so that they are always ready for production. This technique addresses two main problems of traditional machine learning model development: long cycle time between training models and deploying them to production, which often includes manually converting the model to production-ready code; and using production models that had been trained with stale data.
A continuous delivery pipeline of a machine learning model has two triggers: (1) changes to the structure of the model and (2) changes to the training and test data sets. For this to work we need to both version the data sets and the model's source code. The pipeline often includes steps such as testing the model against the test data set, applying automatic conversion of the model (if necessary) with tools such as H2O, and deploying the model to production to deliver value.
