MlOps : Integrating ML with DevOps

5 min readMay 30, 2020

Hello Readers!!!!
As in this fast and growing IT world,we want everything to be available in just one click.And in MLOps world,the hardest part of machine learning today is deploying and maintaining accurate models, as it requires constant access to new data to update them and improve their accuracy…..

The risk of ML models not doing well is continuous and needs continuous monitoring and evaluation if they are performing within expected bound. On live data metrics like Accuracy, Precision, recall, etc. cannot be used as live data does not have labels.

MLOps:-
It is the communication between data scientists and operations teams. MLOps have mix capabilities of Data scientists and services designed to provide automation in ML pipelines and get more precious insights in production systems. It provides reproducibility, visibility, managed access control, and the computing resources to test, train, and deploy AI algorithms to Data engineers, business analyst, and operations team.

Here is one Project,base on this Mlops problem,and i have tried to automate the training of model in order to get accuracy upto 90%,and used in Malaria dataset.Use the deep learning for Malaria Detection.

Deep Learning for Malaria Detection

With regular manual diagnosis of blood smears, it is an intensive manual process requiring proper expertise in classifying and counting the parasitized and uninfected cells. Typically this may not scale well and might cause problems if we do not have the right expertise in specific regions around the world. Some advancements have been made in leveraging state-of-the-art (SOTA) image processing and analysis techniques to extract hand-engineered features and build machine learning based classification models. However these models are not scalable with more data being available for training and given the fact that hand-engineered features take a lot of time.

Deep Learning models, or to be more specific, Convolutional Neural Networks (CNNs) have proven to be really effective in a wide variety of computer vision tasks. While we assume that you have some knowledge on CNNs, in case you don’t, feel free to dive deeper into them by checking out this article here. Briefly, The key layers in a CNN model include convolution and pooling layers as depicted in the following figure.

A typical CNN architeture (Source: kaggle.com)

Convolution layers learn spatial hierarchical patterns from the data, which are also translation invariant. Thus they are able to learn different aspects of images. For example, the first convolution layer will learn small and local patterns such as edges and corners, a second convolution layer will learn larger patterns based on the features from the first layers, and so on. This allows CNNs to automate feature engineering and learn effective features which generalize well on new data points. Pooling layers help with down sampling and dimension reduction.

Task description

1. Create container image that’s has Python3 and Keras or numpy installed using dockerfile

2. When we launch this image, it should automatically starts train the model in the container.

3. Create a job chain of job1, job2, job3, job4 and job5 using build pipeline plugin in Jenkins

4. Job1 : Pull the Github repo automatically when some developers push repo to Github.

5. Job2 : By looking at the code or program file, Jenkins should automatically start the respective machine learning software installed interpreter install image container to deploy code and start training( eg. If code uses CNN, then Jenkins should start the container that has already installed all the softwares required for the cnn processing).

6. Job3 : Train your model and predict accuracy or metrics.

7. Job4 : if metrics accuracy is less than 80% , then tweak the machine learning model architecture.

8. Job5: Retrain the model or notify that the best model is being created

9. Create One extra job job6 for monitor : If container where app is running. fails due to any reason then this job should automatically start the container again from where the last trained model left

Solution:-

step1:- Create the docker file and build it to get a customized docker image..

Step2:-Creating the docker image

Step3:-Creating jobs in jenkins:-

Job1:-It will pull the github repository,As soon as developer push the ML code..I use Poll SCM for triggering the job1 and will copy all the content from workspace to /root/jenkins_work/