👋 Hi! I’m Bibin Wilson. In each edition, I share practical DevOps, MLOps & LLMOps deep dives. If someone forwarded this email to you, you can subscribe here to never miss out!
✉️ In Today’s MLOps Edition
In the last edition, we looked at the Kserver Inference platform and understood how ML models are deployed. Once the model is serving live traffic, how do you monitor the model performance?
Here is what we will cover today.
What is Model Monitoring?
Ground truth, Model performance & data monitoring metrics
Continuous Vs Batch Monitoring
Complete Model Monitoring Pipeline Architecture
Hands-on model monitoring demo using Evidently AI
FAQ: Is traditional MLOps Dead?
and more..
Before You Continue: This is part of an ongoing MLOps series. You can check this repo to go through all the previous editions in order.
Offers and Learning Resources
Planning to get CKA, CKAD, CKS, KCNA, or other Linux Foundation certifications? DevOpsCube readers can get up to 48% off using code SUMMER26CT. Check it out
Learn Claude Code, Claude Agent SDK, MCP, multi-agent systems, evaluations, and guardrails by building 13 hands-on projects. The Udacity Nanodegree also includes mentor feedback and real-world production use cases. Check it out
What is Model Monitoring?
A healthy infrastructure does not mean a healthy model.
Let’s understand this using the employee attrition model we deployed in the previous edition.
The model is running on EKS. The pod is healthy, CPU usage is normal, P95 latency is 80 milliseconds, and every request returns an HTTP 200 response. From an infrastructure point of view, everything looks fine.
However, the model may be making wrong predictions.
For example, imagine the model was trained using employee data from 2024. Two years later, hiring trends, work culture, and employee behavior may have changed. The application will continue to serve predictions without errors, but the predictions may no longer be reliable.
This is an important challenge with production ML systems.
Unlike regular applications, an ML model can lose accuracy over time even when the application and infrastructure are working perfectly. That is why MLOps requires an additional layer of monitoring called model monitoring.
Note: In the previous Data Drift, Model Decay edition, I covered a few important concepts that will help you better understand model monitoring.
Model Monitoring KPIs
For infrastructure and regular applications, the production monitoring setup goes like this. First, the infra and application team define and document a set of key performance indicators (KPIs).
For example, teams typically monitor.
Request latency (p95, p99)
Throughput (Requests per Second)
HTTP status codes (2xx, 4xx, 5xx)
CPU and memory utilization
Network and disk I/O, etc.
These metrics tell us whether the application and infrastructure are healthy. These come under operational monitoring.
Similarly, we define another set of KPIs to monitor the health of the deployed model. These metrics generally fall into two categories.
1. Model Performance Monitoring Metrics
The following are the key model performance metrics. We learned how these metrics are calculated in the model training edition (except F1 score)
Accuracy: How often the model gives the correct prediction.
Precision: When the model predicts that an employee will leave, how often it is correct.
Recall: Out of all employees who actually leave, how many the model correctly identifies.
F1 Score: It combines precision and recall into a single score.
ROC AUC: It checks whether employees who actually leave receive higher attrition risk scores than employees who stay.
These metrics measure how well the model performs by comparing its predictions against the ground truth.
So what is a ground truth?
Ground truth is the actual, verified outcome of a prediction.
In many use cases, ground truth is not available immediately. It may take days, weeks, or even months to collect the actual outcome. For our employee attrition model, the ground truth comes from the company’s HR system. So after the prediction, let’s say we define 90 days to find the ground truth.
For example, assume the model predicts that an employee may leave. After 90 days, the monitoring pipeline checks the HR system.
If the employee has left, the actual outcome is Leave.
If the employee is still working, the actual outcome is Stay.
The pipeline then matches this actual outcome with the earlier prediction using the employee ID. This allows us to calculate metrics such as accuracy, precision, recall, and F1 score.
2. Data Monitoring Metrics
Data Monitoring is the the important aspect of Model monitoring.
In our employee attrition example, employees leave, new employees join, and workplace patterns change. As a result, the production input data used may slowly become different from the data used to train the model.
These changes can lead to model decay and reduce prediction quality, as shown below.
We use the following metrics to measure the model decay.
Data Drift: How much the production data has changed from the data used to train the model.
Concept Drift: How much the patterns between employee details and whether they leave or stay have changed over time.
Feature Quality: Monitors the percentage of missing, incorrect, or invalid input values.
Important Note: As a DevOps or platform engineer, you are not responsible for selecting these metrics or deciding their thresholds. This is usually done with input from data scientists or ML engineers.
However, you should understand what the metrics mean because you may be responsible for building and operating the pipelines that collect, calculate, store, and alert on them.
Continuous Vs Batch Monitoring
By now, you might already know that not every part of model monitoring can happen continuously because to compare the actual prediction outcome, you need ground truth, which in most cases is not available immediately.
So monitoring is usually done via scheduled batch jobs (Eg: K8s CronJobs, Kubeflow scheduled runs etc..) when ground truth data becomes available.
Data quality, feature drift, and prediction drift can be checked regularly without waiting for ground truth. But it also runs in predefined windows, like once a day.
However, infrastructure monitoring or operational monitoring is continuous. Where we monitor the system’s CPU, GPU & memory usage, Throughput, Request latency, failed predictions, etc.
The model server used for inference (Eg: MLserver) also exposes metrics and traces, including inference metrics, queue metrics, and server (REST/gRPC) metrics.
Collecting Live Inferencing Data
An important part of model monitoring is collecting live inference data. For each inference request, we need to log the input data, prediction, and other metadata to a storage system such as PostgreSQL, AWS S3, or BigQuery.
Once the ground truth data becomes available, we compare this production inference dataset with the stored predictions to evaluate model performance.
To collect live inference data, you can either implement inference logging in the application layer or use the KServe Inference Logger sidecar, as shown in the following image.
Complete Model Monitoring Pipeline
Now let's look at the end-to-end model monitoring workflow. I am using the employee attrition model deployment as an example so that you understand the workflow better.
I am calling this a pipeline because model monitoring is typically a batch workload. The monitoring job runs at scheduled intervals using a Kubernetes Cronjob or a Kubeflow Pipeline that runs periodically.
The following image illustrates the full model monitoring workflow with Evidently AI open source ML monitoring library.
Here is how it works.
When a user sends a prediction request to the inference application, the application forwards the request to the deployed model running on KServe.
The model performs inference and returns the prediction, which is then sent back to the user and to a storage system like PostgreSQL or AWS S3 (production inference dataset)
At scheduled intervals, the monitoring pipeline starts running. It could be a Kubeflow pipeline, K8s CronJob etc..
The pipeline loads the reference dataset (training data) and the production inference data collected since the last monitoring run.
The pipeline then executes the Evidently Python library to compare the production data with the reference data.
Depending on the configured checks in the monitoring script, Evidently generates metrics and reports for data drift, prediction drift, data quality and performance metrics (when ground truth data is available).
The monitoring pipeline can also define thresholds for these metrics. For example, if more than 30% of the features show drift or the model accuracy drops below a predefined value, the pipeline can generate alerts and send notifications to tools like Slack, Email, or PagerDuty.
Finally, the pipeline uploads the generated monitoring reports to the Evidently Platform. It stores the monitoring results, maintains historical trends, and provides dashboards for visualizing drift, data quality, and model performance over time
Data scientists and ML engineers use these dashboards to identify issues, fix data pipelines, retrain the model, or deploy a newer version of the model etc..
Model Monitoring Using Evidently AI (Hands on)
Now, let put our learning into practice.
Important Note: For learning purposes, we will use simulated data through python script with Evidently AI. Using real data requires a complete MLOps stack for collecting inference data, storing ground truth, and running monitoring pipelines.
In the end-to-end MLOps capstone project later in the course, we will use actual model inference data for monitoring and trigger the monitoring pipeline using either Kubeflow Pipelines or a Kubernetes CronJob managed through Argo CD.
Here is what we are going to do in this setup.
Set up Evidently platform with PostgreSQL using our custom helm chart.
Test model monitoring using Evidently AI library using simulated data.
Validate model metrics in Evidently AI dashboard.
👉 Detailed Guide: Setup Evidently AI on Kubernetes for ML Model Monitoring
That’s a Wrap!
With this edition, the MLOps for DevOps series comes to an end.
If you've followed the entire series, you now have a solid foundation to start your LLMOps journey. I have many more practical LLMOps topics and hands-on guides to share with the community.
If you skipped any edition, I highly recommend going back and completing it. More importantly, don't just read the concepts. Do the hands-on exercises. As a DevOps engineer, the best way to learn is by building, experimenting, and troubleshooting the tools yourself.
Is traditional MLOps Dead?
Short answer No!
One question I usually get is,
Is traditional MLOps for classical ML models still worth learning? Meaning learning tools like MLflow, Kubeflow, feature stores, model registry, KServe, monitoring, retraining, etc.)
My answer is yes!
Many organizations are expanding their AI platforms to include foundation models and generative AI. And many use case still use tradition ML. For example, recommendation systems, fraud detection etc.
The good thing is that these tools are evolving with AI. Tools like MLflow and Kubeflow now support both traditional ML and GenAI workflows. So the time you spend learning them today will also help you build LLMOps platforms in the future.
Also, as a DevOps engineer, you learn the traditional ML concepts and apply them in the infrastructure layer rather than developing ML algorithms . So even if the industry is moving towards 100% foundation models and Gen AI, your learning will be still relevant.
Dont take my word for it! Go through MLOps job profiles and you will realize that most MLOps tools you learn will be part of the job description.
See you in the next edition!
