👋 Hi! I’m Bibin Wilson. Every week, I publish a practical MLOps guide with hands-on examples and illustrations. If someone forwarded this email to you, you can subscribe here to never miss out!
✉️ In Today’s MLOps Edition
In the last edition, we looked into MLflow and how to register a trained model for deployment.
Today we will look at how to take the registered model and turn it into a production inference service on Kubernetes using KServe.
It covers,
What is a Model Server?
What are KServe runtimes
How KServe deploys AI models on Kubernetes
Deploy an MLflow model on KServe (hands-on)
How to roll out new model versions
Rolling Updates, Canary, A/B Testing, and Shadow Deployments
and more..
Before You Continue: This is part of an ongoing MLOps series. You can check this repo to go through all the previous editions and the complete source code used throughout this series.
Also, view the online version with all diagrams and code snippets here.
Planning to take a Kubernetes certification? The Linux Foundation is currently running a limited-time sale.
Use code JULY26CCCT at kube.promo/devops to get flat 35% OFF CKA, CKAD, CKS, and other certifications. For 45% bundle discounts and other offers, check this GitHub repository.
KServe is a CNCF incubating project for serving ML models on Kubernetes.
Simply put, KServe takes a trained model and deploys it as a scalable inference service on Kubernetes. It handles deployment, networking, autoscaling, and health checks for the model.
A key thing to understand about KServe is that it is no longer only about classic ML models.
As an inference platform, it supports two categories of AI/ML workloads.
Predictive AI (classic ML): For example, scikit-learn, XGBoost, MLflow packaged models, etc.
Generative AI (LLMs): For example, serving LLMs using the vLLM backend with GPU support.
Important Note: Our employee attrition model is a predictive AI workload. So in this edition, we will be focusing on the Predictive AI side of KServe. The core concepts you learn today, like the InferenceService, serving runtimes, and model storage, apply to LLM serving as well.
I will publish a deep dive edition of the LLM part in future editions of this series.
Didn't We Already Use KServe?
Yes. Back in Phase 1 (Step 4), we deployed the attrition model with KServe. But look closely at what we deployed back then. Here is the manifest from phase-1-local-dev/k8s/inference.yaml.
spec:
predictor:
containers:
- name: kserve-container
image: techiescamp/attrition-inference:1.0.0The model file was baked into the Docker image. Our FastAPI code and model.pkl shipped together as one image. We did that to keep the initial deployment simple.
However, in a production setup, we deploy the models in a different way. That is what you are going to learn in this edition.
What is a Model Server?
KServe is responsible for deploying and managing model-serving workloads on Kubernetes. A key thing to understand is that it is the Kubernetes orchestration layer. It doesn't perform inference itself.
Instead, KServe launches a serving pod with a model server (also called an inference server). The model server loads the model into memory, exposes HTTP/gRPC APIs, and serves inference requests.
It is similar to an Nginx server serving web pages. A model server serves a model. The following image shows how the model server runs in a pod.
Some commonly used model servers are MLServer, TensorFlow Serving (TensorFlow models), Triton Inference Server, etc.
Note: For our employee attrition model serving, we will be using MLServer. It is a Python-based inference server built on top of FastAPI and gRPC. It also has built-in support for Prometheus metrics, and it supports distributed tracing using OpenTelemetry.
Kserve Runtimes
We now know that KServe deploys a model server to serve inference requests.
But how does KServe know which model server to use?? Which container image to run? And how to configure it?
This is where KServe Runtimes come in.
A runtime is a custom resource that defines how a serving pod should be created. KServe supports two types of runtimes.
ServingRuntimes (Namespace Scoped)
ClusterServingRuntimes (Cluster Scoped)
These runtimes tell KServe which model server to launch and how to configure it for serving specific model formats.
Runtimes usually include details like the container image that contains the model server and all required libraries, startup scripts, CPU and memory resources, and the environment variables needed to run the model serving pod.
The following image illustrates it better.
Without runtimes, every team would need to build and maintain its own serving container.
Key Insight:
Large organizations usually have a platform engineering or MLOps platform team. So instead of every ML team building Docker images, the platform team typically defines cluster-wide runtimes for standard ML frameworks.
These runtimes usually include the organization's approved base images with security patches, approved library versions, logging, etc.
Individual teams usually create namespace-specific runtimes only when they need custom behavior.
KServe InferenceService
KServe gives you one Custom Resource called InferenceService.
Usually, to deploy an application, you would create a Deployment, a Service, an HPA, an Ingress, or a VirtualService.
An InferenceService combines all of that into one resource. You give it a model location, and KServe generates the deployment, networking, autoscaling, and health checks for you.
The following image shows the high-level workflow of KServe.
Here is how it works.
The KServe controller running in the Kubernetes cluster continuously watches for newly created InferenceService resources.
When a user creates an InferenceService, KServe detects it and reads the runtime configuration and model location specified in the resource.
Based on the InferenceService specification, KServe creates the required Kubernetes resources such as a Deployment, Service, and Horizontal Pod Autoscaler (HPA).
When the serving pod starts, the kserve storage initializer init container downloads the model artifacts from the configured model store (such as an S3 bucket used by MLflow) and stores them in the shared volume.
Next, the kserve-container (main container) starts the model server by running the container image specified in the selected ServingRuntime or ClusterServingRuntime. It loads the model artifacts from the shared volume (/mnt/models) into memory and serves inference requests.
Now, let's look at some core concepts of KServe.
KServe Deployment Modes
KServe has the following two deployment modes. The mode you pick depends on your model serving and platform requirements.
Mode | How It Runs | What You Get |
|---|---|---|
Knative Mode (Serverless) | On top of Knative + Istio | Scale to zero, request-based autoscaling, canary traffic splitting |
Standard Mode (Raw Deployment) | Plain Deployment + HPA and supports KEDA | Lighter footprint, but no canary rollout and no scale to zero |
For this edition, we will be using the raw Deployment mode to keep things simpler.
Deploy KServe on Kubernetes
Important Note:
We are going to deploy the MLflow model that we registered in Amazon S3 in the previous edition.
If you're following along with the hands-on, complete the previous guide up to the model registration step before continuing. If you're just reading to understand the concepts, you can safely skip it.
The following commands install certmanage, Kserver CDRs, and deploy the Kserver controller in standard mode.
$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.19.0/cert-manager.yaml
$ helm install kserve-crd oci://ghcr.io/kserve/charts/kserve-crd --version v0.18.0 -n kserve --create-namespace
$ helm install kserve oci://ghcr.io/kserve/charts/kserve-resources --version v0.18.0 \
--set kserve.controller.deploymentMode=Standard \
-n kserve
$ helm install kserve-runtime-configs oci://ghcr.io/kserve/charts/kserve-runtime-configs \
--set kserve.servingruntime.enabled=true \
--version v0.18.0Using the following commands, ensure all the CRD’s are installed and the required services are running.
$ kubectl get crds | grep kserve
$ kubectl get po -n cert-manager
$ kubectl get po -n kserveStep 1: Deploy MLflow KServe Custom Runtime
KServe comes with default runtimes for all the common ML frameworks. You can list them using the following command.
$ kubectl get clusterservingruntimesFor our use case, KServe provides a default runtime named kserve-mlserver that uses an inference server named MLServer. However, there is an MLflow version mismatch. Our project uses MLflow 3.14.0, while the default runtime uses 2.22.1.
So I created an image for the custom runtime (devopscube/mlserver-mlflow:v1.1.0). You can find the Dockerfile under phase-2-enterprise-setup/kserve directory.
Let's deploy the custom runtime with our custom image.
$ cd phase-2-enterprise-setup/kserve
$ kubectl apply -f custom-runtime.yamlNow verify and describe the created runtime to check the details.
$ kubectl get clusterservingruntimes
| grep custom
$ kubectl describe clusterservingruntime kserve-mlserver-mlflow-custom Now let's move on to deploying the model.
Step 2: Deploy the ML Model using Kserve
The inference.yaml manifest is located in the phase-2-enterprise-setup/kserve directory.
It contains the InferenceService resource, with the storageUri pointing to the S3 model artifact created by MLflow. Also, it points to the custom runtime we created (kserve-mlserver-mlflow-custom)
Open inference.yaml and update the storageUri with the location of your model as shown below.
If you are not sure where your model is stored in S3, open the MLflow Dashboard, navigate to the run that registered the model, and then go to Artifacts → MLmodel. There, you will find the artifact path, as shown below.
Key Insight:
If you notice, the storageUri points to the root directory of the model package, not to an individual model file. The storage initializer init container downloads everything under that path into /mnt/models inside the pod.
Then MLServer reads the MLmodel metadata file and determines which model artifact to load.
Once updated, run the following command to apply the manifest. It will create an inference service that creates a deployment, service, and HPA for the model.
$ kubectl apply -f inference.yamlVerify the deployment using the following command.
$ kubectl get inferenceservices,deploy,svc -n mlflowNow that the model has been deployed, let’s test the model endpoint.
Here is what happened when you deployed the inference CRD.
Key Insight:
If you don’t specify a runtime in the InferenceService, KServe looks at the modelFormat specified in the manifest. It then searches for a ServingRuntime or ClusterServingRuntime that supports that model format. If a matching runtime has autoSelect: true, KServe automatically uses it to create the serving pod.
Step 3: Validate the Deployed Model
To test the model, we will port forward its service and send a curl request to get the prediction output.
Use the following command to port forward the service.
kubectl port-forward -n mlflow svc/employee-attrition-champion-predictor 8000:80In a different terminal, use the following curl command to send a prediction request.
curl -s -X POST http://localhost:8000/invocations \
-H "Content-Type: application/json" \
-d '{
"dataframe_split": {
"columns": ["Age", "Gender", "Years at Company", "Job Role", "Monthly Income", "Work-Life Balance", "Job Satisfaction", "Performance Rating", "Number of Promotions", "Overtime", "Distance from Home", "Education Level", "Marital Status", "Number of Dependents", "Job Level", "Company Size", "Company Tenure", "Remote Work", "Leadership Opportunities", "Innovation Opportunities", "Company Reputation", "Employee Recognition"],
"data": [[31, "Male", 19, "Education", 5390, "Excellent", "Medium", "Average", 2, "No", 22, "Associate Degree", "Married", 0, "Mid", "Medium", 89, "No", "No", "No", "Excellent", "Medium"]]
}
}'It should return 0, as shown below.
{"predictions": [0]}This validates that our deployment model can serve predictions for inference requests.
Important Note:
In the above curl command, you may notice the /invocations endpoint instead of KServe's /v2/models/<model-name>/infer endpoint.
This is because we are using the MLflow runtime that implements the MLflow Scoring Protocol. It is a standard REST API specification defined by MLflow for serving models.
Rolling Out a New Model Version
The next important part is rolling out a new model version.
Let's say you have a model running in production and a newer version is registered as the Champion model in MLflow.
To deploy the new model, update the storageUri in the deployed InferenceService to point to the new MLflow model artifacts.
Since we are using Kserve Standard mode, Kserve will roll out the new model using the standard Kubernetes RollingUpdate strategy as shown below.
You can try this out by updating the URI to a different version and then applying the same inference manifest again.
Key Insight:
Model pods have one scaling behavior that application pods usually do not. Every new pod has to download the model from S3 and load it into memory before it becomes ready.
For our small scikit-learn model, that takes a few seconds. For large models, it can take minutes. This is why you need to set minReplicas based on expected baseline traffic instead of relying on the HPA to react fast. Scaling out takes longer for large model artifacts.
Advanced Model Deployment Strategies with KServe
When it comes to large-scale deployment, you cannot rely on a simple Kubernetes rollout strategy. It will work only for small-scale deployments.
The following are commonly used advanced deployment strategies.
Canary Traffic Splitting: Instead of switching all traffic at once, you send a small percentage of live traffic (say 10%) to the new model and watch it. If the predictions look healthy, you increase the percentage step by step until it reaches 100%.
A/B Testing: You route one group of users to model A and another group to model B using sticky routing and then compare business metrics such as click-through rate, conversion rate, or revenue.
Shadow Deployment: Every live request is served by the old model as usual, and the user gets the old model's response. But a copy of the same request is also sent to the new model in the background. Then you compare the logged predictions of both models on identical real traffic.
The following image illustrates it better.
Important Note: These deployment strategies are available when KServe is running in Knative Serverless mode.
What's Coming Next
In the next edition, we will move into model monitoring.
Once a model is in production, you need to continuously monitor its health and performance to ensure it continues making reliable predictions.
We will look at the Key KPIs involved in model monitoring and how to monitor them using Evidently AI.
See you in the next edition.
Note: I would love your feedback. Did you find this guide useful? If you spot anything that could be improved or have questions, let me know in the comments.
